[jira] [Created] (HADOOP-13651) S3Guard: S3AFileSystem Integration with MetadataStore

2016-09-23 Thread Aaron Fabbri (JIRA) 
Aaron Fabbri created HADOOP-13651:
-

 Summary: S3Guard: S3AFileSystem Integration with MetadataStore
 Key: HADOOP-13651
 URL: https://issues.apache.org/jira/browse/HADOOP-13651
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Reporter: Aaron Fabbri


Modify S3AFileSystem et al. to optionally use a MetadataStore for metadata 
consistency and caching.

Implementation should have minimal overhead when no MetadataStore is configured.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13650) S3Guard: Provide command line tools to manipulate metadata store.

2016-09-23 Thread Lei (Eddy) Xu (JIRA) 
Lei (Eddy) Xu created HADOOP-13650:
--

 Summary: S3Guard: Provide command line tools to manipulate 
metadata store.
 Key: HADOOP-13650
 URL: https://issues.apache.org/jira/browse/HADOOP-13650
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Reporter: Lei (Eddy) Xu


Similar systems like EMRFS has the CLI tools to manipulate the metadata store, 
i.e., create or delete metadata store, or {{import}}, {{sync}} the file 
metadata between metadata store and S3. 

http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emrfs-cli-reference.html

S3Guard should offer similar functionality. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13649) s3guard: implement time-based (TTL) expiry for LocalMetadataStore

2016-09-23 Thread Aaron Fabbri (JIRA) 
Aaron Fabbri created HADOOP-13649:
-

 Summary: s3guard: implement time-based (TTL) expiry for 
LocalMetadataStore
 Key: HADOOP-13649
 URL: https://issues.apache.org/jira/browse/HADOOP-13649
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Reporter: Aaron Fabbri
Assignee: Aaron Fabbri


LocalMetadataStore is primarily a reference implementation for testing.  It may 
be useful in narrow circumstances where the workload can tolerate short-term 
lack of inter-node consistency:  Being in-memory, one JVM/node's 
LocalMetadataStore will not see another node's changes to the underlying 
filesystem.

To put a bound on the time during which this inconsistency may occur, we should 
implement time-based (a.k.a. Time To Live / TTL)  expiration for 
LocalMetadataStore




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13648) s3a home directory to be "/"

2016-09-23 Thread Steve Loughran (JIRA) 
Steve Loughran created HADOOP-13648:
---

 Summary: s3a home directory to be "/"
 Key: HADOOP-13648
 URL: https://issues.apache.org/jira/browse/HADOOP-13648
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Affects Versions: 2.8.0
Reporter: Steve Loughran
Priority: Minor


The home directory of an s3a instances is {{/user/" + 
System.getProperty("user.name"))}}. As HADOOP-12774 notes, it gets the user 
wrong: if it were to be correct it should use the shortname of the current 
principal.

I don't think the username is valid here at all. s3a buckets are not 
filesystems with users and permissions; all this per-user home dir appears to 
do is cause confusion, and end up putting the output of an {{hadoop fs -rm}} 
operation into a directory under it.

If we made it "/" then it'd be the same for all users, and "/.Trash" would be 
where deleted files get copied to



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

2016-09-23 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/103/

[Sep 22, 2016 4:37:06 PM] (ozawa) HADOOP-13602. Fix some warnings by findbugs 
in hadoop-maven-plugin.
[Sep 22, 2016 6:43:11 PM] (wang) HDFS-10877. Make 
RemoteEditLogManifest.committedTxnId optional in
[Sep 22, 2016 11:12:56 PM] (rkanter) MAPREDUCE-6632. Master.getMasterAddress() 
should be updated to use
[Sep 22, 2016 11:45:34 PM] (rkanter) YARN-4973. YarnWebParams 
next.fresh.interval should be
[Sep 23, 2016 1:00:49 AM] (naganarasimha_gr) YARN-3692. Allow REST API to set a 
user generated message when killing
[Sep 23, 2016 2:36:16 AM] (aengineer) HDFS-10871. DiskBalancerWorkItem should 
not import jackson relocated by
[Sep 23, 2016 7:53:54 AM] (varunsaxena) TimelineClient failed to retry on 
java.net.SocketTimeoutException: Read
[Sep 23, 2016 7:55:46 AM] (varunsaxena) Revert "TimelineClient failed to retry 
on
[Sep 23, 2016 7:57:31 AM] (varunsaxena) YARN-5539. TimelineClient failed to 
retry on
[Sep 23, 2016 9:01:30 AM] (stevel) HADOOP-13643. Math error in 
AbstractContractDistCpTest. Contributed by




-1 overall


The following subsystems voted -1:
compile unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer 
   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider 
   hadoop.hdfs.TestDatanodeRegistration 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService 
   hadoop.yarn.server.nodemanager.TestNodeManagerShutdown 
   hadoop.yarn.server.timeline.TestRollingLevelDB 
   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices 
   hadoop.yarn.server.timeline.TestTimelineDataManager 
   hadoop.yarn.server.timeline.TestLeveldbTimelineStore 
   hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore 
   hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore 
   
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer 
   hadoop.yarn.server.timelineservice.storage.common.TestRowKeys 
   hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters 
   hadoop.yarn.server.timelineservice.storage.common.TestSeparator 
   hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore 
   hadoop.yarn.server.resourcemanager.TestRMRestart 
   hadoop.yarn.server.resourcemanager.TestResourceTrackerService 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.client.api.impl.TestNMClient 
   hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore 
   hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient 
   hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore 
   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorage 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
 
   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun 
   
hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl
 
   
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapred.TestShuffleHandler 
   hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService 
   hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers 

Timed out junit tests :

   org.apache.hadoop.hdfs.TestFileChecksum 
   org.apache.hadoop.hdfs.TestReconstructStripedFile 
   org.apache.hadoop.hdfs.TestWriteReadStripedFile 
   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache 
   org.apache.hadoop.mapred.TestMRIntermediateDataEncryption 
   org.apache.hadoop.mapred.TestMROpportunisticMaps 
  

   compile:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/103/artifact/out/patch-compile-root.txt
  [308K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/103/artifact/out/patch-compile-root.txt
  [308K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/103/artifact/out/patch-compile-root.txt
  [308K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/103/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [196K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/103/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [52K]

Re: HADOOP-13636 and io.bytes.per.checksum

2016-09-23 Thread Andrew Wang 
Have you git blamed to dig up the original JIRA conversation? I think that
deprecation predates many of us, so you might not get much historical
perspective from the mailing list.

I'm happy to lend a +1 though, since like you said, it doesn't seem like
that config key is going anywhere.

On Fri, Sep 23, 2016 at 1:52 AM, Steve Loughran 
wrote:

> I got silence from HDFS dev here, so I'm raising it on common dev.
>
> Why is HDFS tagging as deprecated " io.bytes.per.checksum ", given its an
> option being set in core-default, and used by other filesystems?
>
>
> >INFO  Configuration.deprecation 
> >(Configuration.java:warnOnceIfDeprecated(1182))
> - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
>
> I don't see why it should be deprecated. If it isn't what HDFS likes, then
> the code there could be smarter: look for the dfs value, and if not found
> then grab the io.bytes one —warning the user.
>
> I will volunteer to write this code if I get a promise that someone agrees
> with the premise and is willing to help nurture it in.
>
> Begin forwarded message:
>
> From: Steve Loughran  >>
> Subject: HADOOP-13636 and io.bytes.per.checksum
> Date: 21 September 2016 at 17:12:00 BST
> To: "hdfs-...@hadoop.apache.org" <
> hdfs-...@hadoop.apache.org>
>
> I'm getting told off for using the deprecated option: io.bytes.per.checksum
>
> https://issues.apache.org/jira/browse/HADOOP-13636
>
> Except: I'm not. FileSystem.getServerDefaults() is, which is used by Trash
> to work out where to delete things.
>
> It strikes me that the system is inconsitent: HdfsConfiguration is
> deprecating a property that everything else is happy to use; I see it in
> four places in production, and various tests, plus core-default.xml
>
> Is it really deprecated? If so, are there any volunteers to remove it from
> the codebase, while pulling up the default value into core-default?
>
> otherwise: how about the complaint is turned off?
>
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2016-09-23 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/173/

[Sep 22, 2016 4:37:06 PM] (ozawa) HADOOP-13602. Fix some warnings by findbugs 
in hadoop-maven-plugin.
[Sep 22, 2016 6:43:11 PM] (wang) HDFS-10877. Make 
RemoteEditLogManifest.committedTxnId optional in
[Sep 22, 2016 11:12:56 PM] (rkanter) MAPREDUCE-6632. Master.getMasterAddress() 
should be updated to use
[Sep 22, 2016 11:45:34 PM] (rkanter) YARN-4973. YarnWebParams 
next.fresh.interval should be
[Sep 23, 2016 1:00:49 AM] (naganarasimha_gr) YARN-3692. Allow REST API to set a 
user generated message when killing
[Sep 23, 2016 2:36:16 AM] (aengineer) HDFS-10871. DiskBalancerWorkItem should 
not import jackson relocated by
[Sep 23, 2016 7:53:54 AM] (varunsaxena) TimelineClient failed to retry on 
java.net.SocketTimeoutException: Read
[Sep 23, 2016 7:55:46 AM] (varunsaxena) Revert "TimelineClient failed to retry 
on
[Sep 23, 2016 7:57:31 AM] (varunsaxena) YARN-5539. TimelineClient failed to 
retry on




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery 
   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/173/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/173/artifact/out/diff-compile-javac-root.txt
  [168K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/173/artifact/out/diff-checkstyle-root.txt
  [16M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/173/artifact/out/diff-patch-pylint.txt
  [16K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/173/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/173/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/173/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/173/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/173/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/173/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [188K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/173/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/173/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [268K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/173/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
  [120K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/173/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-11010) Post-9902 "Umbrella" JIRA

2016-09-23 Thread Allen Wittenauer (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-11010.
---
Resolution: Fixed

> Post-9902 "Umbrella" JIRA
> -
>
> Key: HADOOP-11010
> URL: https://issues.apache.org/jira/browse/HADOOP-11010
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>
> Umbrella JIRA to keep track of bug fixes and enhancements, now that the major 
> portion of the shell script rewrite has been committed.  Some of these are 
> the result of 9902, others are just projects that I missed that should really 
> be done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13647) cannot writeData to HDFS form localfile

2016-09-23 Thread xiboliyalangwolf (JIRA) 
xiboliyalangwolf created HADOOP-13647:
-

 Summary: cannot writeData to HDFS form localfile
 Key: HADOOP-13647
 URL: https://issues.apache.org/jira/browse/HADOOP-13647
 Project: Hadoop Common
  Issue Type: Bug
Reporter: xiboliyalangwolf


when i try to use hadoop api to write data to cluster HDFS ,alway throw 
exception.like that:
Exception in thread "main" java.util.ServiceConfigurationError: 
org.apache.hadoop.fs.FileSystem: Provider 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem could not be instantiated: 
java.lang.NoClassDefFoundError: org/codehaus/jackson/map/ObjectMapper
at java.util.ServiceLoader.fail(ServiceLoader.java:224)
at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
Caused by: java.lang.NoClassDefFoundError: org/codehaus/jackson/map/ObjectMapper


 FileSystem fs = FileSystem.get(URI.create(file), conf);//this line is wrong?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Fwd: HADOOP-13636 and io.bytes.per.checksum

2016-09-23 Thread Steve Loughran 
I got silence from HDFS dev here, so I'm raising it on common dev.

Why is HDFS tagging as deprecated " io.bytes.per.checksum ", given its an 
option being set in core-default, and used by other filesystems?


>INFO  Configuration.deprecation 
>(Configuration.java:warnOnceIfDeprecated(1182)) - io.bytes.per.checksum is 
>deprecated. Instead, use dfs.bytes-per-checksum

I don't see why it should be deprecated. If it isn't what HDFS likes, then the 
code there could be smarter: look for the dfs value, and if not found then grab 
the io.bytes one —warning the user.

I will volunteer to write this code if I get a promise that someone agrees with 
the premise and is willing to help nurture it in.

Begin forwarded message:

From: Steve Loughran >
Subject: HADOOP-13636 and io.bytes.per.checksum
Date: 21 September 2016 at 17:12:00 BST
To: "hdfs-...@hadoop.apache.org" 
>

I'm getting told off for using the deprecated option: io.bytes.per.checksum

https://issues.apache.org/jira/browse/HADOOP-13636

Except: I'm not. FileSystem.getServerDefaults() is, which is used by Trash to 
work out where to delete things.

It strikes me that the system is inconsitent: HdfsConfiguration is deprecating 
a property that everything else is happy to use; I see it in four places in 
production, and various tests, plus core-default.xml

Is it really deprecated? If so, are there any volunteers to remove it from the 
codebase, while pulling up the default value into core-default?

otherwise: how about the complaint is turned off?

[jira] [Created] (HADOOP-13645) Refine TestRackResolver#testCaching to ensure cache is truly tested

2016-09-23 Thread Weiwei Yang (JIRA) 
Weiwei Yang created HADOOP-13645:


 Summary: Refine TestRackResolver#testCaching to ensure cache is 
truly tested
 Key: HADOOP-13645
 URL: https://issues.apache.org/jira/browse/HADOOP-13645
 Project: Hadoop Common
  Issue Type: Test
  Components: util
Reporter: Weiwei Yang
Assignee: Weiwei Yang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org