Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

2017-10-28 Thread Konstantin Shvachko 
Hey guys,

It is an interesting question whether Ozone should be a part of Hadoop.
There are two main reasons why I think it should not.

1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
sizable community behind, it looks to me like a whole new project.
It is essentially a new storage system, with different (than HDFS)
architecture, separate S3-like APIs. This is really great - the World sure
needs more distributed file systems. But it is not clear why Ozone should
co-exist with HDFS under the same roof.

2. Ozone is probably just the first step in rebuilding HDFS under a new
architecture. With the next steps presumably being HDFS-10419 and
HDFS-8.
The design doc for the new architecture has never been published. I can
only assume based on some presentations and personal communications that
the idea is to use Ozone as a block storage, and re-implement NameNode, so
that it stores only a partial namesapce in memory, while the bulk of it
(cold data) is persisted to a local storage.
Such architecture makes me wonder if it solves Hadoop's main problems.
There are two main limitations in HDFS:
  a. The throughput of Namespace operations. Which is limited by the number
of RPCs the NameNode can handle
  b. The number of objects (files + blocks) the system can maintain. Which
is limited by the memory size of the NameNode.
The RPC performance (a) is more important for Hadoop scalability than the
object count (b). The read RPCs being the main priority.
The new architecture targets the object count problem, but in the expense
of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
Also based on the use patterns on our large clusters we read up to 90% of
the data we write, so cold data is a small fraction and most of it must be
cached.

To summarize:
- Ozone is a big enough system to deserve its own project.
- The architecture that Ozone leads to does not seem to solve the intrinsic
problems of current HDFS.

I will post my opinion in the Ozone jira. Should be more convenient to
discuss it there for further reference.

Thanks,
--Konstantin



On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei  wrote:

> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
>   1.  All services, management scripts
>   2.  Object store APIs, exposed via both REST and RPC
>   3.  Master service UIs, command line interfaces
>   4.  Pluggable pipeline Integration
>   5.  Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
>   6.  Corona - a load generator for Ozone.
>   7.  Essential documentation added to Hadoop site.
>   8.  Version specific Ozone Documentation, accessible via service UI.
>   9.  Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confl
> uence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>


On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei  wrote:

> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
>   1.  All services, management scripts
>   2.  Object store APIs, exposed via both REST and RPC
>   3.  Master service UIs, command line interfaces
>   4.  Pluggable pipeline Integration
>   5.  Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
>   6.  Corona - a load generator for Ozone.
>   7.  Essential documentation added to Hadoop site.
>   8.  Version specific Ozone Documentation, accessible via service UI.
>   9.  Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> cluster+with+docker.
>
>
> We have built a passionate and

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-10-28 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/571/

[Oct 27, 2017 9:41:37 PM] (yufei) YARN-6505. Define the strings used in SLS 
JSON input file format.
[Oct 27, 2017 9:56:05 PM] (jlowe) YARN-7244. ShuffleHandler is not aware of 
disks that are added.
[Oct 27, 2017 10:36:03 PM] (cdouglas) HDFS-12582. Replace HdfsFileStatus 
constructor with a builder pattern.
[Oct 27, 2017 11:46:05 PM] (inigoiri) YARN-7276. Federation Router Web Service 
fixes. Contributed by Inigo
[Oct 28, 2017 1:47:45 AM] (cdouglas) HADOOP-14986. Enforce JDK limitations.
[Oct 28, 2017 4:59:27 AM] (cdouglas) HADOOP-14991. Add missing figures to 
Resource Estimator tool.
[Oct 28, 2017 5:56:41 AM] (Arun Suresh) YARN-7178. Add documentation for 
Container Update API. (asuresh)
[Oct 28, 2017 6:08:18 AM] (Arun Suresh) YARN-7299. Fix 
TestDistributedScheduler. (asuresh)




-1 overall


The following subsystems voted -1:
asflicense findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-tools/hadoop-resourceestimator 
   Dead store to jobHistory in 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.getHistoryResourceSkyline(String,
 String) At 
ResourceEstimatorService.java:org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.getHistoryResourceSkyline(String,
 String) At ResourceEstimatorService.java:[line 196] 
   Incorrect lazy initialization and update of static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStore
 in new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() 
At ResourceEstimatorService.java:of static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStore
 in new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() 
At ResourceEstimatorService.java:[lines 78-82] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.config 
from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 80] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.gson from 
instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 106] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.logParser 
from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 86] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.rleType 
from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 108] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStore
 from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 82] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStoreType
 from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 111] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.solver 
from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 92] 
   Found reliance on default encoding in 
org.apache.hadoop.resourceestimator.translator.impl.BaseLogParser.parseStream(InputStream):in

[jira] [Created] (HDFS-12743) Fix env vars (again)

2017-10-28 Thread Allen Wittenauer (JIRA) 
Allen Wittenauer created HDFS-12743:
---

 Summary: Fix env vars (again)
 Key: HDFS-12743
 URL: https://issues.apache.org/jira/browse/HDFS-12743
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer


Ozone's environment variables are out of whack with the rest of Hadoop again.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12742) Add support for KSM --expunge command

2017-10-28 Thread Shashikant Banerjee (JIRA) 
Shashikant Banerjee created HDFS-12742:
--

 Summary: Add support for KSM --expunge command
 Key: HDFS-12742
 URL: https://issues.apache.org/jira/browse/HDFS-12742
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: HDFS-7240
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee
 Fix For: HDFS-7240


KSM --expunge will delete all the data from the data nodes for all the keys in 
the KSM db. 
User will have no control over the deletion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

2017-10-28 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/14/

[Oct 27, 2017 11:55:12 AM] (stevel) HADOOP-14553. Add (parallelized) 
integration tests to hadoop-azure.
[Oct 27, 2017 2:45:11 PM] (arp) HDFS-9914. Fix configurable WebhDFS 
connect/read timeout. Contributed by
[Oct 27, 2017 3:41:48 PM] (sunilg) YARN-7375. Possible NPE in  RMWebapp when HA 
is enabled and the active
[Oct 27, 2017 5:21:41 PM] (rohithsharmaks) YARN-7289. Application lifetime does 
not work with FairScheduler.
[Oct 27, 2017 10:09:29 PM] (jlowe) YARN-7244. ShuffleHandler is not aware of 
disks that are added.
[Oct 28, 2017 4:42:42 AM] (cdouglas) HADOOP-14986. Enforce JDK limitations.
[Oct 28, 2017 5:01:41 AM] (cdouglas) HADOOP-14991. Add missing figures to 
Resource Estimator tool.
[Oct 28, 2017 5:57:46 AM] (Arun Suresh) YARN-7178. Add documentation for 
Container Update API. (asuresh)
[Oct 28, 2017 6:11:34 AM] (Arun Suresh) YARN-7299. Fix 
TestDistributedScheduler. (asuresh)


[Error replacing 'FILE' - Workspace is not accessible]

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12741) ADD support for KSM --createObjectStore command

2017-10-28 Thread Shashikant Banerjee (JIRA) 
Shashikant Banerjee created HDFS-12741:
--

 Summary: ADD support for KSM --createObjectStore command
 Key: HDFS-12741
 URL: https://issues.apache.org/jira/browse/HDFS-12741
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: HDFS-7240
Reporter: Shashikant Banerjee
 Fix For: HDFS-7240


KSM --createObjectStore command reads the ozone configuration information and 
creates the KSM version file and reads the SCM version file from the SCM 
specified.

The SCM version file is stored in the KSM metadata directory and before 
communicating with an SCM KSM verifies that it is communicating with an SCM 
where the relationship has been established via createObjectStore command.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12740) SCM should support a RPC to share the cluster Id with KSM and DataNodes

2017-10-28 Thread Shashikant Banerjee (JIRA) 
Shashikant Banerjee created HDFS-12740:
--

 Summary: SCM should support a RPC to share the cluster Id with KSM 
and DataNodes
 Key: HDFS-12740
 URL: https://issues.apache.org/jira/browse/HDFS-12740
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: HDFS-7240
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee
 Fix For: HDFS-7240


When the ozone cluster is first Created, SCM --init command will generate 
cluster Id as well as SCM Id and persist it locally. The same cluster Id and 
the SCM id will be shared with KSM during the KSM initialization and Datanodes 
during datanode registration. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12739) Add Support for SCM --init command

2017-10-28 Thread Shashikant Banerjee (JIRA) 
Shashikant Banerjee created HDFS-12739:
--

 Summary: Add Support for SCM --init command
 Key: HDFS-12739
 URL: https://issues.apache.org/jira/browse/HDFS-12739
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: HDFS-7240
Affects Versions: HDFS-7240
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org