On Wed, Apr 15, 2020 at 12:03 PM Nick Dimiduk <ndimi...@gmail.com> wrote: > > Branching off this subject from the original thread. > > On Wed, Apr 15, 2020 at 9:56 AM Andrew Purtell <andrew.purt...@gmail.com> > wrote: > > > Quick Start and Production are exclusive configurations. > > > > Yes absolutely. > > Quick Start, as you say, should have as few steps to up and running as > > possible. > > > > Production requires a real distributed filesystem for persistence and that > > means HDFS and that means, whatever the provisioning and deployment and > > process management (Ambari or k8s or...) choices are going to be, they will > > not be a Quick Start. > > > > We’ve always had this problem. The Quick Start simply can’t produce a > > system capable of durability because prerequisites for durability are not > > quick to set up. > > > > Is this exclusively due to the implementation of `LocalFileSystem` or are > there other issues at play? I've seen there's also `RawLocalFileSystem` but > I haven't investigated their relationship, it's capabilities, or if we > might profit from its use for the Quick Start experience.
There's a difference between a production system that can survive a single node failure without an outage and a production system that can recover given admin intervention when a single node fails. The quickstart guide can not produce the former. It could produce the latter, but currently does not. LocalFileSystem is the main problem. You can not us RawLocalFileSystem just by setting configuration because both it and LocalFileSystem use the "file://" scheme. I *think* current HBase code looks for LocalFileSystem and attempts to unwrap the RawLocalFileSystem inside. I know the ability to do this is not a supported Hadoop thing. I do not know how robust our handling of doing it is.
