Hi Dan, Very appreciated with the sharing!
Regards, xiaochun -----Original Message----- From: Daniel Vimont [mailto:dan...@commonvox.org] Sent: Tuesday, August 16, 2016 10:50 AM To: user@hbase.apache.org Subject: Re: Fully distributed mode on Windows Hello Xiaochun, Below are the notes that I took when I was trying to install HBase in a Windows environment using Cygwin. These notes are from approximately May of 2015, so I have no idea how well they apply to any current attempt to install HBase in this kind of environment. Also, I'm afraid that I can't provide any clarification on anything in the notes below, since I have only the vaguest memories of having performed the steps outlined. Importantly, also note that I attempted this install BEFORE completing the rather-straightforward installation of HBase in Ubuntu, so some of my steps below may have been blatantly wrong in the context of ANY attempted installation of HBase. I simply offer these unedited notes to you "as is" in case there's anything here that might help you. Again, good luck!! Dan =================== *HBase with cygwin installation notes* Decided to try to install HBase using the instructions on the project site itself (although even this resource may not be up-to-date): https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fhbase.apache.org%2fcygwin.html&data=01%7c01%7cxiliang%40064d.mgd.microsoft.com%7cf86db4862d0e467a9e0c08d3c58018ed%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=%2bg%2f7hhViguPrVF%2fY5SXiICTi0TVae%2fQ3zO8bH9PMU4M%3d In parallel, I also consulted the webpage on the O’Reilly site with instructions for installing Hadoop: https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2farchive.oreilly.com%2fpub%2fa%2fother-programming%2fexcerpts%2fhadoop-tdg%2finstalling-apache-hadoop.html&data=01%7c01%7cxiliang%40064d.mgd.microsoft.com%7cf86db4862d0e467a9e0c08d3c58018ed%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=deoNblFWR44lyaZgyDlGQ1dS%2fI4YN2rtNFYYzMjyQE8%3d Downloaded cygwin and installed it. The elaborate instructions that state that setting of Windows Environment Variable (CYGWIN_HOME) and PATH variable appear to be extraneous, since Cygwin “just worked” for me after I installed it. The project site install instructions advise you to install the SSH modules of Cygwin; however, the O’Reilly site advises that SSH is not needed in stand-alone mode (since no daemons will be started). So, for now, I skipped SSH installation (although I will need SSH when I get my new full development environment set up in a few days). Rather curiously, the project site install instructions then say that HBase must be “installed in Cygwin”!! (The O’Reilly instructions merely say that it must be somewhere on the machine, but the O’Reilly instructions assume you’re in Unix, so I will go with the project-site instructions and install HBase in a Cygwin subdirectory, as they advise. Ran the “unzip” (tar) facility to unzip the HBase files after I downloaded them from Apache server. Then the “readme” directed me to the “index.html” file in the “docs” folder of the installed files; I went into this and went directly to “Quick start (standalone HBase)”. The quick-start guide said to set JAVA_HOME, which I’ve already done. Then it says to alter hbase-site.xml in the ./conf subdirectory, in order to set the directories to which HBase and ZooKeeper write data. However, it says that if you do NOT do this, then data will be written to a /tmp directory. This is okay with me for now, so will NOT modify hbase-site.xml for now. Ugh -- quite confusing contradictory instructions in the various guides, so now I go back to the “HBase” chapter in the book, which has very simple (perhaps too simple?) instructions. Followed the chapter’s instructions to create HBASE_HOME environment variable, and then issue the command in Cygwin to add HBASE_HOME/bin to path. However, the “export” command does not apparently work in the Cygwin context, so manually added HBASE_HOME/bin in Windows “path” env variable. Subsequently running the start-hbase.sh script (as simple book instructions say to do) results in errors (various basic HBase classes not found) -- apparent classpath issues. SOOO -- better go back to more complex instructions https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fhbase.apache.org%2fcygwin.html&data=01%7c01%7cxiliang%40064d.mgd.microsoft.com%7cf86db4862d0e467a9e0c08d3c58018ed%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=%2bg%2f7hhViguPrVF%2fY5SXiICTi0TVae%2fQ3zO8bH9PMU4M%3d -- SPECIFICALLY, need to go back to Cygwin installation and set appropriate env and path variables: CYGWIN_HOME and add CYGWIN_HOME/bin to path. Had a hard time interpreting the following (but finally figured it out): > > > > > *JavaOne important thing to remember in shell scripting in general (i.e. > *nix and Windows) is that managing, manipulating and assembling path > names that contains spaces can be very hard, due to the need to escape > and quote those characters and strings. So we try to stay away from > spaces in path names. *nix environments can help us out here very > easily by using symbolic links.Create a link in /usr/local to the Java > home directory by using the following command and substituting the > name of your chosen Java environment:LN -s /cygdrive/c/Program\ > Files/Java/<jre name> /usr/local/<jre name>Test your java installation > by changing directories to your Java folder CD /usr/local/<jre name> > and issueing the command ./bin/java -version. This should output your > version of the chosen JRE.* The left slash after “Program” above is an escape character for the space in “Program Files”!! Major FUBAR: The text says to modify hbase-defaults.xml in the conf/ directory, but the file does not exist. Another source says that hbase-defaults should NOT be modified, but instead hbase-site.xml is the one to be modified. This makes sense; in fact, it makes sense that in the current release of HBase that they TOOK AWAY the defaults.xml, since the user is NOT supposed to edit it anyway. Correct instructions (which have not been applied to the “Windows” instructions) appear to be here: https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fhbase.apache.org%2fbook.html&data=01%7c01%7cxiliang%40064d.mgd.microsoft.com%7cf86db4862d0e467a9e0c08d3c58018ed%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=d9j%2bYlc5aYSiRaQ%2fY4oSoHKtj1Wko%2bsXow2SHRMxXdA%3d ================= Picking up on next day (2015-05-26) -- going back to top of Windows-install instructions and doing all steps. Roadblock: Got to the step for configuration of SSH (even though it seems to not be germane to stand-alone usage): On the bullet point where it says to modify rights-settings on certain files, “passwd” and “group” files DO NOT EXIST!! NOTE that these files DID come into existence after I ran the ssh-host-config script below -- so these steps appear to be out of order!! Shortly after that problem, this ambiguity came up: Instruction sez: Edit the /etc/hosts.allow file using your favorite editor (why not VI in the shell!) and make sure the following two lines are in there before the PARANOID line: ALL : localhost 127.0.0.1/32 : allow ALL : [::1]/128 : allow And my hosts.allow file has the single line: ALL : localhost 127.0.0.1/32 [::1]/128 [::fff:127.0.0.1]/128 : allow I’m gonna presume that this one line covers the need for the two lines in the instructions. Ran the ssh-host-config script, following the vague instructions given. NOTE that cyg_server password is ******** At end of script, message said: To start the [new] service now, call ‘net start sshd’ or ‘cygrunsrv --S sshd’. I suppose that, in this context, “call” simply means -- enter on the command line?? OKAY: ABANDONED NETWORKED MODE -- GOING LOCAL ONLY Will avoid using anything other than local mode on Windows, as it does not appear to be fully supported. (Wait for virtual Ubuntu, after I get new Windows box.) After much searching, found help with missing winutils.exe file. Somebody made them available on GitHub project: https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fzutai.blogspot.com%2f2014%2f06%2fbuild-install-and-run-hadoop-24-240-on.html&data=01%7c01%7cxiliang%40064d.mgd.microsoft.com%7cf86db4862d0e467a9e0c08d3c58018ed%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=RFaCWsTS%2bTcfzbahRRMkWbF%2fuwoJlfnsztISwwrB8K0%3d The “fix” also required making modes to hbase settings script outlined here: https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fstackoverflow.com%2fquestions%2f23136092%2fhadoophbase-cluster-on-windows-winutils-not-found&data=01%7c01%7cxiliang%40064d.mgd.microsoft.com%7cf86db4862d0e467a9e0c08d3c58018ed%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=iA4Oyg7XGFHqRU2ZTC4CmfTzJaQMxNozZp%2b7CODyjms%3d Now I can start and stop HBase in a Cygwin shell with only this type of error/warning: “cygpath: can’t convert empty path” -- hopefully this is just informational and not deadly; however, it certainly suggests Windows environment is not anywhere near to “robust” for HBase. Now I try to create a table in HBase, and get the following error: ERROR: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. Looking this up leads me to stackoverflow entry which suggests mismatch between HBase and Hadoop versions!! CANNOT GET PAST ZooKeeper error (same prob in both v0.98 and v1.1). Abandoning this until I can install/run something in Ubuntu. On Tue, Aug 16, 2016 at 10:53 AM, Xiaochun Liang <xiaochun.li...@skype.net> wrote: > Thanks Dan! > > Looks like it is a little bit harder to setup HBase on Windows with > Cygwin. Thanks a lot for your suggestions. Looks like I still need to > setup HBase with Cygwin. Could you please share the problems you met? > > Regards, > xiaochun > > -----Original Message----- > From: Daniel Vimont [mailto:dan...@commonvox.org] > Sent: Monday, August 15, 2016 11:16 AM > To: user@hbase.apache.org > Subject: Re: Fully distributed mode on Windows > > FWIW, I tried quite some time ago to set up HBase on Windows with > CygWin (just to see if I could do it), and I ran into so many problems > that I simply quit trying after a few hours. > > However, I DO use a Windows box to do all of my HBase-contributor work > on (and I do it in the way that Mike has already recommended to you): > I do that by running VirtualBox with an instance of Ubuntu running > inside of it, and then just do straightforward installation of HBase > in that virtual Ubuntu environment. If you must use a Windows box (or > multiple Windows workstations), you might want to consider a setup > similar to this to avoid what I would anticipate to be a mountain of > frustrations. > > Good luck! > > Dan > > On Sun, Aug 14, 2016 at 9:21 AM, Xiaochun Liang > <xiaochun.li...@skype.net> > wrote: > > > Thanks Mike! Is it possible to setup fully distributed mode of HBase > > on Windows with Cygwin? > > > > Regards, > > xiaochun > > > > -----Original Message----- > > From: Mike Thomsen [mailto:mikerthom...@gmail.com] > > Sent: Saturday, August 13, 2016 8:03 PM > > To: user@hbase.apache.org > > Subject: Re: Fully distributed mode on Windows > > > > You probably shouldn't be using Windows servers for this. For > > example, Cloudera doesn't list any version of Windows at all as a supported > > OS: > > > > https://na01.safelinks.protection.outlook.com/?url= > > http%3a%2f%2fwww.cloudera.com%2fdocumentation%2fcdh%2f5-0-x% > > 2fCDH5-Requirements-and-Supported-Versions%2fcdhrsv_os.html&data=01% > > 7c > > 01% > > 7cxiliang%40064d.mgd.microsoft.com%7c2935de2adccc4f60232508d3c371ce3 > > f% 7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=glqVHjC8sWJTYA% > > 2f2P9nlD63U5sdyQDLqrU%2boMzT%2bN5s%3d > > > > You'd almost certainly be better off setting up a few Linux VMs on > > those servers and running it that way. > > > > On Fri, Aug 12, 2016 at 10:36 PM, Xiaochun Liang > > <xiaochun.li...@skype.net > > > > > wrote: > > > > > Hi, > > > > > > Does Hbase support fully distributed mode on Windows? > > > > > > I am now trying to setup Hbase on Windows servers. I followed the > > > steps in > > > https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fh > > > ba se .apache.org%2fcygwin.html&data=01%7c01%7cxiliang%40064d. > > mgd.microsoft.com%7c2935de2adccc4f60232508d3c371ce3f% > > 7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=wcNiVI9Tvi6v%2b% > > 2fMm3l5d9p9hgS3UXdEmXPBrCxFHI5Q%3d, and can launch a standalone > > Hbase successfully on Windows. Next step I want to setup Hbase on > > Windows servers with distributed mode. However I saw followings in > 'start-hbase.cmd': > > > > > > "if "%distMode%"=="true" ( > > > @echo This is not implemented yet. Stay tuned." > > > > > > I am wondering if distributed mode of Hbase is supported on > > > Windows or > > not. > > > > > > Anyone has experience to setup fully-distributed Hbase on Windows? > > > > > > Regards, > > > xiaochun > > > > > >
