Please disregard.. I think I have it successfully running when I switched to Ubuntu. I ran into a lot of funky issues when running it on Windows, as shown in my previous email.
Joo Wan From: Joo Wan Ro <joo...@microsoft.com> Sent: Monday, February 11, 2019 11:31 AM To: u...@hive.apache.org; dev@hive.apache.org Cc: Karthick Krishnamoorthy <karth...@microsoft.com>; Danny Navon <dan...@microsoft.com> Subject: RE: Setting up a Standalone Hive Metastore [Resending because I needed to add myself to the mailing lists] Hello Hive Experts, I am a software engineer at Microsoft, and I am having trouble trying to run a standalone Hive metastore service on my Windows 10 machine. Your assistance would be greatly appreciated. [0] Github project here (running branch "3.1"): link<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Ftree%2Fmaster%2Fstandalone-metastore&data=02%7C01%7CJoo.Ro%40microsoft.com%7C50b73fc0eec141f768ee08d6905774f8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636855102626401844&sdata=Dm7hvXGBrA2zY52ZsPh%2BSprASTZS7X5I8Aln7Mrl9zE%3D&reserved=0> [1] Related documentation here: link<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FHive%2FAdminManual%2BMetastore%2B3.0%2BAdministration&data=02%7C01%7CJoo.Ro%40microsoft.com%7C50b73fc0eec141f768ee08d6905774f8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636855102626411849&sdata=0En%2FwuSXsR5cniRjw4zofNOh5qHv%2BPVIkZ5r8Tf79GY%3D&reserved=0> [2] Path to setup scripts: ($HOME\hive\standalone-metastore\src\main\scripts) I was able to build it successfully using Maven but having trouble running the jar. Here are my questions: 1. I believe this standalone metastore project is relatively new, as it was introduced in Hive 3.0. When was the stable version of the standalone metastore released? 1. While the documentation link [1] above helps with the config setup, is there a resource for step-by-step guidance on how to bootstrap the standalone metastore service on a Windows 10 or Linux machine? Without it, I've been trying to reverse engineer on how to get it running. * For starters, it seems that the scripts under the directory above [2] are not Windows 10 friendly because of carriage returns. And having to using Cygwin confused me on which path convention to use for the environment/system variables (E:\src\hive\... vs. /cygdrive/e/src/hive/). I removed the carriage returns and used the /cygdrive/ convention to get it working partially. * I had no clue which environment/system variables I needed and if there were any dependencies, which I assumed to be none because the related documentation [1] above notes the independent nature of the standalone metastore project. However, by studying the scripts (base, start-metastore, and metastore.sh) under the path above [2], I found two things: i. The need to define METASTORE_HOME ($HOME\hive\standalone-metastore\target\apache-hive-metastore-3.1.1-bin\apache-hive-metastore-3.1.1-bin) and METASTORE_CONF_DIR environment variables ii. The need to install Hadoop as a dependency because the metastore.sh script uses it to start the metastore service; hence, installing it and then defining the HADOOP_HOME ($HOME\hadoop-3.1.1) environment variable (I also had to remove the carriage returns under $HOME\hadoop-3.1.1) iii. I have no other environment variables or dependencies other than the ones aforementioned * At this point, the metastore service began to start running; however, I ran into an exception "Failed to get schema version" - more information here: link<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity.hortonworks.com%2Fquestions%2F15136%2Forgapachehadoophivemetastorehivemetaexception-fail.html&data=02%7C01%7CJoo.Ro%40microsoft.com%7C50b73fc0eec141f768ee08d6905774f8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636855102626421857&sdata=NnMEdKCjnvKvystqbCJIj%2BpfaBET0JtikbjoLVX8aS4%3D&reserved=0>. I believe this is because the default derby database was not initialized. * So, using the schematool script under my apache-hive-metastore-3.1.1-bin directory, I ran schematool --dbType derby -initSchema. Then I ran into an exception "Unknown version specified for initialization: 3.1.0" - the exception is thrown here link<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fblob%2Fe867d1c693e966706d3b7c6fe18e039a85928f51%2Fstandalone-metastore%2Fmetastore-server%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhive%2Fmetastore%2FMetaStoreSchemaInfo.java%23L137&data=02%7C01%7CJoo.Ro%40microsoft.com%7C50b73fc0eec141f768ee08d6905774f8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636855102626421857&sdata=ngjbv73pAqPGeM5F7OO%2FeXd9jzpPbXnjUMkcKGl550M%3D&reserved=0>. It cannot find the derby schema script, but I confirmed that it is there ($HOME\hive\standalone-metastore\target\apache-hive-metastore-3.1.1-bin\apache-hive-metastore-3.1.1-bin\scripts\metastore\upgrade\derby\hive-schema-3.1.0.derby.sql). This led me to believe that there was a conflict again with file path conventions between Windows "\" and Linux "/" and I have faced a dead end. For the time being, I am redirecting my efforts on setting up a Linux machine to see if I would have a smoother experience, but any help for my concerns/issues above would be greatly appreciated. Thank you! Joo Wan
