Haaa Devin! Thankfully I can leave that firewall to the pros. I'm an analytics folk, whose set up a microCluster at home to learn Hadoop/Hive/Sqoop/db-ETL/cube integration on. And gosh does it works great now! Best investment in learning I've spent all year - and it's fun to shut down a node and watch it keep ticking!
I got the new hardware for a great price, SQL 2014 CTP was free (exp New Year's Eve), Hadoop free, Hive free, Sqoop free & Debian free! Gosh is the world changing or what?! Software is so open:) If you or anyone on this thread have or recommend any RDBMS-Hadoop-Hive-Sqoop blogs, websites, books or whatnot, lemme know. I'm one'a those who has massive amounts of data to analyze (like many of ya) and am a SAS-SQL diehard. Is there any "Kimbal" or "Inmon" of this Hadoop movement I would be sad to not be reading up on? I read Doug Cutting's blog and follow his tweets and all the big vendors I read up on (Cloudera, Horton, MapR, Pivotal, etc...). Anything I'm missing? I am joining a local Hadoop group in my community (Chicago), so I've the local network part covered. I've a nice weekend ahead of me and figure now that my cluster is tip-top, it's time to curl up to a good book, white paper, YouTube/podcast or whatnot. Best, Andy > On Nov 1, 2013, at 8:44 AM, "DSuiter RDX" <[email protected]> wrote: > > Good work! Sometimes it's a simple thing...case sensitivity and the like, or > an extra leading space...glad you were able to track it down. > > Now, if you want another puzzle for later, try putting a firewall in between > MSSQL server and your Hadoop cluster. :-) > > > Devin Suiter > Jr. Data Solutions Software Engineer > > 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212 > Google Voice: 412-256-8556 | www.rdx.com > > >> On Fri, Nov 1, 2013 at 8:01 AM, Andrew Allaway <[email protected]> >> wrote: >> Update - >> >> All's fixed, my TCP/IP port 1433 and UDP 1434 were not open /*blushing*/ >> >> Live'n learn:) >> >> Thanks all for help triaging my question! >> Andy >> >> >> >>> On Oct 30, 2013, at 10:32 PM, "Andrew Allaway" <[email protected]> >>> wrote: >>> >>> Hi Jarcec, et al - >>> >>> Yes, that is rather vague of me. Apologizes, I appreciate your question >>> and aid:) >>> >>> I've some updates, but still getting thrown for a loop by Hadoop-Sqoop:) >>> /* I can't resist 0> <--not'a bad ice-cream-cone :) */ >>> >>> * I was able to uninstall the sqoop 1.4.4 package and re-install it with >>> the version that says sqoop-bin.1.4.4. That worked magically! Woohoo. I >>> can now run sqoop! What was the difference of the stable release with bin >>> vs w/o bin? >>> >>> * Now the down side, I cant, for the life of me, get sqoop to connect to my >>> SQL Server. I've the specific jar file to run sqoop for sql server at >>> nameNode: ...sqoop/lib/sqljdbc4.jar >>> >>> Questions: >>> 1) When I submit the below from .../sqoop/, I get nowhere:) advice? >>> >>> bin/sqoop list-databases --connect >>> 'jdbc:sqlserver://Andy-ABC-1-HP\BOB:1433;databaseName=andy_dev;user=Andy-ABC-1;password=***;' >>> >>> What I think is the relevant part of error: >>> ... >>> >>> 13/10/30 06:20:29 ERROR manager.CatalogQueryManager: Failed to list >>> databases >>> com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to >>> the host Andy-ABC-1-HP, port 1433 has failed. Error: "null. Verify the >>> connection properties. Make sure that an instance of SQL Server is running >>> on the host and accepting TCP/IP connections at the port. Make sure that >>> TCP connections to the port are not blocked by a firewall.". >>> at >>> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:190) >>> at >>> com.microsoft.sqlserver.jdbc.SQLServerException.ConvertConnectExceptionToSQLServerException(SQLServerException.java:241) >>> .. >>> >>> What I've got: >>> >>> a) >>> My HP Laptop, set to static: IP4 - 192.168.10.13 >>> SQLServer running on the above laptop: >>> Host name: Andy-ABC-1-HP >>> Instance: BOB >>> Port (per: start>sql server config. mgr>SQL native client>client >>> protocols>tcp/ip>properties>port 1433 >>> DB: andy_dev >>> schema: dbo >>> Login: Andy-ABC-1 >>> Pw : >>> >>> b) namenode, 192.168.10.10, Debian, can ping 192.168.10.13 and >>> 192.168.10.13 can ssh into namenode >>> >>> 2) When I lauch sqoop, it keeps saying "Error: /usr/lib/hadoop does not >>> exist! Please set $HADOOP_COMMON_HOME to the root of your Hadoop >>> installation. & Please set $HADOOP_MAPRED_HOME to the root of your Hadoop >>> MapReduce installation." I then run: export export >>> HADOOP_COMMON_HOME=/usr/local/hadoop and export >>> HADOOP_MAPRED_HOME=/usr/local/hadoop. It works but after reboot it is >>> back??? How to perm set it? >>> >>> Firewall was off when all above run... >>> >>> Any advice appreciated! >>> >>> Thanks! >>> Andy >>> >>> > Date: Wed, 30 Oct 2013 14:28:52 -0700 >>> > From: [email protected] >>> > To: [email protected] >>> > Subject: Re: Sqoop, sending for a loop - newby lost - SQL Server/Sqoop >>> > >>> > Hi Andrew, >>> > would you mind sharing with use exact commands and exact exceptions that >>> > you are seeing? It will help us to understand your issue better. >>> > >>> > Jarcec >>> > >>> > On Mon, Oct 28, 2013 at 07:29:47PM -0500, Andrew Allaway wrote: >>> > > Hey Abe - >>> > > >>> > > I haven't because I'm working on running a real lean install on a ARM >>> > > cluster. >>> > > >>> > > I want a bare bones build (Hadoop, Hive & Scoop). Starting to feel that >>> > > although these packages are open source (Hadoop, Hive, Sqoop, etc), >>> > > figuring out how to build a solution from the bottom, w/o a full >>> > > package (Bigtop, Cloudera, MapR, Horton, et al), is quite hard, due to >>> > > not knowing what dependencies are required. >>> > > >>> > > The Doc's of Sqoop 1.4.4 don't mention Hbase being required. What am I >>> > > missing? >>> > > >>> > > Anyone tried running a barebones setup and know what I'm missing? >>> > > >>> > > Thanks Abe for the tip, I've your distro on a VM and it has all the >>> > > bells and whistles, just was hoping to get a three node solution >>> > > running swell with the barebones, just can't figure out what base >>> > > packages are needed to tie SS 2012/14 and a cluster together with only >>> > > Hadoop, Hive and Sqoop. >>> > > >>> > > Also I forgot to mention, when I run sqoop help it also says something >>> > > about missing a Java class. I've OpenJDK running, pointing to ARM (i.e. >>> > > export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-armhf) >>> > > >>> > > Thanks to all in advance for your suggestions! >>> > > >>> > > >>> > > Andy >>> > > >>> > > >>> > > >>> > > >>> > > > On Oct 28, 2013, at 6:20 PM, "Abraham Elmahrek" <[email protected]> >>> > > > wrote: >>> > > > >>> > > > Andy, >>> > > > >>> > > > Have you tried installing using Apache Bigtop? or some other packaged >>> > > > installation provider? Hbase client libs are used for Hbase import. >>> > > > Sqoop is compiled with Hbase support I think. >>> > > > >>> > > > -Abe >>> > > > >>> > > > >>> > > >> On Mon, Oct 28, 2013 at 4:04 PM, Andrew Allaway >>> > > >> <[email protected]> wrote: >>> > > >> Sorry for the bad title:) >>> > > >> >>> > > >> Have: >>> > > >> 3 nodes >>> > > >> Debian/wheezy >>> > > >> Hadoop 1.2.1 >>> > > >> Hive 0.11.0 >>> > > >> >>> > > >> All's working great:) >>> > > >> >>> > > >> Want to connect SQL Server 2012 and SQL Serv. 2014 CTP to the above >>> > > >> >>> > > >> >>> > > >> I'm totally lost >>> > > >> >>> > > >> Namenode (aka node1): 192.168.10.10 >>> > > >> Node2 192.168.10.11 >>> > > >> Node3 192.168.10.12 >>> > > >> >>> > > >> Have Windows7 (static ip4 192.168.10.13), connected via ethernet >>> > > >> thru a switch. I can ssh into nodes 1-3 easy. >>> > > >> >>> > > >> All's swell. >>> > > >> >>> > > >> On Win7 have a full sql server instance "bob", database "test_db", >>> > > >> schema "test_schema" & table "test_table" login "abc" pw "xyz". >>> > > >> >>> > > >> On the cluster I've hadoop here: >>> > > >> /usr/local/hadoop >>> > > >> >>> > > >> Just untared Scoop to /usr/lib/sqoop >>> > > >> >>> > > >> Then when I tried to run$ sqoop help from the above dir, it said it >>> > > >> didn't know where my hadoop was. So I ran the hadoop_home >>> > > >> /usr/local.... >>> > > >> >>> > > >> Then ran$ sqoop help and it said it can't find hdfs. So I ran the >>> > > >> same$ export home_hdfs usr/local.... >>> > > >> >>> > > >> Then ran sqoop help and it said it needs Hbase???? >>> > > >> >>> > > >> Does it? Why does it need Hbase to run? >>> > > >> >>> > > >> Not sure how to go from here. I want to install these packages as I >>> > > >> learn them. I don't intend to learn Hbase at the moment, can I >>> > > >> "live" w/o it? >>> > > >> >>> > > >> Even if sqoop worked I still don't understand how to pull the table >>> > > >> above (test_table) into hdfs and into Hive?? >>> > > >> >>> > > >> Thoughts? >>> > > >> >>> > > >> Best, >>> > > >> Andy >>> > > > >
