Re: spark 1.1.0 - hbase 0.98.6-hadoop2 version - py4j.protocol.Py4JJavaError java.lang.ClassNotFoundException
forgot to copy user list On Sat, Oct 4, 2014 at 3:12 PM, Nick Pentreath wrote: > what version did you put in the pom.xml? > > it does seem to be in Maven central: > http://search.maven.org/#artifactdetails%7Corg.apache.hbase%7Chbase%7C0.98.6-hadoop2%7Cpom > > > org.apache.hbase > hbase > 0.98.6-hadoop2 > > > Note you shouldn't need to rebuild Spark, I think just the example project > via sbt examples/assembly > > On Fri, Oct 3, 2014 at 10:55 AM, serkan.dogan > wrote: > >> Hi, >> I installed hbase-0.98.6-hadoop2. It's working not any problem with that. >> >> >> When i am try to run spark hbase python examples, (wordcount examples >> working - not python issue) >> >> ./bin/spark-submit --master local --driver-class-path >> ./examples/target/spark-examples_2.10-1.1.0.jar >> ./examples/src/main/python/hbase_inputformat.py localhost myhbasetable >> >> the process exit with ClassNotFoundException... >> >> I search lots of blogs, sites all says "spark 1.1 version built with hbase >> 0.94.6 rebuild with own hbase version". >> >> I try first, >> change hbase version number - in pom.xml -- nothing found maven central >> >> I try second, >> compile hbase from src and copy hbase/lib folder hbase jars to >> spark/lib_managed folder and edit spark-defaults.conf >> >> my spark-defaults.conf >> >> spark.executor.extraClassPath >> >> /home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-server-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-protocol-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-hadoop2-compat-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-client-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-commont-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/htrace-core-2.04.jar >> >> >> My question is how i can work with hbase 0.98.6-hadoop2 with spark 1.1.0 >> >> Here is the exception message >> >> >> Using Spark's default log4j profile: >> org/apache/spark/log4j-defaults.properties >> 14/10/03 11:27:15 WARN Utils: Your hostname, xxx.yyy.com resolves to a >> loopback address: 127.0.0.1; using 1.1.1.1 instead (on interface eth0) >> 14/10/03 11:27:15 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to >> another address >> 14/10/03 11:27:15 INFO SecurityManager: Changing view acls to: root, >> 14/10/03 11:27:15 INFO SecurityManager: Changing modify acls to: root, >> 14/10/03 11:27:15 INFO SecurityManager: SecurityManager: authentication >> disabled; ui acls disabled; users with view permissions: Set(root, ); >> users >> with modify permissions: Set(root, ) >> 14/10/03 11:27:16 INFO Slf4jLogger: Slf4jLogger started >> 14/10/03 11:27:16 INFO Remoting: Starting remoting >> 14/10/03 11:27:16 INFO Remoting: Remoting started; listening on addresses >> :[akka.tcp://sparkdri...@1-1-1-1-1.rev.mydomain.io:49256] >> 14/10/03 11:27:16 INFO Remoting: Remoting now listens on addresses: >> [akka.tcp://sparkdri...@1-1-1-1-1.rev.mydomain.io:49256] >> 14/10/03 11:27:16 INFO Utils: Successfully started service 'sparkDriver' >> on >> port 49256. >> 14/10/03 11:27:16 INFO SparkEnv: Registering MapOutputTracker >> 14/10/03 11:27:16 INFO SparkEnv: Registering BlockManagerMaster >> 14/10/03 11:27:16 INFO DiskBlockManager: Created local directory at >> /tmp/spark-local-20141003112716-298d >> 14/10/03 11:27:16 INFO Utils: Successfully started service 'Connection >> manager for block manager' on port 35106. >> 14/10/03 11:27:16 INFO ConnectionManager: Bound socket to port 35106 with >> id >> = ConnectionManagerId(1-1-1-1-1.rev.mydomain.io,35106) >> 14/10/03 11:27:16 INFO MemoryStore: MemoryStore started with capacity >> 267.3 >> MB >> 14/10/03 11:27:16 INFO BlockManagerMaster: Trying to register BlockManager >> 14/10/03 11:27:16 INFO BlockManagerMasterActor: Registering block manager >> 1-1-1-1-1.rev.mydomain.io:35106 with 267.3 MB RAM >> 14/10/03 11:27:16 INFO BlockManagerMaster: Registered BlockManager >> 14/10/03 11:27:16 INFO HttpFileServer: HTTP File server directory is >> /tmp/spark-f60b0533-998f-4af2-a208-d04c571eab82 >> 14/10/03 11:27:16 INFO HttpServer: Starting HTTP Server >> 14/10/03 11:27:16 INFO Utils: Successfully started service 'HTTP file >> server' on port 49611. >> 14/10/03 11:27:16 INFO Utils: Successfully started service 'SparkUI' on >> port >> 4040. >> 14/10/03 11:27:16 INFO SparkUI: Started SparkUI at >> http://1-1-1-1-1.rev.mydomain.io:4040 >> 14/10/03 11:27:16 INFO Utils: Copying >> >> /home/downloads/spark/spark-1.1.0/./examples/src/main/python/hbase_inputformat.py >> to /tmp/spark-7232227a-0547-454e-9f68-805fa7b0c2f0/hbase_inputformat.py >> 14/10/03 11:27:16 INFO SparkContext: Added file >> >> file:/home/downloads/spark/spark-1.1.0/./examples/src/main/python/hbase_inputformat.py >> at http://1.1.1.1:49611/files/hbase_inputformat.py with timestamp >> 1412324836837 >> 14/10/03 11:27:16 INFO AkkaUtils: Connecting to HeartbeatReceiver: >> akka.tcp:// >> spa
spark 1.1.0 - hbase 0.98.6-hadoop2 version - py4j.protocol.Py4JJavaError java.lang.ClassNotFoundException
Hi, I installed hbase-0.98.6-hadoop2. It's working not any problem with that. When i am try to run spark hbase python examples, (wordcount examples working - not python issue) ./bin/spark-submit --master local --driver-class-path ./examples/target/spark-examples_2.10-1.1.0.jar ./examples/src/main/python/hbase_inputformat.py localhost myhbasetable the process exit with ClassNotFoundException... I search lots of blogs, sites all says "spark 1.1 version built with hbase 0.94.6 rebuild with own hbase version". I try first, change hbase version number - in pom.xml -- nothing found maven central I try second, compile hbase from src and copy hbase/lib folder hbase jars to spark/lib_managed folder and edit spark-defaults.conf my spark-defaults.conf spark.executor.extraClassPath /home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-server-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-protocol-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-hadoop2-compat-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-client-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-commont-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/htrace-core-2.04.jar My question is how i can work with hbase 0.98.6-hadoop2 with spark 1.1.0 Here is the exception message Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 14/10/03 11:27:15 WARN Utils: Your hostname, xxx.yyy.com resolves to a loopback address: 127.0.0.1; using 1.1.1.1 instead (on interface eth0) 14/10/03 11:27:15 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 14/10/03 11:27:15 INFO SecurityManager: Changing view acls to: root, 14/10/03 11:27:15 INFO SecurityManager: Changing modify acls to: root, 14/10/03 11:27:15 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, ); users with modify permissions: Set(root, ) 14/10/03 11:27:16 INFO Slf4jLogger: Slf4jLogger started 14/10/03 11:27:16 INFO Remoting: Starting remoting 14/10/03 11:27:16 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkdri...@1-1-1-1-1.rev.mydomain.io:49256] 14/10/03 11:27:16 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkdri...@1-1-1-1-1.rev.mydomain.io:49256] 14/10/03 11:27:16 INFO Utils: Successfully started service 'sparkDriver' on port 49256. 14/10/03 11:27:16 INFO SparkEnv: Registering MapOutputTracker 14/10/03 11:27:16 INFO SparkEnv: Registering BlockManagerMaster 14/10/03 11:27:16 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20141003112716-298d 14/10/03 11:27:16 INFO Utils: Successfully started service 'Connection manager for block manager' on port 35106. 14/10/03 11:27:16 INFO ConnectionManager: Bound socket to port 35106 with id = ConnectionManagerId(1-1-1-1-1.rev.mydomain.io,35106) 14/10/03 11:27:16 INFO MemoryStore: MemoryStore started with capacity 267.3 MB 14/10/03 11:27:16 INFO BlockManagerMaster: Trying to register BlockManager 14/10/03 11:27:16 INFO BlockManagerMasterActor: Registering block manager 1-1-1-1-1.rev.mydomain.io:35106 with 267.3 MB RAM 14/10/03 11:27:16 INFO BlockManagerMaster: Registered BlockManager 14/10/03 11:27:16 INFO HttpFileServer: HTTP File server directory is /tmp/spark-f60b0533-998f-4af2-a208-d04c571eab82 14/10/03 11:27:16 INFO HttpServer: Starting HTTP Server 14/10/03 11:27:16 INFO Utils: Successfully started service 'HTTP file server' on port 49611. 14/10/03 11:27:16 INFO Utils: Successfully started service 'SparkUI' on port 4040. 14/10/03 11:27:16 INFO SparkUI: Started SparkUI at http://1-1-1-1-1.rev.mydomain.io:4040 14/10/03 11:27:16 INFO Utils: Copying /home/downloads/spark/spark-1.1.0/./examples/src/main/python/hbase_inputformat.py to /tmp/spark-7232227a-0547-454e-9f68-805fa7b0c2f0/hbase_inputformat.py 14/10/03 11:27:16 INFO SparkContext: Added file file:/home/downloads/spark/spark-1.1.0/./examples/src/main/python/hbase_inputformat.py at http://1.1.1.1:49611/files/hbase_inputformat.py with timestamp 1412324836837 14/10/03 11:27:16 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkdri...@1-1-1-1-1.rev.mydomain.io:49256/user/HeartbeatReceiver Traceback (most recent call last): File "/home/downloads/spark/spark-1.1.0/./examples/src/main/python/hbase_inputformat.py", line 70, in conf=conf) File "/home/downloads/spark/spark-1.1.0/python/pyspark/context.py", line 471, in newAPIHadoopRDD jconf, batchSize) File "/usr/lib/python2.6/site-packages/py4j-0.8.2.1-py2.6.egg/py4j/java_gateway.py", line 538, in __call__ self.target_id, self.name) File "/usr/lib/python2.6/site-packages/py4j-0.8.2.1-py2.6.egg/py4j/protocol.py", line 300, in get_return_value format(target_id, '.', name), value) py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.newAPIHadoopRDD. : java.l
spark 1.1.0 - hbase 0.98.6-hadoop2 version - py4j.protocol.Py4JJavaError java.lang.ClassNotFoundException
Hi, I installed hbase-0.98.6-hadoop2. It's working not any problem with that. When i am try to run spark hbase python examples, (wordcount examples working - not python issue) ./bin/spark-submit --master local --driver-class-path ./examples/target/spark-examples_2.10-1.1.0.jar ./examples/src/main/python/hbase_inputformat.py localhost myhbasetable the process exit with ClassNotFoundException... I search lots of blogs, sites all says "spark 1.1 version built with hbase 0.94.6 rebuild with own hbase version". I try first, change hbase version number - in pom.xml -- nothing found maven central I try second, compile hbase from src and copy hbase/lib folder hbase jars to spark/lib_managed folder and edit spark-defaults.conf my spark-defaults.conf spark.executor.extraClassPath /home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-server-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-protocol-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-hadoop2-compat-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-client-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-commont-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/htrace-core-2.04.jar My question is how i can work with hbase 0.98.6-hadoop2 with spark 1.1.0 Here is the exception message Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 14/10/03 11:27:15 WARN Utils: Your hostname, testmachine1.tracescore.com resolves to a loopback address: 127.0.0.1; using 1.1.1.1 instead (on interface eth0) 14/10/03 11:27:15 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 14/10/03 11:27:15 INFO SecurityManager: Changing view acls to: root, 14/10/03 11:27:15 INFO SecurityManager: Changing modify acls to: root, 14/10/03 11:27:15 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, ); users with modify permissions: Set(root, ) 14/10/03 11:27:16 INFO Slf4jLogger: Slf4jLogger started 14/10/03 11:27:16 INFO Remoting: Starting remoting 14/10/03 11:27:16 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkdri...@1-1-1-1-1.rev.mydomain.io:49256] 14/10/03 11:27:16 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkdri...@1-1-1-1-1.rev.mydomain.io:49256] 14/10/03 11:27:16 INFO Utils: Successfully started service 'sparkDriver' on port 49256. 14/10/03 11:27:16 INFO SparkEnv: Registering MapOutputTracker 14/10/03 11:27:16 INFO SparkEnv: Registering BlockManagerMaster 14/10/03 11:27:16 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20141003112716-298d 14/10/03 11:27:16 INFO Utils: Successfully started service 'Connection manager for block manager' on port 35106. 14/10/03 11:27:16 INFO ConnectionManager: Bound socket to port 35106 with id = ConnectionManagerId(1-1-1-1-1.rev.mydomain.io,35106) 14/10/03 11:27:16 INFO MemoryStore: MemoryStore started with capacity 267.3 MB 14/10/03 11:27:16 INFO BlockManagerMaster: Trying to register BlockManager 14/10/03 11:27:16 INFO BlockManagerMasterActor: Registering block manager 1-1-1-1-1.rev.mydomain.io:35106 with 267.3 MB RAM 14/10/03 11:27:16 INFO BlockManagerMaster: Registered BlockManager 14/10/03 11:27:16 INFO HttpFileServer: HTTP File server directory is /tmp/spark-f60b0533-998f-4af2-a208-d04c571eab82 14/10/03 11:27:16 INFO HttpServer: Starting HTTP Server 14/10/03 11:27:16 INFO Utils: Successfully started service 'HTTP file server' on port 49611. 14/10/03 11:27:16 INFO Utils: Successfully started service 'SparkUI' on port 4040. 14/10/03 11:27:16 INFO SparkUI: Started SparkUI at http://1-1-1-1-1.rev.mydomain.io:4040 14/10/03 11:27:16 INFO Utils: Copying /home/downloads/spark/spark-1.1.0/./examples/src/main/python/hbase_inputformat.py to /tmp/spark-7232227a-0547-454e-9f68-805fa7b0c2f0/hbase_inputformat.py 14/10/03 11:27:16 INFO SparkContext: Added file file:/home/downloads/spark/spark-1.1.0/./examples/src/main/python/hbase_inputformat.py at http://1.1.1.1:49611/files/hbase_inputformat.py with timestamp 1412324836837 14/10/03 11:27:16 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkdri...@1-1-1-1-1.rev.mydomain.io:49256/user/HeartbeatReceiver Traceback (most recent call last): File "/home/downloads/spark/spark-1.1.0/./examples/src/main/python/hbase_inputformat.py", line 70, in conf=conf) File "/home/downloads/spark/spark-1.1.0/python/pyspark/context.py", line 471, in newAPIHadoopRDD jconf, batchSize) File "/usr/lib/python2.6/site-packages/py4j-0.8.2.1-py2.6.egg/py4j/java_gateway.py", line 538, in __call__ self.target_id, self.name) File "/usr/lib/python2.6/site-packages/py4j-0.8.2.1-py2.6.egg/py4j/protocol.py", line 300, in get_return_value format(target_id, '.', name), value) py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.newAPIHad
