RE: Spark with HBase Error - Py4JJavaError
Hi Ram, Thanks very much it worked. Puneet From: ram kumar [mailto:ramkumarro...@gmail.com] Sent: Thursday, July 07, 2016 6:51 PM To: Puneet Tripathi Cc: user@spark.apache.org Subject: Re: Spark with HBase Error - Py4JJavaError Hi Puneet, Have you tried appending --jars $SPARK_HOME/lib/spark-examples-*.jar to the execution command? Ram On Thu, Jul 7, 2016 at 5:19 PM, Puneet Tripathi <puneet.tripa...@dunnhumby.com<mailto:puneet.tripa...@dunnhumby.com>> wrote: Guys, Please can anyone help on the issue below? Puneet From: Puneet Tripathi [mailto:puneet.tripa...@dunnhumby.com<mailto:puneet.tripa...@dunnhumby.com>] Sent: Thursday, July 07, 2016 12:42 PM To: user@spark.apache.org<mailto:user@spark.apache.org> Subject: Spark with HBase Error - Py4JJavaError Hi, We are running Hbase in fully distributed mode. I tried to connect to Hbase via pyspark and then write to hbase using saveAsNewAPIHadoopDataset , but it failed the error says: Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.saveAsHadoopDataset. : java.lang.ClassNotFoundException: org.apache.spark.examples.pythonconverters.StringToImmutableBytesWritableConverter at java.net.URLClassLoader$1.run(URLClassLoader.java:366) I have been able to create pythonconverters.jar and then did below: 1. I think we have to copy this to a location on HDFS, /sparkjars/ seems a good a directory to create as any. I think the file has to be world readable 2. Set the spark_jar_hdfs_path property in Cloudera Manager e.g. hdfs:///sparkjars It still doesn’t seem to work can someone please help me with this. Regards, Puneet dunnhumby limited is a limited company registered in England and Wales with registered number 02388853 and VAT registered number 927 5871 83. Our registered office is at Aurora House, 71-75 Uxbridge Road, London W5 5SL. The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender. dunnhumby may monitor and record all emails. The views expressed in this email are those of the sender and not those of dunnhumby. dunnhumby limited is a limited company registered in England and Wales with registered number 02388853 and VAT registered number 927 5871 83. Our registered office is at Aurora House, 71-75 Uxbridge Road, London W5 5SL. The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender. dunnhumby may monitor and record all emails. The views expressed in this email are those of the sender and not those of dunnhumby. dunnhumby limited is a limited company registered in England and Wales with registered number 02388853 and VAT registered number 927 5871 83. Our registered office is at Aurora House, 71-75 Uxbridge Road, London W5 5SL. The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender. dunnhumby may monitor and record all emails. The views expressed in this email are those of the sender and not those of dunnhumby.
Re: Spark with HBase Error - Py4JJavaError
Hi Puneet, Have you tried appending --jars $SPARK_HOME/lib/spark-examples-*.jar to the execution command? Ram On Thu, Jul 7, 2016 at 5:19 PM, Puneet Tripathi < puneet.tripa...@dunnhumby.com> wrote: > Guys, Please can anyone help on the issue below? > > > > Puneet > > > > *From:* Puneet Tripathi [mailto:puneet.tripa...@dunnhumby.com] > *Sent:* Thursday, July 07, 2016 12:42 PM > *To:* user@spark.apache.org > *Subject:* Spark with HBase Error - Py4JJavaError > > > > Hi, > > > > We are running Hbase in fully distributed mode. I tried to connect to > Hbase via pyspark and then write to hbase using *saveAsNewAPIHadoopDataset > *, but it failed the error says: > > > > Py4JJavaError: An error occurred while calling > z:org.apache.spark.api.python.PythonRDD.saveAsHadoopDataset. > > : java.lang.ClassNotFoundException: > org.apache.spark.examples.pythonconverters.StringToImmutableBytesWritableConverter > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > > I have been able to create pythonconverters.jar and then did below: > > > > 1. I think we have to copy this to a location on HDFS, /sparkjars/ > seems a good a directory to create as any. I think the file has to be world > readable > > 2. Set the spark_jar_hdfs_path property in Cloudera Manager e.g. > hdfs:///sparkjars > > > > It still doesn’t seem to work can someone please help me with this. > > > > Regards, > > Puneet > > dunnhumby limited is a limited company registered in England and Wales > with registered number 02388853 and VAT registered number 927 5871 83. Our > registered office is at Aurora House, 71-75 Uxbridge Road, London W5 5SL. > The contents of this message and any attachments to it are confidential and > may be legally privileged. If you have received this message in error you > should delete it from your system immediately and advise the sender. > dunnhumby may monitor and record all emails. The views expressed in this > email are those of the sender and not those of dunnhumby. > dunnhumby limited is a limited company registered in England and Wales > with registered number 02388853 and VAT registered number 927 5871 83. Our > registered office is at Aurora House, 71-75 Uxbridge Road, London W5 5SL. > The contents of this message and any attachments to it are confidential and > may be legally privileged. If you have received this message in error you > should delete it from your system immediately and advise the sender. > dunnhumby may monitor and record all emails. The views expressed in this > email are those of the sender and not those of dunnhumby. >
RE: Spark with HBase Error - Py4JJavaError
Guys, Please can anyone help on the issue below? Puneet From: Puneet Tripathi [mailto:puneet.tripa...@dunnhumby.com] Sent: Thursday, July 07, 2016 12:42 PM To: user@spark.apache.org Subject: Spark with HBase Error - Py4JJavaError Hi, We are running Hbase in fully distributed mode. I tried to connect to Hbase via pyspark and then write to hbase using saveAsNewAPIHadoopDataset , but it failed the error says: Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.saveAsHadoopDataset. : java.lang.ClassNotFoundException: org.apache.spark.examples.pythonconverters.StringToImmutableBytesWritableConverter at java.net.URLClassLoader$1.run(URLClassLoader.java:366) I have been able to create pythonconverters.jar and then did below: 1. I think we have to copy this to a location on HDFS, /sparkjars/ seems a good a directory to create as any. I think the file has to be world readable 2. Set the spark_jar_hdfs_path property in Cloudera Manager e.g. hdfs:///sparkjars It still doesn't seem to work can someone please help me with this. Regards, Puneet dunnhumby limited is a limited company registered in England and Wales with registered number 02388853 and VAT registered number 927 5871 83. Our registered office is at Aurora House, 71-75 Uxbridge Road, London W5 5SL. The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender. dunnhumby may monitor and record all emails. The views expressed in this email are those of the sender and not those of dunnhumby. dunnhumby limited is a limited company registered in England and Wales with registered number 02388853 and VAT registered number 927 5871 83. Our registered office is at Aurora House, 71-75 Uxbridge Road, London W5 5SL. The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender. dunnhumby may monitor and record all emails. The views expressed in this email are those of the sender and not those of dunnhumby.
Spark with HBase Error - Py4JJavaError
Hi, We are running Hbase in fully distributed mode. I tried to connect to Hbase via pyspark and then write to hbase using saveAsNewAPIHadoopDataset , but it failed the error says: Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.saveAsHadoopDataset. : java.lang.ClassNotFoundException: org.apache.spark.examples.pythonconverters.StringToImmutableBytesWritableConverter at java.net.URLClassLoader$1.run(URLClassLoader.java:366) I have been able to create pythonconverters.jar and then did below: 1. I think we have to copy this to a location on HDFS, /sparkjars/ seems a good a directory to create as any. I think the file has to be world readable 2. Set the spark_jar_hdfs_path property in Cloudera Manager e.g. hdfs:///sparkjars It still doesn't seem to work can someone please help me with this. Regards, Puneet dunnhumby limited is a limited company registered in England and Wales with registered number 02388853 and VAT registered number 927 5871 83. Our registered office is at Aurora House, 71-75 Uxbridge Road, London W5 5SL. The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender. dunnhumby may monitor and record all emails. The views expressed in this email are those of the sender and not those of dunnhumby.
