Hi Josh,
I downloaded the Apache Phoenix v4.4.0-HBase-1.1
<https://github.com/apache/phoenix/releases/tag/v4.4.0-HBase-1.1> and
tried packaging it through *apache-maven-3.3.9*
When I try to build it using maven I am getting following error
maven command I used to build Phoenix mvn package -DskipTests
-Dhadoop.profile=2
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Phoenix ..................................... SUCCESS [
2.550 s]
[INFO] Phoenix Core ....................................... SUCCESS [
36.224 s]
[INFO] Phoenix - Flume .................................... SUCCESS [
2.575 s]
[INFO] Phoenix - Pig ...................................... SUCCESS [
1.809 s]
[INFO] Phoenix Query Server Client ........................ SUCCESS [
0.734 s]
[INFO] Phoenix Query Server ............................... SUCCESS [
12.427 s]
[INFO] Phoenix - Pherf .................................... SUCCESS [
13.394 s]
[INFO] Phoenix - Spark .................................... FAILURE [
18.693 s]
[INFO] Phoenix Assembly ................................... SKIPPED
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 01:28 min
[INFO] Finished at: 2016-04-19T12:44:23+08:00
[INFO] Final Memory: 73M/732M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
net.alchim31.maven:scala-maven-plugin:3.2.0:testCompile
(scala-test-compile) on project phoenix-spark: wrap:
org.apache.commons.exec.ExecuteException: Process exited with an error:
-10000 (Exit value: -10000) -> [Help 1]
[ERROR]
Could please help me finding the the issue .
Would really appreciate your help .
Thanks,
Divya
On 11 April 2016 at 02:45, Josh Mahonin <jmaho...@gmail.com> wrote:
> Hi Divya,
>
> No, there is a separate JAR that would look like
> 'phoenix-4.4.0.XXX-client-spark.jar'. If you download a binary release of
> Phoenix, or compile the latest version yourself, you will be able to see
> and use it. It does not come with the HDP 2.3.4 platform, at least last I
> checked.
>
> Regards,
>
> Josh
>
> On Sat, Apr 9, 2016 at 2:24 PM, Divya Gehlot <divya.htco...@gmail.com>
> wrote:
>
>> Hi Josh,
>> Thank you very much for your help.
>> I could see there is phoenix-spark-4.4.0.2.3.4.0-3485.jar in my
>> phoenix/lib.
>> Please confirm is the above jar you are talking about?
>>
>> Thanks,
>> Divya
>>
>> Josh Mahonin <jmahonin@
>>
>> On 9 April 2016 at 23:01, Josh Mahonin <jmaho...@gmail.com> wrote:
>>
>>> Hi Divya,
>>>
>>> You don't have the phoenix client-spark JAR in your classpath, which is
>>> required for the phoenix-spark integration to work (as per the
>>> documentation).
>>>
>>> As well, you aren't using the vanilla Apache project that this mailing
>>> list supports, but are using a vendor packaged platform (Hortonworks).
>>> Since they maintain their own patches and forks to the upstream Apache
>>> versions, in general you should opt for filing support tickets with them
>>> first. In this particular case, HDP 2.3.4 doesn't actually provide the
>>> necessary phoenix client-spark JAR by default, so your options are limited
>>> here. Again, I recommend filing a support ticket with Hortonworks.
>>>
>>> Regards,
>>>
>>> Josh
>>>
>>> On Sat, Apr 9, 2016 at 9:11 AM, Divya Gehlot <divya.htco...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> The code which I using to connect to Phoenix for writing
>>>> def writeToTable(df: DataFrame,dbtable: String) = {
>>>> val phx_properties = collection.immutable.Map[String, String](
>>>> "zkUrl" -> "localhost:2181:/hbase-unsecure",
>>>> "table" -> dbtable)
>>>>
>>>> df.write.format("org.apache.phoenix.spark").mode(SaveMode.Overwrite).options(phx_properties).saveAsTable(dbtable)
>>>> }
>>>>
>>>> While Submitting Spark job
>>>> * spark-shell --properties-file /TestDivya/Spark/Phoenix.properties
>>>> --jars
>>>> /usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler-1.2.1.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/zookeeper.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-protocol.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
>>>> --driver-class-path
>>>> /usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
>>>> --packages com.databricks:spark-csv_2.10:1.4.0 --master yarn-client -i
>>>> /TestDivya/Spark/WriteToPheonix.scala*
>>>>
>>>>
>>>> Getting the below error :
>>>>
>>>> org.apache.spark.sql.AnalysisException:
>>>> org.apache.phoenix.spark.DefaultSource
>>>> does not allow user-specified schemas.;
>>>>
>>>> Am I on the right track or missing any properties ?
>>>>
>>>> Because of this I am unable to proceed with Phoenix and have to find
>>>> alternate options.
>>>> Would really appreciate the help
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: Divya Gehlot <divya.htco...@gmail.com>
>>>> Date: 8 April 2016 at 19:54
>>>> Subject: Re: [HELP:]Save Spark Dataframe in Phoenix Table
>>>> To: Josh Mahonin <jmaho...@gmail.com>
>>>>
>>>>
>>>> Hi Josh,
>>>> I am doing in the same manner as mentioned in Phoenix Spark manner.
>>>> Using the latest version of HDP 2.3.4 .
>>>> In case of version mismatch/lack of spark Phoenix support it's should
>>>> have thrown the error at read also.
>>>> Which is working fine as expected .
>>>> Will surely pass on the code snippets once I log on to my System.
>>>> In the mean while I would like to know the zkURL parameter.If I build
>>>> it with HbaseConfiguration and passing zk quorom ,znode and port .
>>>> It throws error for example localhost :2181/hbase-unsecure
>>>> This localhost gets replaced by all the quorom
>>>> Like quorum1,quorum2:2181/hbase-unsecure
>>>>
>>>> I am just providing the IP address of my HBase master.
>>>>
>>>> I feel like I am not on right track so asked for the help .
>>>> How to connect to Phoenix through Spark on hadoop cluster .
>>>> Thanks for the help.
>>>> Cheers,
>>>> Divya
>>>> On Apr 8, 2016 7:06 PM, "Josh Mahonin" <jmaho...@gmail.com> wrote:
>>>>
>>>>> Hi Divya,
>>>>>
>>>>> That's strange. Are you able to post a snippet of your code to look
>>>>> at? And are you sure that you're saving the dataframes as per the docs (
>>>>> https://phoenix.apache.org/phoenix_spark.html)?
>>>>>
>>>>> Depending on your HDP version, it may or may not actually have
>>>>> phoenix-spark support. Double-check that your Spark configuration is setup
>>>>> with the right worker/driver classpath settings. and that the phoenix JARs
>>>>> contain the necessary phoenix-spark classes
>>>>> (e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
>>>>> following up with Hortonworks.
>>>>>
>>>>> Josh
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <divya.htco...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> I hava a Hortonworks Hadoop cluster having below Configurations :
>>>>>> Spark 1.5.2
>>>>>> HBASE 1.1.x
>>>>>> Phoenix 4.4
>>>>>>
>>>>>> I am able to connect to Phoenix through JDBC connection and able to
>>>>>> read the Phoenix tables .
>>>>>> But while writing the data back to Phoenix table
>>>>>> I am getting below error :
>>>>>>
>>>>>> org.apache.spark.sql.AnalysisException:
>>>>>> org.apache.phoenix.spark.DefaultSource does not allow user-specified
>>>>>> schemas.;
>>>>>>
>>>>>> Can any body help in resolving the above errors or any other solution
>>>>>> of saving Spark Dataframes to Phoenix.
>>>>>>
>>>>>> Would really appareciate the help.
>>>>>>
>>>>>> Thanks,
>>>>>> Divya
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
