Hi Josh, You mean because of missing Jar I am not unable to write it to DB and getting this exception "org.apache.spark.sql.AnalysisException: org.apache.phoenix.spark.DefaultSource does not allow user-specified schemas "
If I add the jar you mentioned I will be able to write it DB ? Thanks, Divya On 9 April 2016 at 23:01, Josh Mahonin <jmaho...@gmail.com> wrote: > Hi Divya, > > You don't have the phoenix client-spark JAR in your classpath, which is > required for the phoenix-spark integration to work (as per the > documentation). > > As well, you aren't using the vanilla Apache project that this mailing > list supports, but are using a vendor packaged platform (Hortonworks). > Since they maintain their own patches and forks to the upstream Apache > versions, in general you should opt for filing support tickets with them > first. In this particular case, HDP 2.3.4 doesn't actually provide the > necessary phoenix client-spark JAR by default, so your options are limited > here. Again, I recommend filing a support ticket with Hortonworks. > > Regards, > > Josh > > On Sat, Apr 9, 2016 at 9:11 AM, Divya Gehlot <divya.htco...@gmail.com> > wrote: > >> Hi, >> The code which I using to connect to Phoenix for writing >> def writeToTable(df: DataFrame,dbtable: String) = { >> val phx_properties = collection.immutable.Map[String, String]( >> "zkUrl" -> "localhost:2181:/hbase-unsecure", >> "table" -> dbtable) >> >> df.write.format("org.apache.phoenix.spark").mode(SaveMode.Overwrite).options(phx_properties).saveAsTable(dbtable) >> } >> >> While Submitting Spark job >> * spark-shell --properties-file /TestDivya/Spark/Phoenix.properties >> --jars >> /usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler-1.2.1.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/zookeeper.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-protocol.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar >> --driver-class-path >> /usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar >> --packages com.databricks:spark-csv_2.10:1.4.0 --master yarn-client -i >> /TestDivya/Spark/WriteToPheonix.scala* >> >> >> Getting the below error : >> >> org.apache.spark.sql.AnalysisException: >> org.apache.phoenix.spark.DefaultSource >> does not allow user-specified schemas.; >> >> Am I on the right track or missing any properties ? >> >> Because of this I am unable to proceed with Phoenix and have to find >> alternate options. >> Would really appreciate the help >> >> >> >> >> >> ---------- Forwarded message ---------- >> From: Divya Gehlot <divya.htco...@gmail.com> >> Date: 8 April 2016 at 19:54 >> Subject: Re: [HELP:]Save Spark Dataframe in Phoenix Table >> To: Josh Mahonin <jmaho...@gmail.com> >> >> >> Hi Josh, >> I am doing in the same manner as mentioned in Phoenix Spark manner. >> Using the latest version of HDP 2.3.4 . >> In case of version mismatch/lack of spark Phoenix support it's should >> have thrown the error at read also. >> Which is working fine as expected . >> Will surely pass on the code snippets once I log on to my System. >> In the mean while I would like to know the zkURL parameter.If I build it >> with HbaseConfiguration and passing zk quorom ,znode and port . >> It throws error for example localhost :2181/hbase-unsecure >> This localhost gets replaced by all the quorom >> Like quorum1,quorum2:2181/hbase-unsecure >> >> I am just providing the IP address of my HBase master. >> >> I feel like I am not on right track so asked for the help . >> How to connect to Phoenix through Spark on hadoop cluster . >> Thanks for the help. >> Cheers, >> Divya >> On Apr 8, 2016 7:06 PM, "Josh Mahonin" <jmaho...@gmail.com> wrote: >> >>> Hi Divya, >>> >>> That's strange. Are you able to post a snippet of your code to look at? >>> And are you sure that you're saving the dataframes as per the docs ( >>> https://phoenix.apache.org/phoenix_spark.html)? >>> >>> Depending on your HDP version, it may or may not actually have >>> phoenix-spark support. Double-check that your Spark configuration is setup >>> with the right worker/driver classpath settings. and that the phoenix JARs >>> contain the necessary phoenix-spark classes >>> (e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest >>> following up with Hortonworks. >>> >>> Josh >>> >>> >>> >>> On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <divya.htco...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> I hava a Hortonworks Hadoop cluster having below Configurations : >>>> Spark 1.5.2 >>>> HBASE 1.1.x >>>> Phoenix 4.4 >>>> >>>> I am able to connect to Phoenix through JDBC connection and able to >>>> read the Phoenix tables . >>>> But while writing the data back to Phoenix table >>>> I am getting below error : >>>> >>>> org.apache.spark.sql.AnalysisException: >>>> org.apache.phoenix.spark.DefaultSource does not allow user-specified >>>> schemas.; >>>> >>>> Can any body help in resolving the above errors or any other solution >>>> of saving Spark Dataframes to Phoenix. >>>> >>>> Would really appareciate the help. >>>> >>>> Thanks, >>>> Divya >>>> >>> >>> >> >