issue with regexp_replace

Hi Team, I am trying to use regexp_replace in spark sql it throwing error expected , but found Scalar in 'reader', line 9, column 45: ... select translate(payload, '"', '"') as payload i am trying to remove all character from \\\" with "

convert josn string in spark sql

Hi Team, I have kafka messages where json is coming as string how can create table after converting json string to json using spark sql

Re: Spark-hive integration on HDInsight

Hey jay How you are making your cluster are you using spark cluster All this thing should be set up automatically Sent from my iPhone > On Feb 21, 2019, at 12:12 PM, Felix Cheung wrote: > > You should check with HDInsight support > > From: Jay Singh > Sent: Wednesday, February 20,

executing stored procedure through spark

Hi /team, The way we call java program to executed stored procedure is there any way we can achieve the same using pyspark

Re: Can we deploy python script on a spark cluster

Hi Lehak You can make a scala project with oozing class And one run class which will ship your python file to cluster Define oozie coordinator with spark action or shell action We are deploying pyspark based machine learning code Sent from my iPhone > On Aug 2, 2018, at 8:46 AM, Lehak

Pyspark is not picking up correct python version on azure hdinsight

Hi Guys, Pyspark is not picking up correct python version on azure hdinsight property setup in spark2-env PYSPARK_PYTHON=${PYSPARK3_PYTHON:-/usr/bin/anaconda/envs/py35/bin/python3} export PYSPARK_DRIVER_PYTHON=${PYSPARK3_PYTHON:-/usr/bin/anaconda/envs/py35/bin/python3} Thanks

Re: help needed in perforance improvement of spark structured streaming

ream .trigger(ProcessingTime("60 seconds")) .option("checkpointLocation", checkpointUrl) .foreach(new SimpleSqlServerSink(jdbcUrl, connectionProperties)) On Sat, May 5, 2018 at 12:20 PM, amit kumar singh wrote: > Hi Community, > > I have a use case where i n

How can we group by messages coming in per batch of structured streaming

Hi Team, I have a requirement where i need to to combine all json messages coming in batch of structured streaming into one single json messages which can be separated by comma or any other delimiter and store it i have tried to group by kafka partition i tried using concat but its not working

help in copying data from one azure subscription to another azure subscription

HI Team, We are trying to move data between one azure subscription to another azure subscription is there a faster way to do through spark i am using distcp and its taking for ever thanks rohit

help needed in perforance improvement of spark structured streaming

Hi Community, I have a use case where i need to call stored procedure through structured streaming. I am able to send kafka message and call stored procedure , but since foreach sink keeps on executing stored procedure per message i want to combine all the messages in single dtaframe and then

User class threw exception: java.lang.ClassNotFoundException: Failed to find data source: kafka. Please find packages at http://spark.apache.org/third-party-projects.html

Hi Team, I am working on structured streaming i have added all libraries in build,sbt then also its not picking up right library an failing with error User class threw exception: java.lang.ClassNotFoundException: Failed to find data source: kafka. Please find packages at

how to call stored procedure from spark

Hi Guys I have stored procedure which does transformation and write it to sql server table i am not able to execute this through spark is there any way in which spark streaming sink can just be calling these stored procedure

How to bulk insert using spark streaming job

How to bulk insert using spark streaming job Sent from my iPhone

Re: optimize hive query to move a subset of data from one partition table to another table

iao <richardqiao2...@gmail.com> wrote: > Would you mind share your code with us to analyze? > > > On Feb 10, 2018, at 10:18 AM, amit kumar singh <amitiem...@gmail.com> > wrote: > > > > Hi Team, > > > > We have hive external table which has 50 tb

optimize hive query to move a subset of data from one partition table to another table

Hi Team, We have hive external table which has 50 tb of data partitioned on year month day i want to move last 2 month of data into another table when i try to do this through spark ,more than 120k task are getting created what is the best way to do this thanks Rohit

How to configure spark with java

Hello everyone I want to use spark with java API Please let me know how can I configure it Thanks A - To unsubscribe e-mail: user-unsubscr...@spark.apache.org