Hi Kiran,
Thanks for responding. We would like to know how industry is dealing scenario
like Update in SPARK. Here is our scenario Manjunath, We are in process of
migrating our SQL server data to Spark. We have our logic in stored procedure,
where we dynamically create SQL String and execute that SQL String (Dynamic
SQL), we would like to implement Dynamic string and submit to hive context and
execute it .
Here is the query in SQL
UPDATE table1
SET X = A
,Y = B
FROM Table1
WHERE ISNULL([Z] ,'') <> ''
AND [ColumnW] NOT IN ('X' ,'ACD', 'A', 'B', 'C')
AND [ColumnA] IS NULL
We would like to convert using Spark SQL , the other way i would think of is
using of Data frame with "WithColumn" along with WHEN condition for each column
i.e X and Y , here when condition will have same repetitive code applied on
each column based on above where clause stmt/condition . I would like to know
Industry practices for these kind of scenarios.
On 10/26/2016 4:09 AM, Manjunath, Kiran wrote:
Hi,
Can you elaborate with sample example on why you would want to do so?
Ideally there would be a better approach than solving such problems as
mentioned below.
A sample example would help to understand the problem.
Regards,
Kiran
From: Mahender Sarangam
<mahender.bigd...@outlook.com><mailto:mahender.bigd...@outlook.com>
Date: Wednesday, October 26, 2016 at 2:05 PM
To: user <user@spark.apache.org><mailto:user@spark.apache.org>
Subject: Any Dynamic Compilation of Scala Query
Hi,
Is there any way to dynamically execute a string which has scala code
against spark engine. We are dynamically creating scala file, we would
like to submit this scala file to Spark, but currently spark accepts
only JAR file has input from Remote Job submission. Is there any other
way to submit .SCALA instead of .JAR to REST API of Spark ?
/MS
