Re: How to apply ranger policies on Spark

[Dennis Suhari]

Hi Joyan,

Spark uses its own metastore. Using Ranger you need to use the Hive Metastore. 
For this you need to point to Hive Metastore and use HiveContext in your Spark 
Code.

Br,

Dennis

Von meinem iPhone gesendet

> Am 23.11.2020 um 19:04 schrieb joyan sil :
> 
> 
> Hi,
> 
> We have ranger policies defined on the hive table and authorization works as 
> expected when we use hive cli and beeline. But when we access those hive 
> tables using spark-shell or spark-submit it does not work.
> 
>  Any suggestions to make Ranger work with Spark?
> 
> Regards
> Joyan

Building High-performance Lake for Spark using OSS, Hudi, Alluxio

[Bin Fan]

 Hi Spark Users,

Check out this blog on Building High-performance Data Lake using Apache
Hudi, Spark and Alluxio at T3Go 


Cheers

- Bin Fan

Re: How to submit a job via REST API?

[Dennis Suhari]

Hi Yang,

I am using Livy Server for submitting jobs.

Br,

Dennis



Von meinem iPhone gesendet

> Am 24.11.2020 um 03:34 schrieb Zhou Yang :
> 
> 
> Dear experts,
> 
> I found a convenient way to submit job via Rest API at 
> https://gist.github.com/arturmkrtchyan/5d8559b2911ac951d34a#file-submit_job-sh.
> But I did not know whether can I append `—conf` parameter like what I did in 
> spark-submit. Can someone can help me with this issue?
> 
> Regards, Yang
> 

how to manage HBase connections in Executors of Spark Streaming ?

[big data]

Hi,

Does any best practices about how to manage Hbase connections with 
kerberos authentication in Spark Streaming (YARN) environment?


Want to now how executors manage the HBase connections,how to create 
them, close them and refresh Kerberos expires.


Thanks.

How to submit a job via REST API?

[Zhou Yang]

Dear experts,

I found a convenient way to submit job via Rest API at 
https://gist.github.com/arturmkrtchyan/5d8559b2911ac951d34a#file-submit_job-sh.
But I did not know whether can I append `—conf` parameter like what I did in 
spark-submit. Can someone can help me with this issue?

Regards, Yang

How to apply ranger policies on Spark

[joyan sil]

Hi,

We have ranger policies defined on the hive table and authorization works
as expected when we use hive cli and beeline. But when we access those hive
tables using spark-shell or spark-submit it does not work.

 Any suggestions to make Ranger work with Spark?


Regards

Joyan

Re: spark cassandra questiom

[Sonal Goyal]

Yes, it should be good to use Spark for this use case in my opinion. You
can look into using the Cassandra Spark connector for persisting your
updated data into Cassandra.

Cheers,
Sonal
Nube Technologies 
Join me at
Data Con LA Oct 23 | Big Data Conference Europe. Nov 24 | GIDS AI/ML Dec 3




On Tue, Nov 10, 2020 at 6:39 PM adfel70  wrote:

> I an very very new to both spark and spark structured streaming. I have to
> write an application that receives a very very large csv files in hdfs
> folder. the app must take the file and on each row it must read from
> Cassandra data base some rows (not many rows will be returned for each row
> in csv). On each row it red it must preform some simple calculations and
> update the rows it red with the results and save the updated rows to
> Cassandra.
>
> I have spark version 2.4 and must use python.
>
> Is this a suitable scenario for spark structured streaming?
>
> thanks
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>