The Dataset unit test is much slower than the RDD unit test (in Scala)

[Tanin Na Nakorn]

Hi All,

Our data job is very complex (e.g. 100+ joins), and we have switched from
RDD to Dataset recently.

We've found that the unit test takes much longer. We profiled it and have
found that it's the planning phase that is slow, not execution.

I wonder if anyone has encountered this issue before and if there's a way
to make the planning phase faster (e.g. maybe disabling certain optimizers).

Any thoughts or input would be appreciated.

Thank you,
Tanin

Dynamic allocation on K8

[Nikhil Goyal]

Hi folks,
When running spark on Kubernetes is it possible to use dynamic allocation?
Some blog posts

mentioned that dynamic allocation is available, however I am not sure how
it works. Spark official docs

say that shuffle service is not yet available.

Thanks
Nikhil

Re: Prometheus with spark

[Raja bhupati]

We have use case where we would like process Prometheus metrics data with
spark

On Tue, Oct 25, 2022, 19:49 Jacek Laskowski  wrote:

> Hi Raj,
>
> Do you want to do the following?
>
> spark.read.format("prometheus").load...
>
> I haven't heard of such a data source / format before.
>
> What would you like it for?
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books 
> Follow me on https://twitter.com/jaceklaskowski
>
> 
>
>
> On Fri, Oct 21, 2022 at 6:12 PM Raj ks  wrote:
>
>> Hi Team,
>>
>>
>> We wanted to query Prometheus data with spark. Any suggestions will
>> be appreciated
>>
>> Searched for documents but did not got any prompt one
>>
>

Re: Prometheus with spark

[Jacek Laskowski]

Hi Raj,

Do you want to do the following?

spark.read.format("prometheus").load...

I haven't heard of such a data source / format before.

What would you like it for?

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski




On Fri, Oct 21, 2022 at 6:12 PM Raj ks  wrote:

> Hi Team,
>
>
> We wanted to query Prometheus data with spark. Any suggestions will
> be appreciated
>
> Searched for documents but did not got any prompt one
>