Hello,
Better to ask your question on the spark operator github and not on this
mailing list. For the answer, try:
type: Always
Best regards,
Ali Gouta.
On Sun, May 2, 2021 at 6:15 PM Sachit Murarka
wrote:
> Hi All,
>
> I am using Spark with Kubernetes, Can anyone please tell me
Thanks Mich !
Ali Gouta.
On Sun, Apr 4, 2021 at 6:44 PM Mich Talebzadeh
wrote:
> Hi Ali,
>
> The old saying of one experiment is worth a hundred hypotheses, still
> stands.
>
> As per Test driven approach have a go at it and see what comes out. Forum
> members including my
Great, so SSS provides also an api that allows handling RDDs through
dataFrames using foreachBatch. Still that I am not sure this is a
good practice in general right ? Well, it depends on the use case in any
way.
Thank you so much for the hints !
Best regards,
Ali Gouta.
On Sun, Apr 4, 2021 at
Thank you guys for your answers, I will dig more this new way of doing
things and why not consider leaving the old Dstreams and use instead
structured streaming. Hope that strucrured streaming + spark on Kubernetes
works well and the combination is production ready.
Best regards,
Ali Gouta.
Le
ark structured streaming to the concerned consumer group ?
Best regards,
Ali Gouta.
think this is the simplest way to achieve what you want to do.
Best regards,
Ali Gouta.
On Tue, Mar 9, 2021 at 11:30 AM forece85 wrote:
> We are doing batch processing using Spark Streaming with Kinesis with a
> batch
> size of 5 mins. We want to send all events with same eventI
The spark UI is misleading in spark 2.4.4. I moved to spark 2.4.5 and it
fixed it. Now, your problem should be somewhere else. Probably related to
memory consumption but not the one you see in the UI.
Best regards,
Ali Gouta.
On Sun, May 17, 2020 at 7:36 PM András Kolbert
wrote:
> Hi,
&g
. Then
have pod anti-affinity to make sure they are not running on the same node.
You may achieve this by running an NFS fiesystem and then create a PV/PVC
that mounts to that shared file system. The persistentVolumeClaim defined
in your Yaml should call the PVC you created.
Best regards,
Ali Gouta
about it.
Ali Gouta.
Le 3 févr. 2017 22:24, "Jacek Laskowski" a écrit :
Hi,
An interesting case. You don't use Spark resources whatsoever.
Creating a SparkConf does not use YARN...yet. I think any run mode
would have the same effect. So, although spark-submit could have
returned e
For hive, you may use sqoop to achieve this. In my opinion, you may also
run a spark job to make it..
Le 9 avr. 2016 00:25, "Ashok Kumar" a écrit :
Hi,
Anyone has suggestions how to create and copy Hive and Spark tables from
Production to UAT.
One way would be to copy table data to external fil
Something like
Stream.foreachRdd(rdd=>
rdd.collect.foreach(print accum))
Should answer your question. You get things printed in Each batch interval
Ali Gouta
Le 25 déc. 2015 04:22, "Roberto Coluccio" a
écrit :
> Hello,
>
> I have a batch and a streaming driver using sam
nerateRDD, "foo");
That's it...
At last, be carreful while defining your sets of your "conf". For instance
you may end-up changing the localhost by the real IP adresse of your
Elasticsearch node...
Ali Gouta.
On Mon, Dec 14, 2015 at 1:52 PM, Spark Enthusiast
wrote:
&
Indeed, you are right! I felt like I was missing or misunderstanding
something.
Thank you so much!
Ali Gouta.
On Thu, Dec 10, 2015 at 10:04 PM, Cody Koeninger wrote:
> I'm a little confused as to why you have fake events rather than just
> doing foreachRDD or foreachPartition on
13 matches
Mail list logo