Re: Spark for Image Processing Acceleration

You could distribute the computation across a cluster with Spark and Horovod (and Petastorm) for example: https://github.com/horovod/horovod https://github.com/uber/petastorm If you're at a few hours, it may not be worth it - it's not hard to set up but is more involved. You may do better with a

Spark for Image Processing Acceleration

Hi, My name is Peiwen. I'm working with Dr. Brain, an AI company focused on medical imaging processing and deep learning. Our website ishttp://drbrain.net/index_en.aspx We basically do 2 major things. 1. image process, like lesion drawing 2. deep learning for neural disease prediction, like

Spark on k8s

I have done some observations on running Spark on Kubernetes (AKA k8s). The model works on the basis of the "one-container-per-Pod" model meaning that for each node of the cluster you will have one node running the driver and each remaining

Accessing a kerberized HDFS using Spark on Openshift

Hi, I have a pod on openshift 4.6 running a jupyter notebook with spark 3.1.1 and python 3.7 (based on open data hub, tweaked the dockerfile because I wanted this specific python version). I'm trying to run spark in client mode using the image of google's spark operator