Not able to convert Image binary to an image

Hi, I am working with Apache Spark 2.3.2, implementing an image grep application using Scala 2.11. I am reading images from HDFS using ImageSchema package. The series of step I run are: 1. import org.apache.spark.ml.image.ImageSchema 2. val df = ImageSchema.readImages("hdfs://filepath/*") // all

Re: K8s-Spark client mode : Executor image not able to download application jar from driver

Hi Nikhil, Application jar by default is added to spark.jars so it is fetched by executors when tasks are launched (behind the scenes SparkContext will

Re: Difference between Checkpointing and Persist

Hi Subash, I'm not sure how the checkpointing works, but with StorageLevel.MEMORY_AND_DISK, Spark will store the RDD in on-heap memory, and spill to disk if necessary. However, the data is only usable by that Spark job. Saving the RDD will write the data out to an external storage system, like HDF

Re: writing into oracle database is very slow

Thanks for interesting ideas! Looks like spark directly writing to relational database is not as straight forward as I expected. Sent from my iPhone > On Apr 19, 2019, at 06:58, Khare, Ankit wrote: > > Hi Jiang > > We faced similar issue so we write the file and then use sqoop to export data

Re: Error: NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT while running a Spark-Hive Job

How did you resolved this problem -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: writing into oracle database is very slow

Hi Jiang We faced similar issue so we write the file and then use sqoop to export data to mssql. We achieved a great time benefit with this strategy. Sent from my iPhone On 19. Apr 2019, at 10:47, spark receiver mailto:spark.recei...@gmail.com>> wrote: hi Jiang, i was facing the very same i

Re: BigDL and Analytics Zoo talks at upcoming Spark+AI Summit and Strata London

Thanks for sharing. Sent from my iPhone On 19. Apr 2019, at 01:35, Jason Dai mailto:jason@gmail.com>> wrote: Hi all, Please see below for a list of upcoming technical talks on BigDL and Analytics Zoo (https://github.com/intel-analytics/analytics-zoo/) in the coming weeks: * Engineers

Re: writing into oracle database is very slow

hi Jiang, i was facing the very same issue ,the solution is write to file and using oracle external table to do the insert. hope this could help. Dalin On Thu, Apr 18, 2019 at 11:43 AM Jörn Franke wrote: > What is the size of the data? How much time does it need on HDFS and how > much on Orac