Try https://mapr.com/blog/spark-data-source-api-extending-our-spark-sql-query-engine/
Thanks, Assaf. -----Original Message----- From: OBones [mailto:obo...@free.fr] Sent: Monday, June 12, 2017 1:01 PM To: user@spark.apache.org Subject: [How-To] Custom file format as source Hello, I have an application here that generates data files in a custom binary format that provides the following information: Column list, each column has a data type (64 bit integer, 32 bit string index, 64 bit IEEE float, 1 byte boolean) Catalogs that give modalities for some columns (ie, column 1 contains only the following values: A, B, C, D) Array for actual data, each row has a fixed size according to the columns. Here is an example: Col1, 64bit integer Col2, 32bit string index Col3, 64bit integer Col4, 64bit float Catalog for Col1 = 10, 20, 30, 40, 50 Catalog for Col2 = Big, Small, Large, Tall Catalog for Col3 = 101, 102, 103, 500, 5000 Catalog for Col4 = (no catalog) Data array = 8 bytes, 4 bytes, 8 bytes, 8 bytes, 8 bytes, 4 bytes, 8 bytes, 8 bytes, 8 bytes, 4 bytes, 8 bytes, 8 bytes, 8 bytes, 4 bytes, 8 bytes, 8 bytes, 8 bytes, 4 bytes, 8 bytes, 8 bytes, ... I would like to use this kind of file as a source for various ML related computations (CART, RandomForrest, Gradient boosting...) and Spark is very interesting in this area. However, I'm a bit lost as to what I should write to have Spark use that file format as a source for its computation. Considering that those files are quite big (100 million lines, hundreds of gigs on disk), I'd rather not create something that writes a new file in a built-in format, but I'd rather write some code that makes Spark accept the file as it is. I looked around and saw the textfile method but it is not applicable to my case. I also saw the spark.read.format("libsvm") syntax which tells me that there is a list of supported formats known to spark, which I believe are called Dataframes, but I could not find any tutorial on this subject. Would you have any suggestion or links to documentation that would get me started? Regards, Olivier --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org