+1, If carbondata sdk can support load data from parquet, ORC, CSV, Avro and JSON file, it will more convenient for users to use CarbonData. It avoid every user to parser different fileformat and convert to carbondata format by coding.
CarbonData SDK can refer spark-sql implementation， but CarbonSDK should dependency Spark to implement this function because carbonData SDK is usually used for no spark environment. CarbonData can integrate Parquet, Json, CSV parser SDK to implement it. For API design: 1. For load parquet, avro , json, csv files: (1) it's better to support load many times data after create CarbonWriter. (2) it's better to support schema map, for example, carbondata and parquet has different schema, such as column name is different, column name is col1 in carbon , but column name is col2 in parquet, it's better for CarbonSDK to support schema map, map parquet col2 to carbon col1. 2. API name: public CarbonWriterBuilder withParquetFile(String filePath) =>suggestion: Parquet maybe has many files in the same folder. withParquetFile is easily misunderstand one parquet file. so we can use withParquet(String filePath) or withParquetPath withOrcFile and withCsvFile is the same. and it better to support load files list , for example, CarbonData SDK can support load three files of five files in the same folder, which can be selected by users. 3. lic void write() throws IOException; => we should support load data after create CarbonWriter, and load data to memory and convert to carbon after call load method. 4. now we write data to disk/obs/hdfs by calling close method, it's better to support flush. -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
