In terms of schema and ParquetIO source/sink, there was an answer in some previous thread:
Currently (without introducing any change in ParquetIO) there is no way to not pass the avro schema. It will probably be replaced with Beam's schema in the future () [1] https://lists.apache.org/thread.html/a466ddeb55e47fd780be3bcd8eec9d6b6eaf1dfd566ae5278b5fb9e8@%3Cuser.beam.apache.org%3E wt., 31 lip 2018 o 10:19 Akanksha Sharma B <akanksha.b.sha...@ericsson.com> napisał(a): > Hi, > > > I am hoping to get some hints/pointers from the experts here. > > I hope the scenario described below was understandable. I hope it is a > valid use-case. Please let me know if I need to explain the scenario > better. > > > Regards, > > Akanksha > > ------------------------------ > *From:* Akanksha Sharma B > *Sent:* Friday, July 27, 2018 9:44 AM > *To:* dev@beam.apache.org > *Subject:* Re: pipeline with parquet and sql > > > Hi, > > > Please consider following pipeline:- > > > Source is Parquet file, having hundreds of columns. > > Sink is Parquet. Multiple output parquet files are generated after > applying some sql joins. Sql joins to be applied differ for each output > parquet file. Lets assume we have a sql queries generator or some > configuration file with the needed info. > > > Can this be implemented generically, such that there is no need of the > schema of the parquet files involved or any intermediate POJO or beam > schema. > > i.e. the way spark can handle it - read parquet into dataframe, create > temp view and apply sql queries to it, and write it back to parquet. > > As I understand, beam SQL needs (Beam Schema or POJOs) and parquetIO needs > avro schemas. Ideally we dont want to see POJOs or schemas. > If there is a way we can achieve this with beam, please do help. > > Regards, > Akanksha > > ------------------------------ > *From:* Akanksha Sharma B > *Sent:* Tuesday, July 24, 2018 4:47:25 PM > *To:* u...@beam.apache.org > *Subject:* pipeline with parquet and sql > > > Hi, > > > Please consider following pipeline:- > > > Source is Parquet file, having hundreds of columns. > > Sink is Parquet. Multiple output parquet files are generated after > applying some sql joins. Sql joins to be applied differ for each output > parquet file. Lets assume we have a sql queries generator or some > configuration file with the needed info. > > > Can this be implemented generically, such that there is no need of the > schema of the parquet files involved or any intermediate POJO or beam > schema. > > i.e. the way spark can handle it - read parquet into dataframe, create > temp view and apply sql queries to it, and write it back to parquet. > > As I understand, beam SQL needs (Beam Schema or POJOs) and parquetIO needs > avro schemas. Ideally we dont want to see POJOs or schemas. > If there is a way we can achieve this with beam, please do help. > > Regards, > Akanksha > > > >
