Hi Eila - While I am not aware of a transpose transform available for CSV files, there is a sample pipeline available to transpose a BigQuery table and write the results to a different table[1]. It might be possible to modify this to work on a CSV source.
[1] https://github.com/GoogleCloudPlatform/professional-services/tree/master/examples/dataflow-bigquery-transpose On Sun, Jan 13, 2019 at 1:58 AM OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote: > Hi all, > > I am working with many CSV files where the common part is the row names > and therefore, my processing should be by columns. My plan is to have the > tables transposed and have the combines tables written into BQ. > So , the code should perform: > 1. transpose the tables (columns -> new_rows, rows->new_columns). new_rows > x new_columns = new_table > 2. extract the new_rows values from the new_tables and write them to big > query. > > Is there an easy way to transpose the CSV files? I am avoiding the usage > of pandas library because the size of the tables could be very large. > should I be concern by the table size. Is this consideration relevant or > should the Apache Beam be able to handle the resources for the pandas? > > What is my other option? is there any built in transpose method that I am > not aware of? > > Thanks for your help, > -- > Eila > www.orielresearch.org > https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/> > p.co <https://www.meetup.com/Deep-Learning-In-Production/> > m/Deep-Learning-In-Production/ > <https://www.meetup.com/Deep-Learning-In-Production/> > > >
