Hi all, I am working with many CSV files where the common part is the row names and therefore, my processing should be by columns. My plan is to have the tables transposed and have the combines tables written into BQ. So , the code should perform: 1. transpose the tables (columns -> new_rows, rows->new_columns). new_rows x new_columns = new_table 2. extract the new_rows values from the new_tables and write them to big query.
Is there an easy way to transpose the CSV files? I am avoiding the usage of pandas library because the size of the tables could be very large. should I be concern by the table size. Is this consideration relevant or should the Apache Beam be able to handle the resources for the pandas? What is my other option? is there any built in transpose method that I am not aware of? Thanks for your help, -- Eila www.orielresearch.org https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/>p.co <https://www.meetup.com/Deep-Learning-In-Production/> m/Deep-Learning-In-Production/ <https://www.meetup.com/Deep-Learning-In-Production/>
