Our users ask for it....
Regard, Junfeng Chen On Wed, Apr 4, 2018 at 5:45 PM, Gourav Sengupta <gourav.sengu...@gmail.com> wrote: > Hi Junfeng, > > can I ask why it is important to remove the empty column? > > Regards, > Gourav Sengupta > > On Tue, Apr 3, 2018 at 4:28 AM, Junfeng Chen <darou...@gmail.com> wrote: > >> I am trying to read data from kafka and writing them in parquet format >> via Spark Streaming. >> The problem is, the data from kafka are in variable data structure. For >> example, app one has columns A,B,C, app two has columns B,C,D. So the data >> frame I read from kafka has all columns ABCD. When I decide to write the >> dataframe to parquet file partitioned with app name, >> the parquet file of app one also contains columns D, where the columns D >> is empty and it contains no data actually. So how to filter the empty >> columns when I writing dataframe to parquet? >> >> Thanks! >> >> >> Regard, >> Junfeng Chen >> > >
