From: Joel D [mailto:games2013....@gmail.com]
Sent: Monday, May 29, 2017 9:04 PM
To: user@spark.apache.org
Subject: Schema Evolution Parquet vs Avro
Hi,
We are trying to come up with the best storage format for handling schema
changes in ingested data.
We noticed that both avro and parquet allows one to select based on column name
instead of the data index/position of data. However, we are inclined towards
parquet for better read performance since it's columnar and we will be
selecting few columns instead of all. Data will be processed and saved to
partitions on which we will have hive external tables.
Will parquet be able to handle the following:
- Column renaming from between data
- Column removal from between
- DataType change of existing column (int to bigint should be allowed, right?)
Please advise.
Thanks,
Sam
