Thanks Alex for the quick answer and confirming my suspicion. > So requested schema must always be compatible (same types) and be a subset of > the file schema.
In that case, Parquet does't allow type promotion. and the type in requestSchema should be the "same" type. And requestedSchema only useful for projection pushdown, right? Regards, Mohammad On Wednesday, October 14, 2015 6:25 PM, Alex Levenson <[email protected]> wrote: It's always a cooperation between the two. file schema is how the file was written. requested schema is what subset of the file schema you want to read. So requested schema must always be compatible (same types) and be a subset of the file schema. On Wed, Oct 14, 2015 at 6:10 PM, Mohammad Islam <[email protected]> wrote: > Hi, > > Before I investigate deep into the code, it will be really helpful if > someone can help me on this issue. > > My main question is : if both schemas are provided, the parquet file > reader uses which schema. > For example, I provide requestedSchema and readSupportMetadata in > ReadContext. Looks like, parquet reader is using the requested schema to > read the file. > > In my case, my request schema has a different data type for a column > compared to file schema. For example, one field type is "int" in file > schema but "bigint" in requested schema. I got this exception. > > > "Failed with exception > java.io.IOException:parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file file:/tmp/my/my_year.pq". But if I use the "int" > in my requestedSchema, it works fine. > > Any quick help or pointer is highly appreciated. > > Regards, > Mohammad > -- Alex Levenson @THISWILLWORK
