There are other distributed execution engines (like hive, trino) that do support non-string data types for partition columns such as date and integer. Any idea why this restriction exists in Spark? ..
On Tue, 11 Apr 2023 at 20:34, Chitral Verma <chitralve...@gmail.com> wrote: > Because the name of the directory cannot be an object, it has to be a > string to create partitioned dirs like "date=2023-04-10" > > On Tue, 11 Apr, 2023, 8:27 pm Charles vinodh, <mig.flan...@gmail.com> > wrote: > >> >> Hi Team, >> >> We are running into the below error when we are trying to run a simple >> query a partitioned table in Spark. >> >> *MetaException(message:Filtering is supported only on partition keys of type >> string) >> * >> >> >> Our the partition column has been to type *date *instead of string and >> query is a very simple SQL as shown below. >> >> *SELECT * FROM my_table WHERE partition_col = date '2023-04-11'* >> >> Any idea why spark mandates partition columns to be of type string?. Is >> there a recommended work around for this issue? >> >> >>
