mshauneu opened a new issue, #1239:
URL: https://github.com/apache/datafusion-ballista/issues/1239
**Describe the bug**
Get an error when query a partitioned data:
```
Error: ArrowError(ExternalError(Execution("Job i0BByln failed: Error
planning job i0BByln: DataFusionError(Context(\"type_coercion\",
SchemaError(DuplicateQualifiedField { qualifier: Bare { table: \"bhive\" },
name: \"year\" }, Some(\"\"))))")), None)
```
DataFusion works as expected.
**To Reproduce**
```rust
use ballista::prelude::SessionContextExt;
use datafusion::arrow::datatypes::DataType;
use datafusion::datasource::file_format::parquet::ParquetFormat;
use datafusion::datasource::listing::{
ListingOptions, ListingTable, ListingTableConfig, ListingTableUrl,
};
use datafusion::error::Result;
use datafusion::prelude::SessionContext;
use std::sync::Arc;
#[tokio::main]
async fn main() -> Result<()> {
test(&SessionContext::new()).await?;
test(&SessionContext::standalone().await?).await?;
Ok(())
}
async fn test(ctx: &SessionContext) -> Result<()> {
let session_state = ctx.state();
let listing_table_url = ListingTableUrl::parse("data/bhive")?;
let table_partition_cols = vec![
("year".to_owned(), DataType::Int64),
("month".to_owned(), DataType::Int64),
];
let file_format = ParquetFormat::new()
.with_enable_pruning(true)
.with_skip_metadata(true);
let listing_options =
ListingOptions::new(Arc::new(file_format)).with_table_partition_cols(table_partition_cols);
let resolved_schema = listing_options
.infer_schema(&session_state, &listing_table_url)
.await?;
let config = ListingTableConfig::new(listing_table_url)
.with_listing_options(listing_options)
.with_schema(resolved_schema);
let listing_table = ListingTable::try_new(config)?;
ctx.register_table("bhive", Arc::new(listing_table))?;
ctx.sql("SELECT year, month FROM bhive GROUP BY year, month ORDER BY
year, month")
.await?
.show()
.await?;
Ok(())
}
````
**Expected behavior**
```
+------+-------+
| year | month |
+------+-------+
| 2011 | 1 |
| 2011 | 2 |
| 2011 | 3 |
| 2011 | 4 |
| 2011 | 5 |
...
```
**Additional context**
```
tree bhive
bhive
├── year=2011
│ ├── month=1
│ │ └── trips_0.parquet
│ ├── month=2
│ │ └── trips_0.parquet
│ ├── month=3
│ │ └── trips_0.parquet
│ ├── month=4
│ │ └── trips_0.parquet
│ ├── month=5
│ │ └── trips_0.parquet
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
