[ https://issues.apache.org/jira/browse/ARROW-16302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Weston Pace resolved ARROW-16302. --------------------------------- Fix Version/s: 9.0.0 Resolution: Fixed Issue resolved by pull request 12977 [https://github.com/apache/arrow/pull/12977] > [C++] Null values in partitioning field for FilenamePartitioning > ---------------------------------------------------------------- > > Key: ARROW-16302 > URL: https://issues.apache.org/jira/browse/ARROW-16302 > Project: Apache Arrow > Issue Type: Bug > Components: C++ > Affects Versions: 8.0.0 > Reporter: Sanjiban Sengupta > Assignee: Sanjiban Sengupta > Priority: Major > Labels: bug, partitioning, pull-request-available > Fix For: 9.0.0 > > Time Spent: 5h 40m > Remaining Estimate: 0h > > While using FilenamePartitioning, currently when we read a dataset, say from > PyArrow, the partitioning field only has Null values. > > The issue can be reproduced with the following code > > {code:python} > table = pa.table([ > pa.array(range(20)), pa.array(np.random.randn(20)), > pa.array(np.repeat(['a', 'b'], 10))], > names=["f1", "f2", "part"] > ) > part = ds.partitioning(pa.schema([("part", pa.string())]), flavor="filename") > # test is the directory where partitions are written > ds.write_dataset( > table, "test", > format="parquet", partitioning=part > ) > result = ds.dataset( > "test", format="parquet", partitioning=part, > ).to_table(){code} > This results in something this: > {code:python} > pyarrow.Table > f1: int64 > f2: double > part: string > ---- > f1: [[0,1,2,3,4,5,6,7,8,9],[10,11,12,13,14,15,16,17,18,19]] > f2: > [[-1.1753280347394899,-0.9640239222827617,0.7907247451009602,1.3667778347936321,0.005079832420686733,0.9024313772071855,-1.01618656608383,-1.1459911861999188,-0.7407261867306765,-0.012823499364722428],[0.29893685698088185,1.3907720928021299,-0.48826416913435605,-1.3436821154932153,-0.5492388164165941,-0.07093280675027104,0.009918818541272493,-1.05561750529359,-2.0209000426858927,-0.28081085330210676]] > part: > [[null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,null]] > {code} > > > -- This message was sent by Atlassian Jira (v8.20.7#820007)