thijsheijden commented on issue #2057:
URL:
https://github.com/apache/iceberg-python/issues/2057#issuecomment-3019511937
I am experiencing this bug as well. I am trying to insert the TPC-H dataset
generated using DuckDB, written to Parquet files, into Iceberg using the
`add_files` method. I have attached my Parquet generation code as well as my
Iceberg creation code.
Generation of TPC-H Parquet files:
```
# Connect and load TPCH extension
con = duckdb.connect()
con.execute("INSTALL tpch; LOAD tpch")
# Define TPCH table names
tpch_tables = ['orders', 'lineitem']
# Optional: Set scale factor (controls number of rows)
scale = 1.0
con.execute(f"CALL dbgen(sf={scale})")
for table in tpch_tables:
print(f'Generating files for table {table}')
cur_offset = 0
file_idx = 0
batch_idx = 0
while True:
if file_idx % args.b == 0:
Path(f'{table}/batch_{batch_idx}').mkdir(exist_ok=True)
batch_idx += 1
print(f'Writing batch {batch_idx}')
query = f"SELECT * FROM '{table}' LIMIT 1000 OFFSET {cur_offset}"
df = con.execute(query).arrow() # Get as Arrow Table
# Write to a single Parquet file
pq.write_table(df, f'{table}/batch_{batch_idx -
1}/part{file_idx}.parquet')
if len(df) < 1000:
break
file_idx += 1
cur_offset += 1000
```
Insertion into Iceberg table:
```
warehouse_path = "/data"
catalog = load_catalog(
"pyiceberg",
**{
'type': 'sql',
"uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db",
"warehouse": f"file://{warehouse_path}",
},
)
catalog.create_namespace_if_not_exists("default")
# Load the batches of files to import
batches = os.listdir(args.file_dir)
first_file = os.path.join(args.file_dir, "batch_0",
os.listdir(os.path.join(args.file_dir, "batch_0"))[0])
# Create table using schema of the first file
df = pq.read_table(first_file)
table = catalog.create_table_if_not_exists(
f"default.{args.table}",
schema=df.schema,
)
batch_idx = 1
for batch_dir in batches:
print(f"Adding batch {batch_idx}")
batch_dir = os.path.join(args.file_dir, batch_dir)
file_paths = os.listdir(batch_dir)
file_paths = [batch_dir + '/' + s for s in file_paths]
table.add_files(file_paths=file_paths, check_duplicate_files=False)
batch_idx += 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
