culpgrant opened a new issue, #4548:
URL: https://github.com/apache/arrow-adbc/issues/4548
### What happened?
When passing a `pyarrow.Table` to `adbc_ingest` the diver always produces a
single Parquet file, a single PUT and a single COPY INTO. Regardless of data
size of any connection option
I would expected multiple parquet files with parallel PUTS and overlapping
COPY INTO statements. I got just 1 parquet file, 1 PUT and 1 COPY INTO.
Here is a code snipped I made to showcase this:
```import pyarrow as pa
import adbc_driver_snowflake.dbapi as adbc_snowflake
import numpy as np
# Create a 80M row test table
n = 80_000_000
data = pa.table(
{
"ID": pa.array(range(n), type=pa.int64()),
"VALUE": pa.array(np.random.randn(n), type=pa.float64()),
"CATEGORY": pa.array(["A", "B", "C", "D"] * (n // 4),
type=pa.string()),
}
)
print(
f"Table: {data.num_rows} rows, {round(data.nbytes / 1024**2)} MB in
memory"
)
db_kwargs = {
"adbc.snowflake.sql.account": "**redacted**",
"adbc.snowflake.sql.role": "**redacted**",
"username": "**redacted**",
"adbc.snowflake.sql.warehouse": "**redacted**_SMALL",
"adbc.snowflake.sql.auth_type": "auth_ext_browser",
"adbc.snowflake.sql.client_option.tracing": "debug",
}
print("Connecting...")
with adbc_snowflake.connect(db_kwargs=db_kwargs) as conn:
with conn.cursor() as cursor:
cursor.execute("USE DATABASE **redacted**")
cursor.execute("USE SCHEMA **redacted**")
# Try setting options on the statement
cursor.adbc_statement.set_options(
**{
"adbc.snowflake.statement.ingest_target_file_size":
"10485760",
"adbc.snowflake.statement.ingest_writer_concurrency": "4",
}
)
print("Ingesting...")
rows = cursor.adbc_ingest("ADBC_SPLIT_TEST", data,
mode="create_append")
print(f"Ingested {rows} rows")
conn.commit()
```
### Stack Trace
I didnt want to upload the full log information but here is some snippets:
```
INFO[0016]file_transfer_agent.go:756
gosnowflake.(*snowflakeFileTransferAgent).upload uploading 1 large files
LOG_SESSION_ID=4385957164815143
DEBU[0015]log.go:190 gosnowflake.(*defaultLogger).Debugf upload command
initiated - file count: 1, query ID: 01c5d751-0f13-dca0-000f-95042301408b,
encryption materials: 1
### How can we reproduce the bug?
Please use the script from above.
### Environment/Setup
`Python 3.12.9`
`adbc-driver-snowflake==1.11.0`
`adbc-driver-manager==1.11.0`
Using uv for package manager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
