djouallah opened a new issue, #1004:
URL: https://github.com/apache/iceberg-python/issues/1004
### Feature Request / Improvement
currently using this in a an environment with limited RAM
```
df=final.arrow()
catalog , table_location = connect_catalog(storage)
catalog.create_table_if_not_exists(db+".scada",schema=df.schema,location=
table_location+f'/{db}/scada' )
catalog.load_table(db+".scada").append(df)
```
I get sometimes out of memory because arrow table needs to be fully
materialized in memory, I can generate recordbatch from my source system which
will use less memory,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
