This is an automated email from the ASF dual-hosted git repository.
ianmcook pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-experiments.git
The following commit(s) were added to refs/heads/main by this push:
new 21449f5 Add IPC file to `rand-many-types` data (#48)
21449f5 is described below
commit 21449f5e96869cb7b33498e5fea06951038645e2
Author: Ian Cook <[email protected]>
AuthorDate: Thu Jun 19 16:24:27 2025 +0200
Add IPC file to `rand-many-types` data (#48)
---
data/rand-many-types/Makefile | 9 ++++++---
data/rand-many-types/README.md | 2 +-
data/rand-many-types/{Makefile => arrows-to-arrow.py} | 19 +++++++++++--------
data/rand-many-types/random.arrow | 3 +++
4 files changed, 21 insertions(+), 12 deletions(-)
diff --git a/data/rand-many-types/Makefile b/data/rand-many-types/Makefile
index ce42068..ff7dbf0 100644
--- a/data/rand-many-types/Makefile
+++ b/data/rand-many-types/Makefile
@@ -15,12 +15,15 @@
# specific language governing permissions and limitations
# under the License.
-all: arrow parquet duckdb
+all: arrows arrow parquet duckdb
-arrow: generate.py
+arrows: generate.py
python ./generate.py
-parquet: arrow
+arrow: arrows
+ python ./arrows-to-arrow.py
+
+parquet: arrows
python ./arrows-to-parquet.py
duckdb: parquet
diff --git a/data/rand-many-types/README.md b/data/rand-many-types/README.md
index f14b1f8..9fd8ad5 100644
--- a/data/rand-many-types/README.md
+++ b/data/rand-many-types/README.md
@@ -21,7 +21,7 @@
This directory contains a file `random.arrows` in Arrow IPC stream format with
randomly generated values in 20+ columns exercising many different Arrow data
types. The Python script `generate.py` that generated the data file is included.
-The same data is also included as a Parquet file (`random.parquet`) and as a
DuckDB database file (`random.duckdb`) as the table named `random`. The Python
and SQL used to generate these files is included.
+The same data is also included as a file in Arrow IPC file format
(`random.arrow`), as a Parquet file (`random.parquet`), and as a DuckDB
database file (`random.duckdb`) as the table named `random`. The Python and SQL
used to generate these files is included.
To re-generate the data files (for example, if you change `generate.py`),
diff --git a/data/rand-many-types/Makefile
b/data/rand-many-types/arrows-to-arrow.py
similarity index 72%
copy from data/rand-many-types/Makefile
copy to data/rand-many-types/arrows-to-arrow.py
index ce42068..78c81ff 100644
--- a/data/rand-many-types/Makefile
+++ b/data/rand-many-types/arrows-to-arrow.py
@@ -15,13 +15,16 @@
# specific language governing permissions and limitations
# under the License.
-all: arrow parquet duckdb
+import pyarrow as pa
-arrow: generate.py
- python ./generate.py
-parquet: arrow
- python ./arrows-to-parquet.py
-
-duckdb: parquet
- duckdb -f ./parquet-to-duckdb.sql
+with (
+ open("random.arrows", "rb") as f,
+ pa.ipc.open_stream(f) as reader,
+ pa.ipc.new_file("random.arrow", reader.schema) as writer
+):
+ while True:
+ try:
+ writer.write_batch(reader.read_next_batch())
+ except StopIteration:
+ break
diff --git a/data/rand-many-types/random.arrow
b/data/rand-many-types/random.arrow
new file mode 100644
index 0000000..6850007
--- /dev/null
+++ b/data/rand-many-types/random.arrow
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6750b23dc41231858f8b2b38a28bb2cb94c767ebfbf5fd6c413f6c1520206718
+size 13554522
