djnavarro commented on code in PR #14514:
URL: https://github.com/apache/arrow/pull/14514#discussion_r1028641963
##########
r/vignettes/arrow.Rmd:
##########
@@ -1,227 +1,221 @@
---
-title: "Using the Arrow C++ Library in R"
-description: "This document describes the low-level interface to the Apache
Arrow C++ library in R and reviews the patterns and conventions of the R
package."
+title: "Get started with Arrow"
+description: >
+ An overview of the Apache Arrow project and the `arrow` R package
output: rmarkdown::html_vignette
-vignette: >
- %\VignetteIndexEntry{Using the Arrow C++ Library in R}
- %\VignetteEngine{knitr::rmarkdown}
- %\VignetteEncoding{UTF-8}
---
-The Apache Arrow C++ library provides rich, powerful features for working with
columnar data. The `arrow` R package provides both a low-level interface to the
C++ library and some higher-level, R-flavored tools for working with it. This
vignette provides an overview of how the pieces fit together, and it describes
the conventions that the classes and methods follow in R.
+Apache Arrow is a software development platform for building high performance
applications that process and transport large data sets. It is designed to
improve the performance of data analysis methods, and to increase the
efficiency of moving data from one system or programming language to another.
-# Features
+The `arrow` package provides a standard way to use Apache Arrow in R. It
provides a low-level interface to the [Arrow C++
library](https://arrow.apache.org/docs/cpp), and some higher-level tools for
working with it in a way designed to feel natural to R users. This article
provides an overview of how the pieces fit together, and it describes the
conventions that the classes and methods follow in R.
-## Multi-file datasets
+## Package conventions
-The `arrow` package lets you work efficiently with large, multi-file datasets
-using `dplyr` methods. See `vignette("dataset", package = "arrow")` for an
overview.
+The `arrow` R package builds on top of the Arrow C++ library, and C++ is an
object oriented language. As a consequence, the core logic of the Arrow C++
library is encapsulated in classes and methods. In the `arrow` R package these
are implemented as [`R6`](https://r6.r-lib.org) classes that all adopt
"TitleCase" naming conventions. Some examples of these include:
-## Reading and writing files
+- Two-dimensional, tabular data structures such as `Table`, `RecordBatch`, and
`Dataset`
+- One-dimensional, vector-like data structures such as `Array` and
`ChunkedArray`
+- Classes for reading, writing, and streaming data such as `ParquetFileReader`
and `CsvTableReader`
-`arrow` provides some simple functions for using the Arrow C++ library to read
and write files.
-These functions are designed to drop into your normal R workflow
-without requiring any knowledge of the Arrow C++ library
-and use naming conventions and arguments that follow popular R packages,
particularly `readr`.
-The readers return `data.frame`s
-(or if you use the `tibble` package, they will act like `tbl_df`s),
-and the writers take `data.frame`s.
+This low-level interface allows you to interact with the Arrow C++ library in
a very flexible way, but in many common situations you may never need to use it
at all, because `arrow` also supplies a high-level interface using functions
that follow a "snake_case" naming convention. Some examples of this include:
-Importantly, `arrow` provides basic read and write support for the [Apache
-Parquet](https://parquet.apache.org/) columnar data file format.
+- `arrow_table()` allows you to create Arrow tables without directly using the
`Table` object
+- `read_parquet()` allows you to open Parquet files without directly using the
`ParquetFileReader` object
-```r
-library(arrow)
-df <- read_parquet("path/to/file.parquet")
+All the examples used in this article rely on this high-level interface.
+
+For developers interested in learning more about the package structure, see
the [developer guide](./developing.html).
+
+
+## Tabular data in Arrow
+
+A critical component of Apache Arrow is its in-memory columnar format, a
standardized, language-agnostic specification for representing structured,
table-like datasets in-memory. In the `arrow` R package, the `Table` class is
used to store these objects. Tables are roughly analogous to data frames and
have similar behavior. The `arrow_table()` function allows you to generate new
Arrow Tables in much the same way that `data.frame()` is used to create new
data frames:
+
+```{r}
+library(arrow, warn.conflicts = FALSE)
+
+dat <- arrow_table(x = 1:3, y = c("a", "b", "c"))
+dat
```
-Just as you can read, you can write Parquet files:
+You can use `[` to specify subsets of Arrow Table in the same way you would
for a data frame:
-```r
-write_parquet(df, "path/to/different_file.parquet")
+```{r}
+dat[1:2, 1:2]
```
-The `arrow` package also includes a faster and more robust implementation of
the
-[Feather](https://github.com/wesm/feather) file format, providing
`read_feather()` and
-`write_feather()`. This implementation depends
-on the same underlying C++ library as the Python version does,
-resulting in more reliable and consistent behavior across the two languages, as
-well as [improved
performance](https://wesmckinney.com/blog/feather-arrow-future/).
-`arrow` also by default writes the Feather V2 format
-([the Arrow IPC file
format](https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format)),
-which supports a wider range of data types, as well as compression.
-
-For CSV and line-delimited JSON, there are `read_csv_arrow()` and
`read_json_arrow()`, respectively.
-While `read_csv_arrow()` currently has fewer parsing options for dealing with
-every CSV format variation in the wild, for the files it can read, it is
-often significantly faster than other R CSV readers, such as
-`base::read.csv`, `readr::read_csv`, and `data.table::fread`.
-
-## Working with Arrow data in Python
-
-Using [`reticulate`](https://rstudio.github.io/reticulate/), `arrow` lets you
-share data between R and Python (`pyarrow`) efficiently, enabling you to take
-advantage of the vibrant ecosystem of Python packages that build on top of
-Apache Arrow. See `vignette("python", package = "arrow")` for details.
+Along the same lines, the `$` operator can be used to extract named columns:
-## Access to Arrow messages, buffers, and streams
+```{r}
+dat$y
+```
-The `arrow` package also provides many lower-level bindings to the C++
library, which enable you
-to access and manipulate Arrow objects. You can use these to build connectors
-to other applications and services that use Arrow. One example is Spark: the
-[`sparklyr`](https://spark.rstudio.com/) package has support for using Arrow to
-move data to and from Spark, yielding [significant performance
-gains](https://arrow.apache.org/blog/2019/01/25/r-spark-improvements/).
+Note the output: individual columns in an Arrow Table are represented as
Chunked Arrays, which are one-dimensional data structures in Arrow that are
roughly analogous to vectors in R.
-# Object hierarchy
-
-## Metadata objects
-
-Arrow defines the following classes for representing metadata:
-
-| Class | Description | How to
create an instance |
-| ---------- | -------------------------------------------------- |
-------------------------------- |
-| `DataType` | attribute controlling how values are represented | functions
in `help("data-type")` |
-| `Field` | a character string name and a `DataType` |
`field(name, type)` |
-| `Schema` | list of `Field`s |
`schema(...)` |
-
-## Data objects
-
-Arrow defines the following classes for representing zero-dimensional (scalar),
-one-dimensional (array/vector-like), and two-dimensional (tabular/data
-frame-like) data:
-
-| Dim | Class | Description | How to
create an instance
|
-| --- | -------------- | ----------------------------------------- |
------------------------------------------------------------------------------------------------------|
-| 0 | `Scalar` | single value and its `DataType` |
`Scalar$create(value, type)`
|
-| 1 | `Array` | vector of values and its `DataType` |
`Array$create(vector, type)`
|
-| 1 | `ChunkedArray` | vectors of values and their `DataType` |
`ChunkedArray$create(..., type)` or alias `chunked_array(..., type)`
|
-| 2 | `RecordBatch` | list of `Array`s with a `Schema` |
`RecordBatch$create(...)` or alias `record_batch(...)`
|
-| 2 | `Table` | list of `ChunkedArray` with a `Schema` |
`Table$create(...)`, alias `arrow_table(...)`, or `arrow::read_*(file,
as_data_frame = FALSE)` |
-| 2 | `Dataset` | list of `Table`s with the same `Schema` |
`Dataset$create(sources, schema)` or alias `open_dataset(sources, schema)`
|
-
-Each of these is defined as an `R6` class in the `arrow` R package and
-corresponds to a class of the same name in the Arrow C++ library. The `arrow`
-package provides a variety of `R6` and S3 methods for interacting with
instances
-of these classes.
-
-For convenience, the `arrow` package also defines several synthetic classes
that
-do not exist in the C++ library, including:
-
-* `ArrowDatum`: inherited by `Scalar`, `Array`, and `ChunkedArray`
-* `ArrowTabular`: inherited by `RecordBatch` and `Table`
-* `ArrowObject`: inherited by all Arrow objects
-
-# Internals
-
-## Mapping of R <--> Arrow types
-
-Arrow has a rich data type system that includes direct parallels with R's data
types and much more.
+Tables are the primary way to represent rectangular data in-memory using
Arrow, but they are not the only rectangular data structure used by the Arrow
C++ library: there are also Datasets which are used for data stored on-disk
rather than in-memory, and Record Batches which are fundamental building blocks
but not typically used in data analysis.
-In the tables, entries with a `-` are not currently implemented.
+To learn more about the different data object classes in `arrow`, see the
article on [data objects](./data_objects.html).
-### R to Arrow
+## Converting Tables to data frames
-| R type | Arrow type |
-|--------------------------|------------|
-| logical | boolean |
-| integer | int32 |
-| double ("numeric") | float64^1^ |
-| character | utf8^2^ |
-| factor | dictionary |
-| raw | uint8 |
-| Date | date32 |
-| POSIXct | timestamp |
-| POSIXlt | struct |
-| data.frame | struct |
-| list^3^ | list |
-| bit64::integer64 | int64 |
-| hms::hms | time32 |
-| difftime | duration |
-| vctrs::vctrs_unspecified | null |
+Tables are a data structure used to represent rectangular data within memory
allocated by the Arrow C++ library, but they can be coerced to native R data
frames (or tibbles) using `as.data.frame()`
+```{r}
+as.data.frame(dat)
+```
+
+When this coercion takes place, each of the columns in the original Arrow
Table must be converted to native R data objects. In the `dat` Table, for
instance, `dat$x` is stored as the Arrow data type int32 inherited from C++,
which becomes an R integer type when `as.data.frame()` is called.
+It is possible to exercise fine grained control over this conversion process.
To learn more about the different types and how they are converted, see the
[data types](./data_types.html) article.
-^1^: `float64` and `double` are the same concept and data type in Arrow C++;
-however, only `float64()` is used in arrow as the function `double()` already
-exists in base R
-^2^: If the character vector exceeds 2GB of strings, it will be converted to a
-`large_utf8` Arrow type
+## Reading and writing data
-^3^: Only lists where all elements are the same type are able to be translated
-to Arrow list type (which is a "list of" some type).
+One of the main ways to use `arrow` is to read and write data files in
+several common formats. The `arrow` package supplies extremely fast CSV
reading and writing capabilities, but in addition supports data formats like
Parquet and Arrow (also called Feather) that are not widely supported in other
packages. In addition, the `arrow` package supports multi-file data sets in
which a single rectangular data set is stored across multiple files.
+### Individual files
-### Arrow to R
+When the goal is to read a single data file, there are several functions you
can use:
-| Arrow type | R type |
-|-------------------|------------------------------|
-| boolean | logical |
-| int8 | integer |
-| int16 | integer |
-| int32 | integer |
-| int64 | integer^1^ |
-| uint8 | integer |
-| uint16 | integer |
-| uint32 | integer^1^ |
-| uint64 | integer^1^ |
-| float16 | -^2^ |
-| float32 | double |
-| float64 | double |
-| utf8 | character |
-| large_utf8 | character |
-| binary | arrow_binary ^3^ |
-| large_binary | arrow_large_binary ^3^ |
-| fixed_size_binary | arrow_fixed_size_binary ^3^ |
-| date32 | Date |
-| date64 | POSIXct |
-| time32 | hms::hms |
-| time64 | hms::hms |
-| timestamp | POSIXct |
-| duration | difftime |
-| decimal | double |
-| dictionary | factor^4^ |
-| list | arrow_list ^5^ |
-| large_list | arrow_large_list ^5^ |
-| fixed_size_list | arrow_fixed_size_list ^5^ |
-| struct | data.frame |
-| null | vctrs::vctrs_unspecified |
-| map | arrow_list ^5^ |
-| union | -^2^ |
-
-^1^: These integer types may contain values that exceed the range of R's
-`integer` type (32-bit signed integer). When they do, `uint32` and `uint64`
are
-converted to `double` ("numeric") and `int64` is converted to
-`bit64::integer64`. This conversion can be disabled (so that `int64` always
-yields a `bit64::integer64` vector) by setting `options(arrow.int64_downcast =
FALSE)`.
+- `read_parquet()`: read a file in Parquet format
+- `read_feather()`: read a file in Arrow/Feather format
+- `read_delim_arrow()`: read a delimited text file
+- `read_csv_arrow()`: read a comma-separated values (CSV) file
+- `read_tsv_arrow()`: read a tab-separated values (TSV) file
+- `read_json_arrow()`: read a JSON data file
-^2^: Some Arrow data types do not currently have an R equivalent and will
raise an error
-if cast to or mapped to via a schema.
+In every case except JSON, there is a corresponding `write_*()` function
+that allows you to write data files in the appropriate format.
-^3^: `arrow*_binary` classes are implemented as lists of raw vectors.
+By default, the `read_*()` functions will return a data frame or tibble, but
you can also use them to read data into an Arrow Table. To do this, you need to
set the `as_data_frame` argument to `FALSE`.
-^4^: Due to the limitation of R factors, Arrow `dictionary` values are coerced
-to string when translated to R if they are not already strings.
+In the example below, we take the `starwars` data provided by the `dplyr`
package and write it to a Parquet file using `write_parquet()`
-^5^: `arrow*_list` classes are implemented as subclasses of `vctrs_list_of`
-with a `ptype` attribute set to what an empty Array of the value type converts
to.
+```{r}
+library(dplyr, warn.conflicts = FALSE)
+
+file_path <- tempfile(fileext = ".parquet")
+write_parquet(starwars, file_path)
+```
+We can then use `read_parquet()` to load the data from this file. As shown
below, the default behavior is to return a data frame (`sw_frame`) but when we
set `as_data_frame = FALSE` the data are read as an Arrow Table (`sw_table`):
+
+```{r}
+sw_frame <- read_parquet(file_path)
+sw_table <- read_parquet(file_path, as_data_frame = FALSE)
+sw_table
+```
+
+To learn more about reading and writing individual data files, see the
[read/write article](./read_write.html).
+
+### Multi-file data sets
+
+When a tabular data set becomes large, it is often good practice to partition
the data into meaningful subsets and store each one in a separate file. Among
other things, this means that if only one subset of the data are relevant to an
analysis, only one (smaller) file needs to be read. The `arrow` package
provides a convenient way to read, write, and analyze with data stored in this
fashion using the Dataset interface.
Review Comment:
Yeah, that's a nice thought. I'll update to use your suggested phrasing below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
