PhilippeMoussalli commented on code in PR #22587:
URL: https://github.com/apache/beam/pull/22587#discussion_r964470743
##########
examples/notebooks/beam-ml/dataframe_api_preprocessing.ipynb:
##########
@@ -0,0 +1,1907 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Overview\n",
+ "\n",
+ "One of the most common tools used for data exploration and
pre-processing is [pandas
DataFrames](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html).
Pandas has become very popular for its ease of use. It has very intuitive
methods to perform common analytical tasks and data pre-processing. \n",
+ "\n",
+ "Pandas loads all of the data into memory on a single machine (one
node) for rapid execution. This works well when dealing with small-scale
datasets. However, many projects involve datasets that can grow too big to fit
in memory. These use cases generally require the usage of parallel data
processing frameworks such as Apache Beam.\n",
+ "\n",
+ "\n",
+ "## Beam DataFrames\n",
+ "\n",
+ "\n",
+ "Beam DataFrames provide a pandas-like DataFrame\n",
+ "API to declare and define Beam processing pipelines. It provides a
familiar interface for machine learning practioners to build complex
data-processing pipelines by only invoking standard pandas commands.\n",
+ "\n",
+ "> ℹ️ To learn more about Beam DataFrames, take a look at the\n",
+ "[Beam DataFrames
overview](https://beam.apache.org/documentation/dsls/dataframes/overview)
page.\n",
+ "\n",
+ "## Tutorial outline\n",
+ "\n",
+ "In this notebook, we walk through the use of the Beam DataFrames API
to perform common data exploration as well as pre-processing steps that are
necessary to prepare your dataset for machine learning model training and
inference, such as: \n",
+ "\n",
+ "* Removing unwanted columns.\n",
+ "* One-hot encoding categorical columns.\n",
+ "* Normalizing numerical columns.\n",
+ "\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "iFZC1inKuUCy"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Installation\n",
+ "\n",
+ "First, we need to install Apache Beam with the `interactive`
component to be able to use the Interactive runner. The latest implemented
DataFrames API methods invoked in this notebook are available in Beam
<b>2.41</b> or later.\n"
Review Comment:
Good point :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
