[GitHub] [beam] reuvenlax commented on a change in pull request #10767: Document Beam Schemas
reuvenlax commented on a change in pull request #10767: Document Beam Schemas URL: https://github.com/apache/beam/pull/10767#discussion_r386038309 ## File path: website/src/documentation/programming-guide.md ## @@ -1970,7 +1976,1076 @@ records.apply("WriteToText", See the [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/) page for a list of the currently available I/O transforms. -## 6. Data encoding and type safety {#data-encoding-and-type-safety} +## 6. Schemas {#schemas} +Often, the type of records being processed have an obvious structure. Common Beam sources produce +JSON, Avro, Protocol Buffer, or database row objects; all of these types have well defined structures, +structures that can often be determined by examining the type. Even within a pipeline, Simple Java POJOs +(or equivalent structures in other languages) are often used as intermediate types, and these also have a + clear structure that can be inferred by inspecting the class. By understanding the structure of a pipeline’s + records, we can provide much more concise APIs for data processing. + +### 6.1. What is a schema {#what-is-a-schema} +Most structured records share some common characteristics: +* They can be subdivided into separate named fields. Fields usually have string names, but sometimes - as in the case of indexed + tuples - have numerical indices instead. +* There is a confined list of primitive types that a field can have. These often match primitive types in most programming + languages: int, long, string, etc. +* Often a field type can be marked as optional (sometimes referred to as nullable) or required. + +In addition, often records have a nested structure. A nested structure occurs when a field itself has subfields so the +type of the field itself has a schema. Fields that are array or map types is also a common feature of these structured +records. + +For example, consider the following schema, representing actions in a fictitious e-commerce company: + +**Purchase** + + + + Field Name + Field Type + + + + + userId + STRING + + + itemId + INT64 + + + shippingAddress + ROW(ShippingAddress) + + + cost + INT64 + + + transactions + ARRAY[ROW(Transaction)] + + + + + +**ShippingAddress** + + + + Field Name + Field Type + + + + + streetAddress + STRING + + + city + STRING + + + state + nullable STRING + + + country + STRING + + + postCode + STRING + + + + + +**Transaction** + + + + Field Name + Field Type + + + + + bank + STRING + + + purchaseAmount + DOUBLE + + + + + +Purchase event records are represented by the aove purchase schema. Each purchase event contains a shipping address, which +is a nested row containing its own schema. Each purchase also contains a list of credit-card transactions +(a list, because a purchase might be split across multiple credit cards); each item in the transaction list is a row +with its own schema. + +This provides an abstract description of the types involved, one that is abstracted away from any specific programming +language. + +Schemas provide us a type-system for Beam records that is independent of any specific programming-language type. There +might be multiple Java classes that all have the same schema (for example a Protocol-Buffer class or a POJO class), +and Beam will allow us to seamlessly convert between these types. Schemas also provide a simple way to reason about +types across different programming-language APIs. + +A `PCollection` with a schema does not need to have a `Coder` specified, as Beam knows how to encode and decode +Schema rows. + +### 6.2. Schemas for programming language types {#schemas-for-pl-types} +While schemas themselves are language independent, they are designed to embed naturally into the programming languages +of the Beam SDK being used. This allows Beam users to continue using native types while reaping the advantage of +having Beam understand their element schemas. + + {:.language-java} + In Java you could use the following set of classes to represent the purchase schema. Beam will automatically + infer the correct schema based on the members of the class. + +```java +@DefaultSchema(JavaBeanSchema.class) +public class Purchase { + public String getUserId(); // Returns the id of the user who made the purchase. + public long getItemId(); // Returns the identifier of the item that was purchased. + public ShippingAddress getShippingAddress(); // Returns the shipping address, a nested type. + public long getCostCents(); // Returns the cost
[GitHub] [beam] reuvenlax commented on a change in pull request #10767: Document Beam Schemas
reuvenlax commented on a change in pull request #10767: Document Beam Schemas URL: https://github.com/apache/beam/pull/10767#discussion_r386038301 ## File path: website/src/documentation/programming-guide.md ## @@ -1970,7 +1976,1076 @@ records.apply("WriteToText", See the [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/) page for a list of the currently available I/O transforms. -## 6. Data encoding and type safety {#data-encoding-and-type-safety} +## 6. Schemas {#schemas} +Often, the type of records being processed have an obvious structure. Common Beam sources produce +JSON, Avro, Protocol Buffer, or database row objects; all of these types have well defined structures, +structures that can often be determined by examining the type. Even within a pipeline, Simple Java POJOs +(or equivalent structures in other languages) are often used as intermediate types, and these also have a + clear structure that can be inferred by inspecting the class. By understanding the structure of a pipeline’s + records, we can provide much more concise APIs for data processing. + +### 6.1. What is a schema {#what-is-a-schema} +Most structured records share some common characteristics: +* They can be subdivided into separate named fields. Fields usually have string names, but sometimes - as in the case of indexed + tuples - have numerical indices instead. +* There is a confined list of primitive types that a field can have. These often match primitive types in most programming + languages: int, long, string, etc. +* Often a field type can be marked as optional (sometimes referred to as nullable) or required. + +In addition, often records have a nested structure. A nested structure occurs when a field itself has subfields so the +type of the field itself has a schema. Fields that are array or map types is also a common feature of these structured +records. + +For example, consider the following schema, representing actions in a fictitious e-commerce company: + +**Purchase** + + + + Field Name + Field Type + + + + + userId + STRING + + + itemId + INT64 + + + shippingAddress + ROW(ShippingAddress) + + + cost + INT64 + + + transactions + ARRAY[ROW(Transaction)] + + + + + +**ShippingAddress** + + + + Field Name + Field Type + + + + + streetAddress + STRING + + + city + STRING + + + state + nullable STRING + + + country + STRING + + + postCode + STRING + + + + + +**Transaction** + + + + Field Name + Field Type + + + + + bank + STRING + + + purchaseAmount + DOUBLE + + + + + +Purchase event records are represented by the aove purchase schema. Each purchase event contains a shipping address, which +is a nested row containing its own schema. Each purchase also contains a list of credit-card transactions +(a list, because a purchase might be split across multiple credit cards); each item in the transaction list is a row +with its own schema. + +This provides an abstract description of the types involved, one that is abstracted away from any specific programming +language. + +Schemas provide us a type-system for Beam records that is independent of any specific programming-language type. There +might be multiple Java classes that all have the same schema (for example a Protocol-Buffer class or a POJO class), +and Beam will allow us to seamlessly convert between these types. Schemas also provide a simple way to reason about +types across different programming-language APIs. + +A `PCollection` with a schema does not need to have a `Coder` specified, as Beam knows how to encode and decode +Schema rows. + +### 6.2. Schemas for programming language types {#schemas-for-pl-types} +While schemas themselves are language independent, they are designed to embed naturally into the programming languages +of the Beam SDK being used. This allows Beam users to continue using native types while reaping the advantage of +having Beam understand their element schemas. + + {:.language-java} + In Java you could use the following set of classes to represent the purchase schema. Beam will automatically + infer the correct schema based on the members of the class. + +```java +@DefaultSchema(JavaBeanSchema.class) +public class Purchase { + public String getUserId(); // Returns the id of the user who made the purchase. + public long getItemId(); // Returns the identifier of the item that was purchased. + public ShippingAddress getShippingAddress(); // Returns the shipping address, a nested type. + public long getCostCents(); // Returns the cost
[GitHub] [beam] reuvenlax commented on a change in pull request #10767: Document Beam Schemas
reuvenlax commented on a change in pull request #10767: Document Beam Schemas URL: https://github.com/apache/beam/pull/10767#discussion_r386038304 ## File path: website/src/documentation/programming-guide.md ## @@ -1970,7 +1976,1076 @@ records.apply("WriteToText", See the [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/) page for a list of the currently available I/O transforms. -## 6. Data encoding and type safety {#data-encoding-and-type-safety} +## 6. Schemas {#schemas} +Often, the type of records being processed have an obvious structure. Common Beam sources produce +JSON, Avro, Protocol Buffer, or database row objects; all of these types have well defined structures, +structures that can often be determined by examining the type. Even within a pipeline, Simple Java POJOs +(or equivalent structures in other languages) are often used as intermediate types, and these also have a + clear structure that can be inferred by inspecting the class. By understanding the structure of a pipeline’s + records, we can provide much more concise APIs for data processing. + +### 6.1. What is a schema {#what-is-a-schema} +Most structured records share some common characteristics: +* They can be subdivided into separate named fields. Fields usually have string names, but sometimes - as in the case of indexed + tuples - have numerical indices instead. +* There is a confined list of primitive types that a field can have. These often match primitive types in most programming + languages: int, long, string, etc. +* Often a field type can be marked as optional (sometimes referred to as nullable) or required. + +In addition, often records have a nested structure. A nested structure occurs when a field itself has subfields so the +type of the field itself has a schema. Fields that are array or map types is also a common feature of these structured +records. + +For example, consider the following schema, representing actions in a fictitious e-commerce company: + +**Purchase** + + + + Field Name + Field Type + + + + + userId + STRING + + + itemId + INT64 + + + shippingAddress + ROW(ShippingAddress) + + + cost + INT64 + + + transactions + ARRAY[ROW(Transaction)] + + + + + +**ShippingAddress** + + + + Field Name + Field Type + + + + + streetAddress + STRING + + + city + STRING + + + state + nullable STRING + + + country + STRING + + + postCode + STRING + + + + + +**Transaction** + + + + Field Name + Field Type + + + + + bank + STRING + + + purchaseAmount + DOUBLE + + + + + +Purchase event records are represented by the aove purchase schema. Each purchase event contains a shipping address, which +is a nested row containing its own schema. Each purchase also contains a list of credit-card transactions +(a list, because a purchase might be split across multiple credit cards); each item in the transaction list is a row +with its own schema. + +This provides an abstract description of the types involved, one that is abstracted away from any specific programming +language. + +Schemas provide us a type-system for Beam records that is independent of any specific programming-language type. There +might be multiple Java classes that all have the same schema (for example a Protocol-Buffer class or a POJO class), +and Beam will allow us to seamlessly convert between these types. Schemas also provide a simple way to reason about +types across different programming-language APIs. + +A `PCollection` with a schema does not need to have a `Coder` specified, as Beam knows how to encode and decode +Schema rows. + +### 6.2. Schemas for programming language types {#schemas-for-pl-types} +While schemas themselves are language independent, they are designed to embed naturally into the programming languages +of the Beam SDK being used. This allows Beam users to continue using native types while reaping the advantage of +having Beam understand their element schemas. + + {:.language-java} + In Java you could use the following set of classes to represent the purchase schema. Beam will automatically + infer the correct schema based on the members of the class. + +```java +@DefaultSchema(JavaBeanSchema.class) +public class Purchase { + public String getUserId(); // Returns the id of the user who made the purchase. + public long getItemId(); // Returns the identifier of the item that was purchased. + public ShippingAddress getShippingAddress(); // Returns the shipping address, a nested type. + public long getCostCents(); // Returns the cost
[GitHub] [beam] reuvenlax commented on a change in pull request #10767: Document Beam Schemas
reuvenlax commented on a change in pull request #10767: Document Beam Schemas URL: https://github.com/apache/beam/pull/10767#discussion_r386038297 ## File path: website/src/documentation/programming-guide.md ## @@ -1970,7 +1976,1076 @@ records.apply("WriteToText", See the [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/) page for a list of the currently available I/O transforms. -## 6. Data encoding and type safety {#data-encoding-and-type-safety} +## 6. Schemas {#schemas} +Often, the type of records being processed have an obvious structure. Common Beam sources produce +JSON, Avro, Protocol Buffer, or database row objects; all of these types have well defined structures, +structures that can often be determined by examining the type. Even within a pipeline, Simple Java POJOs +(or equivalent structures in other languages) are often used as intermediate types, and these also have a + clear structure that can be inferred by inspecting the class. By understanding the structure of a pipeline’s + records, we can provide much more concise APIs for data processing. + +### 6.1. What is a schema {#what-is-a-schema} +Most structured records share some common characteristics: +* They can be subdivided into separate named fields. Fields usually have string names, but sometimes - as in the case of indexed + tuples - have numerical indices instead. +* There is a confined list of primitive types that a field can have. These often match primitive types in most programming + languages: int, long, string, etc. +* Often a field type can be marked as optional (sometimes referred to as nullable) or required. + +In addition, often records have a nested structure. A nested structure occurs when a field itself has subfields so the +type of the field itself has a schema. Fields that are array or map types is also a common feature of these structured +records. + +For example, consider the following schema, representing actions in a fictitious e-commerce company: + +**Purchase** + + + + Field Name + Field Type + + + + + userId + STRING + + + itemId + INT64 + + + shippingAddress + ROW(ShippingAddress) + + + cost + INT64 + + + transactions + ARRAY[ROW(Transaction)] + + + + + +**ShippingAddress** + + + + Field Name + Field Type + + + + + streetAddress + STRING + + + city + STRING + + + state + nullable STRING + + + country + STRING + + + postCode + STRING + + + + + +**Transaction** + + + + Field Name + Field Type + + + + + bank + STRING + + + purchaseAmount + DOUBLE + + + + + +Purchase event records are represented by the aove purchase schema. Each purchase event contains a shipping address, which +is a nested row containing its own schema. Each purchase also contains a list of credit-card transactions +(a list, because a purchase might be split across multiple credit cards); each item in the transaction list is a row +with its own schema. + +This provides an abstract description of the types involved, one that is abstracted away from any specific programming +language. + +Schemas provide us a type-system for Beam records that is independent of any specific programming-language type. There +might be multiple Java classes that all have the same schema (for example a Protocol-Buffer class or a POJO class), +and Beam will allow us to seamlessly convert between these types. Schemas also provide a simple way to reason about +types across different programming-language APIs. + +A `PCollection` with a schema does not need to have a `Coder` specified, as Beam knows how to encode and decode +Schema rows. + +### 6.2. Schemas for programming language types {#schemas-for-pl-types} Review comment: hmmm this section is trying to get across the basic theory that schemas are _the_ types, and if multiple Pl types have the same schema then they are interchangeable. The goal of this section isn't a reference, though we use some examples to illustrate the point. The other two sections you mention are meant to be more of a reference. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #10767: Document Beam Schemas
reuvenlax commented on a change in pull request #10767: Document Beam Schemas URL: https://github.com/apache/beam/pull/10767#discussion_r386038150 ## File path: website/src/documentation/programming-guide.md ## @@ -1970,7 +1976,1076 @@ records.apply("WriteToText", See the [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/) page for a list of the currently available I/O transforms. -## 6. Data encoding and type safety {#data-encoding-and-type-safety} +## 6. Schemas {#schemas} +Often, the type of records being processed have an obvious structure. Common Beam sources produce +JSON, Avro, Protocol Buffer, or database row objects; all of these types have well defined structures, +structures that can often be determined by examining the type. Even within a pipeline, Simple Java POJOs +(or equivalent structures in other languages) are often used as intermediate types, and these also have a + clear structure that can be inferred by inspecting the class. By understanding the structure of a pipeline’s + records, we can provide much more concise APIs for data processing. + +### 6.1. What is a schema {#what-is-a-schema} +Most structured records share some common characteristics: +* They can be subdivided into separate named fields. Fields usually have string names, but sometimes - as in the case of indexed + tuples - have numerical indices instead. +* There is a confined list of primitive types that a field can have. These often match primitive types in most programming + languages: int, long, string, etc. +* Often a field type can be marked as optional (sometimes referred to as nullable) or required. + +In addition, often records have a nested structure. A nested structure occurs when a field itself has subfields so the +type of the field itself has a schema. Fields that are array or map types is also a common feature of these structured +records. + +For example, consider the following schema, representing actions in a fictitious e-commerce company: + +**Purchase** + + + + Field Name + Field Type + + + + + userId + STRING + + + itemId + INT64 + + + shippingAddress + ROW(ShippingAddress) + + + cost + INT64 + + + transactions + ARRAY[ROW(Transaction)] + + + + + +**ShippingAddress** + + + + Field Name + Field Type + + + + + streetAddress + STRING + + + city + STRING + + + state + nullable STRING + + + country + STRING + + + postCode + STRING + + + + + +**Transaction** + + + + Field Name + Field Type + + + + + bank + STRING + + + purchaseAmount + DOUBLE + + + + + +Purchase event records are represented by the aove purchase schema. Each purchase event contains a shipping address, which +is a nested row containing its own schema. Each purchase also contains a list of credit-card transactions +(a list, because a purchase might be split across multiple credit cards); each item in the transaction list is a row +with its own schema. + +This provides an abstract description of the types involved, one that is abstracted away from any specific programming +language. + +Schemas provide us a type-system for Beam records that is independent of any specific programming-language type. There +might be multiple Java classes that all have the same schema (for example a Protocol-Buffer class or a POJO class), +and Beam will allow us to seamlessly convert between these types. Schemas also provide a simple way to reason about +types across different programming-language APIs. + +A `PCollection` with a schema does not need to have a `Coder` specified, as Beam knows how to encode and decode +Schema rows. + +### 6.2. Schemas for programming language types {#schemas-for-pl-types} +While schemas themselves are language independent, they are designed to embed naturally into the programming languages +of the Beam SDK being used. This allows Beam users to continue using native types while reaping the advantage of +having Beam understand their element schemas. + + {:.language-java} + In Java you could use the following set of classes to represent the purchase schema. Beam will automatically + infer the correct schema based on the members of the class. + +```java +@DefaultSchema(JavaBeanSchema.class) +public class Purchase { + public String getUserId(); // Returns the id of the user who made the purchase. + public long getItemId(); // Returns the identifier of the item that was purchased. + public ShippingAddress getShippingAddress(); // Returns the shipping address, a nested type. + public long getCostCents(); // Returns the cost
[GitHub] [beam] reuvenlax commented on a change in pull request #10767: Document Beam Schemas
reuvenlax commented on a change in pull request #10767: Document Beam Schemas URL: https://github.com/apache/beam/pull/10767#discussion_r386037819 ## File path: website/src/documentation/programming-guide.md ## @@ -1970,7 +1976,1076 @@ records.apply("WriteToText", See the [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/) page for a list of the currently available I/O transforms. -## 6. Data encoding and type safety {#data-encoding-and-type-safety} +## 6. Schemas {#schemas} +Often, the type of records being processed have an obvious structure. Common Beam sources produce +JSON, Avro, Protocol Buffer, or database row objects; all of these types have well defined structures, +structures that can often be determined by examining the type. Even within a pipeline, Simple Java POJOs Review comment: I meant within a pipeline as that's where users create these intermediate objects - maybe SDK pipeline? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #10767: Document Beam Schemas
reuvenlax commented on a change in pull request #10767: Document Beam Schemas URL: https://github.com/apache/beam/pull/10767#discussion_r386037870 ## File path: website/src/documentation/programming-guide.md ## @@ -1970,7 +1976,1076 @@ records.apply("WriteToText", See the [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/) page for a list of the currently available I/O transforms. -## 6. Data encoding and type safety {#data-encoding-and-type-safety} +## 6. Schemas {#schemas} +Often, the type of records being processed have an obvious structure. Common Beam sources produce +JSON, Avro, Protocol Buffer, or database row objects; all of these types have well defined structures, +structures that can often be determined by examining the type. Even within a pipeline, Simple Java POJOs +(or equivalent structures in other languages) are often used as intermediate types, and these also have a + clear structure that can be inferred by inspecting the class. By understanding the structure of a pipeline’s + records, we can provide much more concise APIs for data processing. + +### 6.1. What is a schema {#what-is-a-schema} +Most structured records share some common characteristics: +* They can be subdivided into separate named fields. Fields usually have string names, but sometimes - as in the case of indexed + tuples - have numerical indices instead. +* There is a confined list of primitive types that a field can have. These often match primitive types in most programming + languages: int, long, string, etc. +* Often a field type can be marked as optional (sometimes referred to as nullable) or required. + +In addition, often records have a nested structure. A nested structure occurs when a field itself has subfields so the +type of the field itself has a schema. Fields that are array or map types is also a common feature of these structured +records. + +For example, consider the following schema, representing actions in a fictitious e-commerce company: + +**Purchase** + + + + Field Name + Field Type + + + + + userId + STRING + + + itemId + INT64 + + + shippingAddress + ROW(ShippingAddress) + + + cost + INT64 + + + transactions + ARRAY[ROW(Transaction)] + + + + Review comment: This is true for other tables in this doc as well (e.g. coders). Maybe we should figure out a way to fix all of them at once? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #10767: Document Beam Schemas
reuvenlax commented on a change in pull request #10767: Document Beam Schemas URL: https://github.com/apache/beam/pull/10767#discussion_r386037889 ## File path: website/src/documentation/programming-guide.md ## @@ -1970,7 +1976,1076 @@ records.apply("WriteToText", See the [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/) page for a list of the currently available I/O transforms. -## 6. Data encoding and type safety {#data-encoding-and-type-safety} +## 6. Schemas {#schemas} +Often, the type of records being processed have an obvious structure. Common Beam sources produce +JSON, Avro, Protocol Buffer, or database row objects; all of these types have well defined structures, +structures that can often be determined by examining the type. Even within a pipeline, Simple Java POJOs +(or equivalent structures in other languages) are often used as intermediate types, and these also have a + clear structure that can be inferred by inspecting the class. By understanding the structure of a pipeline’s + records, we can provide much more concise APIs for data processing. + +### 6.1. What is a schema {#what-is-a-schema} +Most structured records share some common characteristics: +* They can be subdivided into separate named fields. Fields usually have string names, but sometimes - as in the case of indexed + tuples - have numerical indices instead. +* There is a confined list of primitive types that a field can have. These often match primitive types in most programming + languages: int, long, string, etc. +* Often a field type can be marked as optional (sometimes referred to as nullable) or required. + +In addition, often records have a nested structure. A nested structure occurs when a field itself has subfields so the +type of the field itself has a schema. Fields that are array or map types is also a common feature of these structured +records. + +For example, consider the following schema, representing actions in a fictitious e-commerce company: + +**Purchase** + + + + Field Name + Field Type + + + + + userId + STRING + + + itemId + INT64 + + + shippingAddress + ROW(ShippingAddress) + + + cost + INT64 + + + transactions + ARRAY[ROW(Transaction)] + + + + + +**ShippingAddress** + + + + Field Name + Field Type + + + + + streetAddress + STRING + + + city + STRING + + + state + nullable STRING + + + country + STRING + + + postCode + STRING + + + + + +**Transaction** + + + + Field Name + Field Type + + + + + bank + STRING + + + purchaseAmount + DOUBLE + + + + + +Purchase event records are represented by the aove purchase schema. Each purchase event contains a shipping address, which Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #10767: Document Beam Schemas
reuvenlax commented on a change in pull request #10767: Document Beam Schemas URL: https://github.com/apache/beam/pull/10767#discussion_r386037750 ## File path: website/src/documentation/programming-guide.md ## @@ -1970,7 +1976,1076 @@ records.apply("WriteToText", See the [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/) page for a list of the currently available I/O transforms. -## 6. Data encoding and type safety {#data-encoding-and-type-safety} +## 6. Schemas {#schemas} +Often, the type of records being processed have an obvious structure. Common Beam sources produce +JSON, Avro, Protocol Buffer, or database row objects; all of these types have well defined structures, +structures that can often be determined by examining the type. Even within a pipeline, Simple Java POJOs +(or equivalent structures in other languages) are often used as intermediate types, and these also have a Review comment: I think better to wait until we add Python schema docs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #10767: Document Beam Schemas
reuvenlax commented on a change in pull request #10767: Document Beam Schemas URL: https://github.com/apache/beam/pull/10767#discussion_r386037654 ## File path: website/src/documentation/programming-guide.md ## @@ -1970,7 +1976,1076 @@ records.apply("WriteToText", See the [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/) page for a list of the currently available I/O transforms. -## 6. Data encoding and type safety {#data-encoding-and-type-safety} +## 6. Schemas {#schemas} +Often, the type of records being processed have an obvious structure. Common Beam sources produce +JSON, Avro, Protocol Buffer, or database row objects; all of these types have well defined structures, +structures that can often be determined by examining the type. Even within a pipeline, Simple Java POJOs +(or equivalent structures in other languages) are often used as intermediate types, and these also have a + clear structure that can be inferred by inspecting the class. By understanding the structure of a pipeline’s + records, we can provide much more concise APIs for data processing. + +### 6.1. What is a schema {#what-is-a-schema} +Most structured records share some common characteristics: +* They can be subdivided into separate named fields. Fields usually have string names, but sometimes - as in the case of indexed + tuples - have numerical indices instead. +* There is a confined list of primitive types that a field can have. These often match primitive types in most programming + languages: int, long, string, etc. +* Often a field type can be marked as optional (sometimes referred to as nullable) or required. + +In addition, often records have a nested structure. A nested structure occurs when a field itself has subfields so the +type of the field itself has a schema. Fields that are array or map types is also a common feature of these structured +records. + +For example, consider the following schema, representing actions in a fictitious e-commerce company: + +**Purchase** + + + + Field Name + Field Type + + + + + userId + STRING + + + itemId + INT64 + + + shippingAddress + ROW(ShippingAddress) + + + cost + INT64 + + + transactions + ARRAY[ROW(Transaction)] + + + + + +**ShippingAddress** + + + + Field Name + Field Type + + + + + streetAddress + STRING + + + city + STRING + + + state + nullable STRING + + + country + STRING + + + postCode + STRING + + + + + +**Transaction** + + + + Field Name + Field Type + + + + + bank + STRING + + + purchaseAmount + DOUBLE + + + + + +Purchase event records are represented by the aove purchase schema. Each purchase event contains a shipping address, which +is a nested row containing its own schema. Each purchase also contains a list of credit-card transactions +(a list, because a purchase might be split across multiple credit cards); each item in the transaction list is a row +with its own schema. + +This provides an abstract description of the types involved, one that is abstracted away from any specific programming +language. + +Schemas provide us a type-system for Beam records that is independent of any specific programming-language type. There +might be multiple Java classes that all have the same schema (for example a Protocol-Buffer class or a POJO class), +and Beam will allow us to seamlessly convert between these types. Schemas also provide a simple way to reason about +types across different programming-language APIs. + +A `PCollection` with a schema does not need to have a `Coder` specified, as Beam knows how to encode and decode +Schema rows. + +### 6.2. Schemas for programming language types {#schemas-for-pl-types} +While schemas themselves are language independent, they are designed to embed naturally into the programming languages +of the Beam SDK being used. This allows Beam users to continue using native types while reaping the advantage of +having Beam understand their element schemas. + + {:.language-java} + In Java you could use the following set of classes to represent the purchase schema. Beam will automatically + infer the correct schema based on the members of the class. + +```java +@DefaultSchema(JavaBeanSchema.class) +public class Purchase { + public String getUserId(); // Returns the id of the user who made the purchase. + public long getItemId(); // Returns the identifier of the item that was purchased. + public ShippingAddress getShippingAddress(); // Returns the shipping address, a nested type. + public long getCostCents(); // Returns the cost
[GitHub] [beam] reuvenlax commented on a change in pull request #10767: Document Beam Schemas
reuvenlax commented on a change in pull request #10767: Document Beam Schemas URL: https://github.com/apache/beam/pull/10767#discussion_r386037615 ## File path: website/src/documentation/programming-guide.md ## @@ -1970,7 +1976,1076 @@ records.apply("WriteToText", See the [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/) page for a list of the currently available I/O transforms. -## 6. Data encoding and type safety {#data-encoding-and-type-safety} +## 6. Schemas {#schemas} +Often, the type of records being processed have an obvious structure. Common Beam sources produce +JSON, Avro, Protocol Buffer, or database row objects; all of these types have well defined structures, +structures that can often be determined by examining the type. Even within a pipeline, Simple Java POJOs +(or equivalent structures in other languages) are often used as intermediate types, and these also have a + clear structure that can be inferred by inspecting the class. By understanding the structure of a pipeline’s + records, we can provide much more concise APIs for data processing. + +### 6.1. What is a schema {#what-is-a-schema} +Most structured records share some common characteristics: +* They can be subdivided into separate named fields. Fields usually have string names, but sometimes - as in the case of indexed + tuples - have numerical indices instead. +* There is a confined list of primitive types that a field can have. These often match primitive types in most programming + languages: int, long, string, etc. +* Often a field type can be marked as optional (sometimes referred to as nullable) or required. + +In addition, often records have a nested structure. A nested structure occurs when a field itself has subfields so the +type of the field itself has a schema. Fields that are array or map types is also a common feature of these structured +records. + +For example, consider the following schema, representing actions in a fictitious e-commerce company: + +**Purchase** + + + + Field Name + Field Type + + + + + userId + STRING + + + itemId + INT64 + + + shippingAddress + ROW(ShippingAddress) + + + cost + INT64 + + + transactions + ARRAY[ROW(Transaction)] + + + + + +**ShippingAddress** + + + + Field Name + Field Type + + + + + streetAddress + STRING + + + city + STRING + + + state + nullable STRING + + + country + STRING + + + postCode + STRING + + + + + +**Transaction** + + + + Field Name + Field Type + + + + + bank + STRING + + + purchaseAmount + DOUBLE + + + + + +Purchase event records are represented by the aove purchase schema. Each purchase event contains a shipping address, which +is a nested row containing its own schema. Each purchase also contains a list of credit-card transactions +(a list, because a purchase might be split across multiple credit cards); each item in the transaction list is a row +with its own schema. + +This provides an abstract description of the types involved, one that is abstracted away from any specific programming +language. + +Schemas provide us a type-system for Beam records that is independent of any specific programming-language type. There +might be multiple Java classes that all have the same schema (for example a Protocol-Buffer class or a POJO class), +and Beam will allow us to seamlessly convert between these types. Schemas also provide a simple way to reason about +types across different programming-language APIs. + +A `PCollection` with a schema does not need to have a `Coder` specified, as Beam knows how to encode and decode +Schema rows. + +### 6.2. Schemas for programming language types {#schemas-for-pl-types} +While schemas themselves are language independent, they are designed to embed naturally into the programming languages +of the Beam SDK being used. This allows Beam users to continue using native types while reaping the advantage of +having Beam understand their element schemas. + + {:.language-java} + In Java you could use the following set of classes to represent the purchase schema. Beam will automatically + infer the correct schema based on the members of the class. + +```java +@DefaultSchema(JavaBeanSchema.class) +public class Purchase { + public String getUserId(); // Returns the id of the user who made the purchase. + public long getItemId(); // Returns the identifier of the item that was purchased. + public ShippingAddress getShippingAddress(); // Returns the shipping address, a nested type. + public long getCostCents(); // Returns the cost