pitrou commented on a change in pull request #10934: URL: https://github.com/apache/arrow/pull/10934#discussion_r712353459
########## File path: experimental/computeir/Literal.fbs ########## @@ -0,0 +1,203 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +namespace org.apache.arrow.computeir.flatbuf; + +table ListLiteral { + values: [Literal]; +} + +table StructLiteral { + values: [KeyValue]; +} + +table KeyValue { + key: string; + value: Literal; +} + +table MapLiteral { + values: [KeyValue]; Review comment: `KeyValue` forces the key to be a string but a map can accept any type as key type, is it deliberate to restrict MapLiteral to string keys? ########## File path: experimental/computeir/Literal.fbs ########## @@ -0,0 +1,203 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +namespace org.apache.arrow.computeir.flatbuf; + +table ListLiteral { + values: [Literal]; +} + +table StructLiteral { + values: [KeyValue]; +} + +table KeyValue { + key: string; + value: Literal; +} + +table MapLiteral { + values: [KeyValue]; +} + +table Int8Literal { + value: int8; +} + +table Int16Literal { + value: int16; +} + +table Int32Literal { + value: int32; +} + +table Int64Literal { + value: int64; +} + +table UInt8Literal { + value: uint8; +} + +table UInt16Literal { + value: uint16; +} + +table UInt32Literal { + value: uint32; +} + +table UInt64Literal { + value: uint64; +} + +table Float16Literal { + value: uint16; +} + +table Float32Literal { + value: float32; +} + +table Float64Literal { + value: float64; +} + +table DecimalLiteral { + value: [byte]; + scale: uint8; + precision: uint8; +} + +table BooleanLiteral { + value: bool; +} + +table NullLiteral {} + +enum DateUnit : uint8 { + DAY, + MILLISECOND, +} + +table DateLiteral { + value: int64; + + // unit of `value` + unit: DateUnit = MILLISECOND; +} + +enum TimeUnit : uint8 { Review comment: `Schema.fbs` already has `DateUnit` and `TimeUnit`, is the redefinition deliberate? ########## File path: experimental/computeir/Literal.fbs ########## @@ -0,0 +1,203 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +namespace org.apache.arrow.computeir.flatbuf; + +table ListLiteral { + values: [Literal]; +} + +table StructLiteral { + values: [KeyValue]; +} + +table KeyValue { + key: string; + value: Literal; +} + +table MapLiteral { + values: [KeyValue]; +} + +table Int8Literal { + value: int8; +} + +table Int16Literal { + value: int16; +} + +table Int32Literal { + value: int32; +} + +table Int64Literal { + value: int64; +} + +table UInt8Literal { + value: uint8; +} + +table UInt16Literal { + value: uint16; +} + +table UInt32Literal { + value: uint32; +} + +table UInt64Literal { + value: uint64; +} + +table Float16Literal { + value: uint16; +} + +table Float32Literal { + value: float32; +} + +table Float64Literal { + value: float64; +} + +table DecimalLiteral { + value: [byte]; + scale: uint8; + precision: uint8; Review comment: Is it deliberate to use `uint8` for scale and precision rather than `int` as in `Decimal`? At least the scale should probably be signed -- negative scales are allowed in Arrow, even though they may not be very compatible with other systems. ########## File path: experimental/computeir/Expression.fbs ########## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +include "../../format/Schema.fbs"; +include "Literal.fbs"; + +namespace org.apache.arrow.computeir.flatbuf; + +/// Access a value for a given map key +table MapKey { + key: string (required); +} + +/// Struct field access +table StructField { + /// The position of the field in the struct schema + position: uint32; +} + +/// Zero-based array index +table ArraySubscript { + position: uint32; +} + +/// Zero-based range of elements in an array +table ArraySlice { + /// The start of an array slice, inclusive + start_inclusive: uint32; + /// The end of an array slice, exclusive + end_exclusive: uint32; +} + +/// Field name in a relation +table FieldName { + position: uint32; +} + +/// A union of possible dereference operations +union Deref { + /// Access a value for a given map key + MapKey, + /// Access the value at a struct field + StructField, + /// Access the element at a given index in an array + ArraySubscript, + /// Access a range of elements in an array + ArraySlice, + /// Access a field of a relation + FieldName, +} + +/// Access the data of a field +table FieldRef { + /// A sequence of field names to allow referencing potentially nested fields Review comment: Or is it me, or is this comment outdated? ########## File path: experimental/computeir/Expression.fbs ########## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +include "../../format/Schema.fbs"; +include "Literal.fbs"; + +namespace org.apache.arrow.computeir.flatbuf; + +/// Access a value for a given map key +table MapKey { + key: string (required); +} + +/// Struct field access +table StructField { + /// The position of the field in the struct schema + position: uint32; +} + +/// Zero-based array index +table ArraySubscript { + position: uint32; +} + +/// Zero-based range of elements in an array +table ArraySlice { + /// The start of an array slice, inclusive + start_inclusive: uint32; + /// The end of an array slice, exclusive + end_exclusive: uint32; +} + +/// Field name in a relation +table FieldName { + position: uint32; +} + +/// A union of possible dereference operations +union Deref { + /// Access a value for a given map key + MapKey, + /// Access the value at a struct field + StructField, + /// Access the element at a given index in an array + ArraySubscript, + /// Access a range of elements in an array + ArraySlice, + /// Access a field of a relation + FieldName, +} + +/// Access the data of a field +table FieldRef { + /// A sequence of field names to allow referencing potentially nested fields + ref: Deref (required); + /// For Expressions which might reference fields in multiple Relations, + /// this index may be provided to indicate which Relation's fields + /// `path` points into. For example in the case of a join, Review comment: What is `path` here? ########## File path: experimental/computeir/Expression.fbs ########## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +include "../../format/Schema.fbs"; +include "Literal.fbs"; + +namespace org.apache.arrow.computeir.flatbuf; + +/// Access a value for a given map key +table MapKey { + key: string (required); Review comment: Note: these files should be consistent in their use of `(required)`. ########## File path: experimental/computeir/Relation.fbs ########## @@ -0,0 +1,209 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +include "../../format/Schema.fbs"; +include "Literal.fbs"; +include "Expression.fbs"; + +namespace org.apache.arrow.computeir.flatbuf; + +/// A data type indicating that a different mapping of columns +/// should occur in the output. +/// +/// For example: +/// +/// Given a query `SELECT b, a FROM t` where `t` has columns a, b, c +/// the mapping value for the projection would equal [1, 0]. +table Remap { + mapping: [uint32] (required); +} + +// Pass through indicates that no output remapping should occur. +table PassThrough {} + +/// A union for the different colum remapping variants +union Emit { + Remap, + PassThrough, +} + +/// Fields common to every relational operator +table RelBase { + /// Output remapping of ordinal columns for a given operation + output_mapping: Emit (required); +} + +/// Filter operation +table Filter { + /// Common options + base: RelBase (required); + /// Child relation + rel: Relation (required); + /// The expression which will be evaluated against input rows + /// to determine whether they should be excluded from the + /// filter relation's output. + predicate: Expression (required); +} + +/// Projection +table Project { Review comment: Same question obviously :-) ########## File path: experimental/computeir/Literal.fbs ########## @@ -0,0 +1,203 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +namespace org.apache.arrow.computeir.flatbuf; + +table ListLiteral { + values: [Literal]; +} + +table StructLiteral { + values: [KeyValue]; +} + +table KeyValue { + key: string; + value: Literal; +} + +table MapLiteral { + values: [KeyValue]; +} + +table Int8Literal { + value: int8; +} + +table Int16Literal { + value: int16; +} + +table Int32Literal { + value: int32; +} + +table Int64Literal { + value: int64; +} + +table UInt8Literal { + value: uint8; +} + +table UInt16Literal { + value: uint16; +} + +table UInt32Literal { + value: uint32; +} + +table UInt64Literal { + value: uint64; +} + +table Float16Literal { + value: uint16; +} + +table Float32Literal { + value: float32; +} + +table Float64Literal { + value: float64; +} + +table DecimalLiteral { + value: [byte]; Review comment: Is this little-endian encoded? ########## File path: experimental/computeir/Literal.fbs ########## @@ -0,0 +1,203 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +namespace org.apache.arrow.computeir.flatbuf; + +table ListLiteral { + values: [Literal]; +} + +table StructLiteral { + values: [KeyValue]; +} + +table KeyValue { + key: string; + value: Literal; +} + +table MapLiteral { + values: [KeyValue]; +} + +table Int8Literal { + value: int8; +} + +table Int16Literal { + value: int16; +} + +table Int32Literal { + value: int32; +} + +table Int64Literal { + value: int64; +} + +table UInt8Literal { + value: uint8; +} + +table UInt16Literal { + value: uint16; +} + +table UInt32Literal { + value: uint32; +} + +table UInt64Literal { + value: uint64; +} + +table Float16Literal { + value: uint16; +} + +table Float32Literal { + value: float32; +} + +table Float64Literal { + value: float64; +} + +table DecimalLiteral { + value: [byte]; + scale: uint8; + precision: uint8; +} + +table BooleanLiteral { + value: bool; +} + +table NullLiteral {} + +enum DateUnit : uint8 { + DAY, + MILLISECOND, +} + +table DateLiteral { + value: int64; + + // unit of `value` + unit: DateUnit = MILLISECOND; +} + +enum TimeUnit : uint8 { + SECOND, + MILLISECOND, + MICROSECOND, + NANOSECOND, +} + +table TimeLiteral { + value: int64; + + // unit of `value` + unit: TimeUnit = MILLISECOND; +} + +table TimestampLiteral { + value: int64; + + // unit of `value` + unit: TimeUnit; + + // timezone value, same definition as used in Schema.fbs. + timezone: string; +} + +table IntervalLiteralMonths { + months: int32; +} + +table IntervalLiteralDaysMilliseconds { + days: int32; + milliseconds: int32; +} + +union IntervalLiteralImpl { + IntervalLiteralMonths, + IntervalLiteralDaysMilliseconds, +} + +table IntervalLiteral { + value: IntervalLiteralImpl; +} + +table DurationLiteral { + value: int64; + + // unit of `value` + unit: TimeUnit = MILLISECOND; +} + +table BinaryLiteral { + value: [byte]; +} + +table StringLiteral { + value: string; +} + +// no union literal is defined as only one branch of a union can be resolved. +// no literals for large string/binary types as flatbuffer is limited to 2gb. Review comment: No fixed-size binary literal either? ########## File path: experimental/computeir/Literal.fbs ########## @@ -0,0 +1,203 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +namespace org.apache.arrow.computeir.flatbuf; + +table ListLiteral { + values: [Literal]; +} + +table StructLiteral { + values: [KeyValue]; +} + +table KeyValue { + key: string; + value: Literal; +} + +table MapLiteral { + values: [KeyValue]; +} + +table Int8Literal { + value: int8; +} + +table Int16Literal { + value: int16; +} + +table Int32Literal { + value: int32; +} + +table Int64Literal { + value: int64; +} + +table UInt8Literal { + value: uint8; +} + +table UInt16Literal { + value: uint16; +} + +table UInt32Literal { + value: uint32; +} + +table UInt64Literal { + value: uint64; +} + +table Float16Literal { + value: uint16; +} + +table Float32Literal { + value: float32; +} + +table Float64Literal { + value: float64; +} + +table DecimalLiteral { + value: [byte]; + scale: uint8; + precision: uint8; +} + +table BooleanLiteral { + value: bool; +} + +table NullLiteral {} + +enum DateUnit : uint8 { + DAY, + MILLISECOND, +} + +table DateLiteral { + value: int64; + + // unit of `value` + unit: DateUnit = MILLISECOND; +} + +enum TimeUnit : uint8 { + SECOND, + MILLISECOND, + MICROSECOND, + NANOSECOND, +} + +table TimeLiteral { + value: int64; + + // unit of `value` + unit: TimeUnit = MILLISECOND; +} + +table TimestampLiteral { + value: int64; + + // unit of `value` + unit: TimeUnit; + + // timezone value, same definition as used in Schema.fbs. + timezone: string; +} + +table IntervalLiteralMonths { + months: int32; +} + +table IntervalLiteralDaysMilliseconds { + days: int32; + milliseconds: int32; +} Review comment: You probably want to add a `IntervalLiteralDaysMonthsNanos` as well. ########## File path: experimental/computeir/Expression.fbs ########## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +include "../../format/Schema.fbs"; +include "Literal.fbs"; + +namespace org.apache.arrow.computeir.flatbuf; + +/// Access a value for a given map key +table MapKey { + key: string (required); +} + +/// Struct field access +table StructField { + /// The position of the field in the struct schema + position: uint32; +} + +/// Zero-based array index +table ArraySubscript { + position: uint32; +} + +/// Zero-based range of elements in an array +table ArraySlice { + /// The start of an array slice, inclusive + start_inclusive: uint32; + /// The end of an array slice, exclusive + end_exclusive: uint32; +} + +/// Field name in a relation +table FieldName { + position: uint32; +} + +/// A union of possible dereference operations +union Deref { + /// Access a value for a given map key + MapKey, + /// Access the value at a struct field + StructField, + /// Access the element at a given index in an array + ArraySubscript, + /// Access a range of elements in an array + ArraySlice, + /// Access a field of a relation + FieldName, +} + +/// Access the data of a field +table FieldRef { + /// A sequence of field names to allow referencing potentially nested fields + ref: Deref (required); + /// For Expressions which might reference fields in multiple Relations, + /// this index may be provided to indicate which Relation's fields + /// `path` points into. For example in the case of a join, + /// 0 refers to the left relation and 1 to the right relation. + relation_index: int = 0; +} + +/// A function call expression +table Call { + /// The function to call + name: string (required); + + /// The arguments passed to `name`. + arguments: [Expression] (required); + + /// Possible ordering of input. These are useful + /// in aggregates where ordering in meaningful such as + /// string concatenation + orderings: [SortKey]; +} + +/// A single WHEN x THEN y fragment. +table CaseFragment { + match: Expression (required); + result: Expression (required); +} + +/// Conditional case statement expression +table ConditionalCase { Review comment: Is there a reason not to express this as a `Call` with a well-defined function name? ########## File path: experimental/computeir/Expression.fbs ########## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +include "../../format/Schema.fbs"; +include "Literal.fbs"; + +namespace org.apache.arrow.computeir.flatbuf; + +/// Access a value for a given map key +table MapKey { + key: string (required); +} + +/// Struct field access +table StructField { + /// The position of the field in the struct schema + position: uint32; +} + +/// Zero-based array index +table ArraySubscript { + position: uint32; +} + +/// Zero-based range of elements in an array +table ArraySlice { + /// The start of an array slice, inclusive + start_inclusive: uint32; + /// The end of an array slice, exclusive + end_exclusive: uint32; +} + +/// Field name in a relation +table FieldName { + position: uint32; +} + +/// A union of possible dereference operations +union Deref { + /// Access a value for a given map key + MapKey, + /// Access the value at a struct field + StructField, + /// Access the element at a given index in an array + ArraySubscript, + /// Access a range of elements in an array + ArraySlice, + /// Access a field of a relation + FieldName, +} + +/// Access the data of a field +table FieldRef { + /// A sequence of field names to allow referencing potentially nested fields + ref: Deref (required); + /// For Expressions which might reference fields in multiple Relations, + /// this index may be provided to indicate which Relation's fields + /// `path` points into. For example in the case of a join, + /// 0 refers to the left relation and 1 to the right relation. + relation_index: int = 0; +} + +/// A function call expression +table Call { + /// The function to call + name: string (required); Review comment: Is it intended to normalize function names for the most common functions at some point? (e.g. `add` etc.) ########## File path: experimental/computeir/Relation.fbs ########## @@ -0,0 +1,209 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +include "../../format/Schema.fbs"; +include "Literal.fbs"; +include "Expression.fbs"; + +namespace org.apache.arrow.computeir.flatbuf; + +/// A data type indicating that a different mapping of columns +/// should occur in the output. +/// +/// For example: +/// +/// Given a query `SELECT b, a FROM t` where `t` has columns a, b, c +/// the mapping value for the projection would equal [1, 0]. +table Remap { + mapping: [uint32] (required); Review comment: Is this desired instead of a bunch of `FieldName` expressions? ########## File path: experimental/computeir/Expression.fbs ########## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +include "../../format/Schema.fbs"; +include "Literal.fbs"; + +namespace org.apache.arrow.computeir.flatbuf; + +/// Access a value for a given map key +table MapKey { + key: string (required); +} + +/// Struct field access +table StructField { + /// The position of the field in the struct schema + position: uint32; +} + +/// Zero-based array index +table ArraySubscript { + position: uint32; +} + +/// Zero-based range of elements in an array +table ArraySlice { + /// The start of an array slice, inclusive + start_inclusive: uint32; + /// The end of an array slice, exclusive + end_exclusive: uint32; +} + +/// Field name in a relation +table FieldName { + position: uint32; +} + +/// A union of possible dereference operations +union Deref { + /// Access a value for a given map key + MapKey, + /// Access the value at a struct field + StructField, + /// Access the element at a given index in an array + ArraySubscript, + /// Access a range of elements in an array + ArraySlice, + /// Access a field of a relation + FieldName, +} + +/// Access the data of a field +table FieldRef { + /// A sequence of field names to allow referencing potentially nested fields + ref: Deref (required); + /// For Expressions which might reference fields in multiple Relations, + /// this index may be provided to indicate which Relation's fields + /// `path` points into. For example in the case of a join, + /// 0 refers to the left relation and 1 to the right relation. + relation_index: int = 0; +} + +/// A function call expression +table Call { + /// The function to call + name: string (required); + + /// The arguments passed to `name`. + arguments: [Expression] (required); + + /// Possible ordering of input. These are useful + /// in aggregates where ordering in meaningful such as + /// string concatenation + orderings: [SortKey]; +} + +/// A single WHEN x THEN y fragment. +table CaseFragment { + match: Expression (required); + result: Expression (required); +} + +/// Conditional case statement expression +table ConditionalCase { + /// List of conditions to evaluate + conditions: [CaseFragment] (required); + /// The default value if no cases match. This is typically NULL in SQL + /// implementations. + /// + /// Defaulting to NULL is a frontend choice, so producers must specify NULL + /// if that's their desired behavior. + else: Expression (required); +} + +/// Switch-style case expression +table SimpleCase { Review comment: Same question. ########## File path: experimental/computeir/Expression.fbs ########## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +include "../../format/Schema.fbs"; +include "Literal.fbs"; + +namespace org.apache.arrow.computeir.flatbuf; + +/// Access a value for a given map key +table MapKey { + key: string (required); +} + +/// Struct field access +table StructField { + /// The position of the field in the struct schema + position: uint32; +} + +/// Zero-based array index +table ArraySubscript { + position: uint32; +} + +/// Zero-based range of elements in an array +table ArraySlice { + /// The start of an array slice, inclusive + start_inclusive: uint32; + /// The end of an array slice, exclusive + end_exclusive: uint32; +} + +/// Field name in a relation +table FieldName { + position: uint32; +} + +/// A union of possible dereference operations +union Deref { + /// Access a value for a given map key + MapKey, + /// Access the value at a struct field + StructField, + /// Access the element at a given index in an array + ArraySubscript, + /// Access a range of elements in an array + ArraySlice, + /// Access a field of a relation + FieldName, +} + +/// Access the data of a field +table FieldRef { + /// A sequence of field names to allow referencing potentially nested fields + ref: Deref (required); + /// For Expressions which might reference fields in multiple Relations, + /// this index may be provided to indicate which Relation's fields + /// `path` points into. For example in the case of a join, + /// 0 refers to the left relation and 1 to the right relation. + relation_index: int = 0; +} + +/// A function call expression +table Call { + /// The function to call + name: string (required); + + /// The arguments passed to `name`. + arguments: [Expression] (required); + + /// Possible ordering of input. These are useful + /// in aggregates where ordering in meaningful such as + /// string concatenation + orderings: [SortKey]; +} + +/// A single WHEN x THEN y fragment. +table CaseFragment { + match: Expression (required); + result: Expression (required); +} + +/// Conditional case statement expression +table ConditionalCase { + /// List of conditions to evaluate + conditions: [CaseFragment] (required); + /// The default value if no cases match. This is typically NULL in SQL + /// implementations. + /// + /// Defaulting to NULL is a frontend choice, so producers must specify NULL + /// if that's their desired behavior. + else: Expression (required); +} + +/// Switch-style case expression +table SimpleCase { + /// The expression whose value will be matched + expression: Expression (required); + /// Matches for `expression` + matches: [CaseFragment] (required); + /// The default value if no cases match + else: Expression (required); +} + +/// Whether lesser values should precede greater or vice versa, +/// also whether nulls should preced or follow values +enum Collation : uint8 { Review comment: "Collation" sounds more like the kind of comparison function you're going to apply (such as unicode collation)? ########## File path: experimental/computeir/Relation.fbs ########## @@ -0,0 +1,209 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +include "../../format/Schema.fbs"; +include "Literal.fbs"; +include "Expression.fbs"; + +namespace org.apache.arrow.computeir.flatbuf; + +/// A data type indicating that a different mapping of columns +/// should occur in the output. +/// +/// For example: +/// +/// Given a query `SELECT b, a FROM t` where `t` has columns a, b, c +/// the mapping value for the projection would equal [1, 0]. +table Remap { + mapping: [uint32] (required); +} + +// Pass through indicates that no output remapping should occur. +table PassThrough {} + +/// A union for the different colum remapping variants +union Emit { + Remap, + PassThrough, +} + +/// Fields common to every relational operator +table RelBase { + /// Output remapping of ordinal columns for a given operation + output_mapping: Emit (required); +} + +/// Filter operation +table Filter { Review comment: Can this be simply be expressed as a `Call`? Do we benefit from having a dedicated `Filter` node? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
