[jira] [Commented] (DRILL-3929) Support the ability to query database tables using external indices
[ https://issues.apache.org/jira/browse/DRILL-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006312#comment-15006312 ] Julian Hyde commented on DRILL-3929: When you say "Index selection is part of physical database design and is a core step in access path selection which occurs at physical planning stage" how have you decided what should occur at logical and physical planning stage? You make it sound as if there is a dictionary definition of "physical planning stage" whereas it is just a design decision which decisions are made when. And indeed every DBMS will have different definitions of "logical" and "physical". I recall that "Access path selection" is a phase in query optimization in a transactional DBMS but is not so clear cut in a modern analytic DBMS. For example Vertica has no indexes, only relations, so you choose access path and relations simultaneously. "I don't see how all indexes can be modeled as [materialized views]". - The contents of any index can be modeled as a query on the table (or tables for join indexes), and the physical layout can be modeled too. A table whose contents is identically equal to a query -- that's the very definition of a materialized view. > Support the ability to query database tables using external indices > -- > > Key: DRILL-3929 > URL: https://issues.apache.org/jira/browse/DRILL-3929 > Project: Apache Drill > Issue Type: New Feature > Components: Execution - Relational Operators, Query Planning & > Optimization >Reporter: Aman Sinha >Assignee: Aman Sinha > > This is a placeholder for adding support in Drill to query database tables > using external indices. I will add more details about the use case and a > preliminary design proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.
[ https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinfeng Ni updated DRILL-3765: -- Assignee: Aman Sinha (was: Jinfeng Ni) > Partition prune rule is unnecessary fired multiple times. > -- > > Key: DRILL-3765 > URL: https://issues.apache.org/jira/browse/DRILL-3765 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Reporter: Jinfeng Ni >Assignee: Aman Sinha > > It seems that the partition prune rule may be fired multiple times, even > after the first rule execution has pushed the filter into the scan operator. > Since partition prune has to build the vectors to contain the partition /file > / directory information, to invoke the partition prune rule unnecessary may > lead to big memory overhead. > Drill planner should avoid the un-necessary partition prune rule, in order to > reduce the chance of hitting OOM exception, while the partition prune rule is > executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.
[ https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006236#comment-15006236 ] Jinfeng Ni commented on DRILL-3765: --- [~amansinha100], could you please review the PR for DRILL-3765? Thanks! https://github.com/apache/drill/pull/255 > Partition prune rule is unnecessary fired multiple times. > -- > > Key: DRILL-3765 > URL: https://issues.apache.org/jira/browse/DRILL-3765 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Reporter: Jinfeng Ni >Assignee: Aman Sinha > > It seems that the partition prune rule may be fired multiple times, even > after the first rule execution has pushed the filter into the scan operator. > Since partition prune has to build the vectors to contain the partition /file > / directory information, to invoke the partition prune rule unnecessary may > lead to big memory overhead. > Drill planner should avoid the un-necessary partition prune rule, in order to > reduce the chance of hitting OOM exception, while the partition prune rule is > executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4081) Handle schema changes in ExternalSort
[ https://issues.apache.org/jira/browse/DRILL-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006233#comment-15006233 ] ASF GitHub Bot commented on DRILL-4081: --- Github user jacques-n commented on the pull request: https://github.com/apache/drill/pull/257#issuecomment-156919949 I think it is fine if you handle the unsupported map behaviors with a clear user exception. Other than the refactoring of getFieldId (if possible), LGTM. +1. (Assuming that this functionality only changes system behavior in the case that Union vector is enabled.) > Handle schema changes in ExternalSort > - > > Key: DRILL-4081 > URL: https://issues.apache.org/jira/browse/DRILL-4081 > Project: Apache Drill > Issue Type: Improvement >Reporter: Steven Phillips >Assignee: Steven Phillips > > This improvement will make use of the Union vector to handle schema changes. > When a new schema appears, the schema will be "merged" with the previous > schema. The result will be a new schema that uses Union type to store the > columns where this is a type conflict. All of the batches (including the > batches that have already arrived) will be coerced into this new schema. > A new comparison function will be included to handle the comparison of Union > type. Comparison of union type will work as follows: > 1. All numeric types can be mutually compared, and will be compared using > Drill implicit cast rules. > 2. All other types will not be compared against other types, but only among > values of the same type. > 3. There will be an overall precedence of types with regards to ordering. > This precedence is not yet defined, but will be as part of the work on this > issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3929) Support the ability to query database tables using external indices
[ https://issues.apache.org/jira/browse/DRILL-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006219#comment-15006219 ] Aman Sinha commented on DRILL-3929: --- Sorry I could not respond sooner .. I was caught up with few other Drill issues. [~julianhyde], I am afraid I would have to disagree; Index selection is part of physical database design and is a core step in access path selection which occurs at physical planning stage.Are you suggesting that relational database systems are wrong in doing that ? While some indexes could be modeled as MVs, I don't see how all indexes can be modeled as such. Indexes have different physical implementations too: hash index will *not* have collation whereas a clustered index will. So how can N different physical implementations be modeled through logical planning ? > Support the ability to query database tables using external indices > -- > > Key: DRILL-3929 > URL: https://issues.apache.org/jira/browse/DRILL-3929 > Project: Apache Drill > Issue Type: New Feature > Components: Execution - Relational Operators, Query Planning & > Optimization >Reporter: Aman Sinha >Assignee: Aman Sinha > > This is a placeholder for adding support in Drill to query database tables > using external indices. I will add more details about the use case and a > preliminary design proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4081) Handle schema changes in ExternalSort
[
https://issues.apache.org/jira/browse/DRILL-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006151#comment-15006151
]
ASF GitHub Bot commented on DRILL-4081:
---
Github user jacques-n commented on a diff in the pull request:
https://github.com/apache/drill/pull/257#discussion_r44883694
--- Diff:
exec/vector/src/main/java/org/apache/drill/exec/vector/complex/FieldIdUtil.java
---
@@ -121,4 +125,63 @@ public static TypedFieldId
getFieldIdIfMatches(ValueVector vector, TypedFieldId.
}
}
}
+
+ public static TypedFieldId getFieldId(ValueVector vector, int id,
SchemaPath expectedPath, boolean hyper) {
--- End diff --
Is it possible to share this code (or much of it with getFieldIdIfMatches()
above). It seems like they are related (although I didn't go line by line)
> Handle schema changes in ExternalSort
> -
>
> Key: DRILL-4081
> URL: https://issues.apache.org/jira/browse/DRILL-4081
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Steven Phillips
>Assignee: Steven Phillips
>
> This improvement will make use of the Union vector to handle schema changes.
> When a new schema appears, the schema will be "merged" with the previous
> schema. The result will be a new schema that uses Union type to store the
> columns where this is a type conflict. All of the batches (including the
> batches that have already arrived) will be coerced into this new schema.
> A new comparison function will be included to handle the comparison of Union
> type. Comparison of union type will work as follows:
> 1. All numeric types can be mutually compared, and will be compared using
> Drill implicit cast rules.
> 2. All other types will not be compared against other types, but only among
> values of the same type.
> 3. There will be an overall precedence of types with regards to ordering.
> This precedence is not yet defined, but will be as part of the work on this
> issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4081) Handle schema changes in ExternalSort
[
https://issues.apache.org/jira/browse/DRILL-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006150#comment-15006150
]
ASF GitHub Bot commented on DRILL-4081:
---
Github user jacques-n commented on a diff in the pull request:
https://github.com/apache/drill/pull/257#discussion_r44883581
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/record/SchemaUtil.java ---
@@ -0,0 +1,159 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.record;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.memory.BufferAllocator;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.sort.RecordBatchData;
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.vector.ValueVector;
+import org.apache.drill.exec.vector.complex.UnionVector;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Utility class for dealing with changing schemas
+ */
+public class SchemaUtil {
+
+ /**
+ * Returns the merger of schemas. The merged schema will include the
union all columns. If there is a type conflict
+ * between columns with the same schemapath but different types, the
merged schema will contain a Union type.
+ * @param schemas
+ * @return
+ */
+ public static BatchSchema mergeSchemas(BatchSchema... schemas) {
+Map> typeSetMap = Maps.newLinkedHashMap();
+
+for (BatchSchema s : schemas) {
+ for (MaterializedField field : s) {
+SchemaPath path = field.getPath();
+Set currentTypes = typeSetMap.get(path);
+if (currentTypes == null) {
+ currentTypes = Sets.newHashSet();
+ typeSetMap.put(path, currentTypes);
+}
+MinorType newType = field.getType().getMinorType();
+if (newType == MinorType.UNION) {
+ for (MinorType subType : field.getType().getSubTypeList()) {
+currentTypes.add(subType);
+ }
+} else {
+ currentTypes.add(newType);
+}
+ }
+}
+
+List fields = Lists.newArrayList();
+
+for (SchemaPath path : typeSetMap.keySet()) {
+ Set types = typeSetMap.get(path);
+ if (types.size() > 1) {
+MajorType.Builder builder =
MajorType.newBuilder().setMinorType(MinorType.UNION).setMode(DataMode.OPTIONAL);
+for (MinorType t : types) {
+ builder.addSubType(t);
+}
+MaterializedField field = MaterializedField.create(path,
builder.build());
+fields.add(field);
+ } else {
+MaterializedField field = MaterializedField.create(path,
Types.optional(types.iterator().next()));
+fields.add(field);
+ }
+}
+
+SchemaBuilder schemaBuilder = new SchemaBuilder();
+BatchSchema s =
schemaBuilder.addFields(fields).setSelectionVectorMode(schemas[0].getSelectionVectorMode()).build();
+return s;
+ }
+
+ /**
+ * Creates a copy a record batch, converting any fields as necessary to
coerce it into the provided schema
+ * @param in
+ * @param toSchema
+ * @param context
+ * @return
+ */
+ public static VectorContainer coerceContainer(VectorAccessible in,
BatchSchema toSchema, OperatorContext context) {
+int r
[jira] [Commented] (DRILL-4081) Handle schema changes in ExternalSort
[
https://issues.apache.org/jira/browse/DRILL-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006149#comment-15006149
]
ASF GitHub Bot commented on DRILL-4081:
---
Github user jacques-n commented on a diff in the pull request:
https://github.com/apache/drill/pull/257#discussion_r44883570
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/record/SchemaUtil.java ---
@@ -0,0 +1,159 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.record;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.memory.BufferAllocator;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.sort.RecordBatchData;
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.vector.ValueVector;
+import org.apache.drill.exec.vector.complex.UnionVector;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Utility class for dealing with changing schemas
+ */
+public class SchemaUtil {
+
+ /**
+ * Returns the merger of schemas. The merged schema will include the
union all columns. If there is a type conflict
+ * between columns with the same schemapath but different types, the
merged schema will contain a Union type.
+ * @param schemas
+ * @return
+ */
+ public static BatchSchema mergeSchemas(BatchSchema... schemas) {
+Map> typeSetMap = Maps.newLinkedHashMap();
+
+for (BatchSchema s : schemas) {
+ for (MaterializedField field : s) {
+SchemaPath path = field.getPath();
+Set currentTypes = typeSetMap.get(path);
+if (currentTypes == null) {
+ currentTypes = Sets.newHashSet();
+ typeSetMap.put(path, currentTypes);
+}
+MinorType newType = field.getType().getMinorType();
+if (newType == MinorType.UNION) {
+ for (MinorType subType : field.getType().getSubTypeList()) {
+currentTypes.add(subType);
+ }
+} else {
+ currentTypes.add(newType);
+}
+ }
+}
+
+List fields = Lists.newArrayList();
+
+for (SchemaPath path : typeSetMap.keySet()) {
+ Set types = typeSetMap.get(path);
+ if (types.size() > 1) {
+MajorType.Builder builder =
MajorType.newBuilder().setMinorType(MinorType.UNION).setMode(DataMode.OPTIONAL);
+for (MinorType t : types) {
+ builder.addSubType(t);
+}
+MaterializedField field = MaterializedField.create(path,
builder.build());
+fields.add(field);
+ } else {
+MaterializedField field = MaterializedField.create(path,
Types.optional(types.iterator().next()));
+fields.add(field);
+ }
+}
+
+SchemaBuilder schemaBuilder = new SchemaBuilder();
+BatchSchema s =
schemaBuilder.addFields(fields).setSelectionVectorMode(schemas[0].getSelectionVectorMode()).build();
+return s;
+ }
+
+ /**
+ * Creates a copy a record batch, converting any fields as necessary to
coerce it into the provided schema
+ * @param in
+ * @param toSchema
+ * @param context
+ * @return
+ */
+ public static VectorContainer coerceContainer(VectorAccessible in,
BatchSchema toSchema, OperatorContext context) {
+int r
[jira] [Commented] (DRILL-4081) Handle schema changes in ExternalSort
[
https://issues.apache.org/jira/browse/DRILL-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006145#comment-15006145
]
ASF GitHub Bot commented on DRILL-4081:
---
Github user jacques-n commented on a diff in the pull request:
https://github.com/apache/drill/pull/257#discussion_r44883499
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/record/SchemaUtil.java ---
@@ -0,0 +1,159 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.record;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.memory.BufferAllocator;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.sort.RecordBatchData;
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.vector.ValueVector;
+import org.apache.drill.exec.vector.complex.UnionVector;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Utility class for dealing with changing schemas
+ */
+public class SchemaUtil {
+
+ /**
+ * Returns the merger of schemas. The merged schema will include the
union all columns. If there is a type conflict
+ * between columns with the same schemapath but different types, the
merged schema will contain a Union type.
+ * @param schemas
+ * @return
+ */
+ public static BatchSchema mergeSchemas(BatchSchema... schemas) {
+Map> typeSetMap = Maps.newLinkedHashMap();
+
+for (BatchSchema s : schemas) {
+ for (MaterializedField field : s) {
+SchemaPath path = field.getPath();
+Set currentTypes = typeSetMap.get(path);
+if (currentTypes == null) {
+ currentTypes = Sets.newHashSet();
+ typeSetMap.put(path, currentTypes);
+}
+MinorType newType = field.getType().getMinorType();
+if (newType == MinorType.UNION) {
+ for (MinorType subType : field.getType().getSubTypeList()) {
+currentTypes.add(subType);
+ }
+} else {
+ currentTypes.add(newType);
+}
+ }
+}
+
+List fields = Lists.newArrayList();
+
+for (SchemaPath path : typeSetMap.keySet()) {
+ Set types = typeSetMap.get(path);
+ if (types.size() > 1) {
--- End diff --
Isn't it possible two different map types would be added above? If so, it
seems like we would produce a union when we shouldn't.
> Handle schema changes in ExternalSort
> -
>
> Key: DRILL-4081
> URL: https://issues.apache.org/jira/browse/DRILL-4081
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Steven Phillips
>Assignee: Steven Phillips
>
> This improvement will make use of the Union vector to handle schema changes.
> When a new schema appears, the schema will be "merged" with the previous
> schema. The result will be a new schema that uses Union type to store the
> columns where this is a type conflict. All of the batches (including the
> batches that have already arrived) will be coerced into this new schema.
> A new comparison function will be included to handle the comparison of Union
> type. Comparison of union type will work as follows:
> 1. All numeric types can be mutually compared, and will be compared using
> Drill implicit cast
[jira] [Commented] (DRILL-4081) Handle schema changes in ExternalSort
[
https://issues.apache.org/jira/browse/DRILL-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006138#comment-15006138
]
ASF GitHub Bot commented on DRILL-4081:
---
Github user jacques-n commented on a diff in the pull request:
https://github.com/apache/drill/pull/257#discussion_r44883068
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java ---
@@ -113,10 +115,45 @@ public boolean equals(Object obj) {
} else if (!fields.equals(other.fields)) {
return false;
}
+for (int i = 0; i < fields.size(); i++) {
+ MajorType t1 = fields.get(i).getType();
+ MajorType t2 = other.fields.get(i).getType();
+ if (t1 == null) {
+if (t2 != null) {
+ return false;
+}
+ } else {
+if (!majorTypeEqual(t1, t2)) {
+ return false;
+}
+ }
+}
if (selectionVectorMode != other.selectionVectorMode) {
return false;
}
return true;
}
+ /**
+ * We treat fields with same set of Subtypes as equal, even if they are
in a different order
+ * @param t1
+ * @param t2
+ * @return
+ */
+ private boolean majorTypeEqual(MajorType t1, MajorType t2) {
+if (t1.equals(t2)) {
+ return true;
+}
+if (!t1.getMinorType().equals(t2.getMinorType())) {
+ return false;
+}
+if (!t1.getMode().equals(t2.getMode())) {
+ return false;
+}
+if
(!Sets.newHashSet(t1.getSubTypeList()).equals(Sets.newHashSet(t2.getSubTypeList(
{
--- End diff --
Should we recursively check that the subtype of subtypes are also set equal
as opposed to your current direct equality (below the second level)
> Handle schema changes in ExternalSort
> -
>
> Key: DRILL-4081
> URL: https://issues.apache.org/jira/browse/DRILL-4081
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Steven Phillips
>Assignee: Steven Phillips
>
> This improvement will make use of the Union vector to handle schema changes.
> When a new schema appears, the schema will be "merged" with the previous
> schema. The result will be a new schema that uses Union type to store the
> columns where this is a type conflict. All of the batches (including the
> batches that have already arrived) will be coerced into this new schema.
> A new comparison function will be included to handle the comparison of Union
> type. Comparison of union type will work as follows:
> 1. All numeric types can be mutually compared, and will be compared using
> Drill implicit cast rules.
> 2. All other types will not be compared against other types, but only among
> values of the same type.
> 3. There will be an overall precedence of types with regards to ordering.
> This precedence is not yet defined, but will be as part of the work on this
> issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4081) Handle schema changes in ExternalSort
[
https://issues.apache.org/jira/browse/DRILL-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006137#comment-15006137
]
ASF GitHub Bot commented on DRILL-4081:
---
Github user jacques-n commented on a diff in the pull request:
https://github.com/apache/drill/pull/257#discussion_r44883050
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java ---
@@ -113,10 +115,45 @@ public boolean equals(Object obj) {
} else if (!fields.equals(other.fields)) {
return false;
}
+for (int i = 0; i < fields.size(); i++) {
+ MajorType t1 = fields.get(i).getType();
+ MajorType t2 = other.fields.get(i).getType();
+ if (t1 == null) {
+if (t2 != null) {
+ return false;
+}
+ } else {
+if (!majorTypeEqual(t1, t2)) {
+ return false;
+}
+ }
+}
if (selectionVectorMode != other.selectionVectorMode) {
return false;
}
return true;
}
+ /**
+ * We treat fields with same set of Subtypes as equal, even if they are
in a different order
+ * @param t1
+ * @param t2
+ * @return
+ */
+ private boolean majorTypeEqual(MajorType t1, MajorType t2) {
--- End diff --
A better function name would be good here. maybe
majorTypeSubfieldSetsAreEqual?
> Handle schema changes in ExternalSort
> -
>
> Key: DRILL-4081
> URL: https://issues.apache.org/jira/browse/DRILL-4081
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Steven Phillips
>Assignee: Steven Phillips
>
> This improvement will make use of the Union vector to handle schema changes.
> When a new schema appears, the schema will be "merged" with the previous
> schema. The result will be a new schema that uses Union type to store the
> columns where this is a type conflict. All of the batches (including the
> batches that have already arrived) will be coerced into this new schema.
> A new comparison function will be included to handle the comparison of Union
> type. Comparison of union type will work as follows:
> 1. All numeric types can be mutually compared, and will be compared using
> Drill implicit cast rules.
> 2. All other types will not be compared against other types, but only among
> values of the same type.
> 3. There will be an overall precedence of types with regards to ordering.
> This precedence is not yet defined, but will be as part of the work on this
> issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4081) Handle schema changes in ExternalSort
[
https://issues.apache.org/jira/browse/DRILL-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006135#comment-15006135
]
ASF GitHub Bot commented on DRILL-4081:
---
Github user jacques-n commented on a diff in the pull request:
https://github.com/apache/drill/pull/257#discussion_r44882984
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/record/BaseVectorWrapper.java
---
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.record;
+
+import org.apache.drill.common.expression.PathSegment;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.vector.ValueVector;
+import org.apache.drill.exec.vector.complex.AbstractContainerVector;
+import org.apache.drill.exec.vector.complex.ListVector;
+import org.apache.drill.exec.vector.complex.UnionVector;
+
+import java.util.List;
+
+public abstract class BaseVectorWrapper implements VectorWrapper {
+
+ protected TypedFieldId getFieldIdIfMatches(ValueVector vector, int id,
SchemaPath expectedPath) {
+if
(!expectedPath.getRootSegment().segmentEquals(vector.getField().getPath().getRootSegment()))
{
+ return null;
+}
+PathSegment seg = expectedPath.getRootSegment();
+
+if (vector instanceof UnionVector) {
--- End diff --
Need docs on all of this.
> Handle schema changes in ExternalSort
> -
>
> Key: DRILL-4081
> URL: https://issues.apache.org/jira/browse/DRILL-4081
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Steven Phillips
>Assignee: Steven Phillips
>
> This improvement will make use of the Union vector to handle schema changes.
> When a new schema appears, the schema will be "merged" with the previous
> schema. The result will be a new schema that uses Union type to store the
> columns where this is a type conflict. All of the batches (including the
> batches that have already arrived) will be coerced into this new schema.
> A new comparison function will be included to handle the comparison of Union
> type. Comparison of union type will work as follows:
> 1. All numeric types can be mutually compared, and will be compared using
> Drill implicit cast rules.
> 2. All other types will not be compared against other types, but only among
> values of the same type.
> 3. There will be an overall precedence of types with regards to ordering.
> This precedence is not yet defined, but will be as part of the work on this
> issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4081) Handle schema changes in ExternalSort
[ https://issues.apache.org/jira/browse/DRILL-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006134#comment-15006134 ] ASF GitHub Bot commented on DRILL-4081: --- Github user jacques-n commented on a diff in the pull request: https://github.com/apache/drill/pull/257#discussion_r44882959 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java --- @@ -830,4 +785,50 @@ private boolean castEqual(ExpressionPosition pos, MajorType from, MajorType to) } } } + --- End diff -- Is this big block a reorganization or actual changes? > Handle schema changes in ExternalSort > - > > Key: DRILL-4081 > URL: https://issues.apache.org/jira/browse/DRILL-4081 > Project: Apache Drill > Issue Type: Improvement >Reporter: Steven Phillips >Assignee: Steven Phillips > > This improvement will make use of the Union vector to handle schema changes. > When a new schema appears, the schema will be "merged" with the previous > schema. The result will be a new schema that uses Union type to store the > columns where this is a type conflict. All of the batches (including the > batches that have already arrived) will be coerced into this new schema. > A new comparison function will be included to handle the comparison of Union > type. Comparison of union type will work as follows: > 1. All numeric types can be mutually compared, and will be compared using > Drill implicit cast rules. > 2. All other types will not be compared against other types, but only among > values of the same type. > 3. There will be an overall precedence of types with regards to ordering. > This precedence is not yet defined, but will be as part of the work on this > issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4081) Handle schema changes in ExternalSort
[
https://issues.apache.org/jira/browse/DRILL-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006133#comment-15006133
]
ASF GitHub Bot commented on DRILL-4081:
---
Github user jacques-n commented on a diff in the pull request:
https://github.com/apache/drill/pull/257#discussion_r44882947
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/UnionFunctions.java
---
@@ -32,16 +33,72 @@
import org.apache.drill.exec.expr.holders.UnionHolder;
import org.apache.drill.exec.expr.holders.IntHolder;
import org.apache.drill.exec.expr.holders.VarCharHolder;
+import org.apache.drill.exec.resolver.TypeCastRules;
import org.apache.drill.exec.vector.complex.impl.UnionReader;
import org.apache.drill.exec.vector.complex.reader.FieldReader;
import javax.inject.Inject;
+import java.util.Set;
/**
* The class contains additional functions for union types in addition to
those in GUnionFunctions
*/
public class UnionFunctions {
+ /**
+ * Returns zero if the inputs have equivalent types. Two numeric types
are considered equivalent, as are a combination
+ * of date/timestamp. If not equivalent, returns a value determined by
the numeric value of the MinorType enum
+ */
+ @FunctionTemplate(names = {"compareType"},
+ scope = FunctionTemplate.FunctionScope.SIMPLE,
+ nulls = NullHandling.INTERNAL)
+ public static class CompareType implements DrillSimpleFunc {
+
+@Param
+FieldReader input1;
+@Param
+FieldReader input2;
+@Output
+IntHolder out;
+
+public void setup() {}
+
+public void eval() {
+ org.apache.drill.common.types.TypeProtos.MinorType type1;
+ if (input1.isSet()) {
+type1 = input1.getType().getMinorType();
+ } else {
+type1 = org.apache.drill.common.types.TypeProtos.MinorType.NULL;
+ }
+ org.apache.drill.common.types.TypeProtos.MinorType type2;
+ if (input2.isSet()) {
+type2 = input2.getType().getMinorType();
+ } else {
+type2 = org.apache.drill.common.types.TypeProtos.MinorType.NULL;
+ }
+
+ out.value =
org.apache.drill.exec.expr.fn.impl.UnionFunctions.compareTypes(type1, type2);
+}
+ }
+
+ public static int compareTypes(MinorType type1, MinorType type2) {
+int typeValue1 = getTypeValue(type1);
+int typeValue2 = getTypeValue(type2);
+return typeValue1 - typeValue2;
+ }
+
+ private static int getTypeValue(MinorType type) {
+if (TypeCastRules.isNumericType(type)) {
--- End diff --
would be good to add comments here.
> Handle schema changes in ExternalSort
> -
>
> Key: DRILL-4081
> URL: https://issues.apache.org/jira/browse/DRILL-4081
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Steven Phillips
>Assignee: Steven Phillips
>
> This improvement will make use of the Union vector to handle schema changes.
> When a new schema appears, the schema will be "merged" with the previous
> schema. The result will be a new schema that uses Union type to store the
> columns where this is a type conflict. All of the batches (including the
> batches that have already arrived) will be coerced into this new schema.
> A new comparison function will be included to handle the comparison of Union
> type. Comparison of union type will work as follows:
> 1. All numeric types can be mutually compared, and will be compared using
> Drill implicit cast rules.
> 2. All other types will not be compared against other types, but only among
> values of the same type.
> 3. There will be an overall precedence of types with regards to ordering.
> This precedence is not yet defined, but will be as part of the work on this
> issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4081) Handle schema changes in ExternalSort
[
https://issues.apache.org/jira/browse/DRILL-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006129#comment-15006129
]
ASF GitHub Bot commented on DRILL-4081:
---
Github user jacques-n commented on a diff in the pull request:
https://github.com/apache/drill/pull/257#discussion_r44882906
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/UnionFunctions.java
---
@@ -32,16 +33,72 @@
import org.apache.drill.exec.expr.holders.UnionHolder;
import org.apache.drill.exec.expr.holders.IntHolder;
import org.apache.drill.exec.expr.holders.VarCharHolder;
+import org.apache.drill.exec.resolver.TypeCastRules;
import org.apache.drill.exec.vector.complex.impl.UnionReader;
import org.apache.drill.exec.vector.complex.reader.FieldReader;
import javax.inject.Inject;
+import java.util.Set;
/**
* The class contains additional functions for union types in addition to
those in GUnionFunctions
*/
public class UnionFunctions {
+ /**
+ * Returns zero if the inputs have equivalent types. Two numeric types
are considered equivalent, as are a combination
+ * of date/timestamp. If not equivalent, returns a value determined by
the numeric value of the MinorType enum
--- End diff --
let's add an additional ordering ordinal to the minortype enum so that we
match Mongo's ordering among types
> Handle schema changes in ExternalSort
> -
>
> Key: DRILL-4081
> URL: https://issues.apache.org/jira/browse/DRILL-4081
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Steven Phillips
>Assignee: Steven Phillips
>
> This improvement will make use of the Union vector to handle schema changes.
> When a new schema appears, the schema will be "merged" with the previous
> schema. The result will be a new schema that uses Union type to store the
> columns where this is a type conflict. All of the batches (including the
> batches that have already arrived) will be coerced into this new schema.
> A new comparison function will be included to handle the comparison of Union
> type. Comparison of union type will work as follows:
> 1. All numeric types can be mutually compared, and will be compared using
> Drill implicit cast rules.
> 2. All other types will not be compared against other types, but only among
> values of the same type.
> 3. There will be an overall precedence of types with regards to ordering.
> This precedence is not yet defined, but will be as part of the work on this
> issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4081) Handle schema changes in ExternalSort
[ https://issues.apache.org/jira/browse/DRILL-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006124#comment-15006124 ] ASF GitHub Bot commented on DRILL-4081: --- GitHub user StevenMPhillips opened a pull request: https://github.com/apache/drill/pull/257 DRILL-4081: Handle schema changes in ExternalSort You can merge this pull request into a Git repository by running: $ git pull https://github.com/StevenMPhillips/drill drill-4081 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/257.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #257 commit db573ffe2c15caad4a9d12dab9d28cc3e0b2f51d Author: Steven Phillips Date: 2015-11-13T19:27:16Z DRILL-4081: Handle schema changes in ExternalSort > Handle schema changes in ExternalSort > - > > Key: DRILL-4081 > URL: https://issues.apache.org/jira/browse/DRILL-4081 > Project: Apache Drill > Issue Type: Improvement >Reporter: Steven Phillips >Assignee: Steven Phillips > > This improvement will make use of the Union vector to handle schema changes. > When a new schema appears, the schema will be "merged" with the previous > schema. The result will be a new schema that uses Union type to store the > columns where this is a type conflict. All of the batches (including the > batches that have already arrived) will be coerced into this new schema. > A new comparison function will be included to handle the comparison of Union > type. Comparison of union type will work as follows: > 1. All numeric types can be mutually compared, and will be compared using > Drill implicit cast rules. > 2. All other types will not be compared against other types, but only among > values of the same type. > 3. There will be an overall precedence of types with regards to ordering. > This precedence is not yet defined, but will be as part of the work on this > issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3854) IOB Exception : CONVERT_FROM (sal, int_be)
[
https://issues.apache.org/jira/browse/DRILL-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006121#comment-15006121
]
ASF GitHub Bot commented on DRILL-3854:
---
Github user hsuanyi closed the pull request at:
https://github.com/apache/drill/pull/256
> IOB Exception : CONVERT_FROM (sal, int_be)
> --
>
> Key: DRILL-3854
> URL: https://issues.apache.org/jira/browse/DRILL-3854
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.2.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Fix For: 1.4.0
>
> Attachments: log, run_time_code.txt
>
>
> CONVERT_FROM function results in IOB Exception
> Drill master commit id : b9afcf8f
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select salary from Emp;
> +-+
> | salary |
> +-+
> | 8 |
> | 9 |
> | 20 |
> | 95000 |
> | 85000 |
> | 9 |
> | 10 |
> | 87000 |
> | 8 |
> | 10 |
> | 99000 |
> +-+
> 11 rows selected (0.535 seconds)
> # create table using above Emp table
> create table tbl_int_be as select convert_to(salary, 'int_be') sal from Emp;
> 0: jdbc:drill:schema=dfs.tmp> alter session set `planner.slice_target`=1;
> +---++
> | ok |summary |
> +---++
> | true | planner.slice_target updated. |
> +---++
> 1 row selected (0.19 seconds)
> # Below query results in IOB on server.
> 0: jdbc:drill:schema=dfs.tmp> select convert_from(sal, 'int_be') from
> tbl_int_be order by sal;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: DrillBuf(ridx: 0, widx: 158,
> cap: 158/158, unwrapped: SlicedByteBuf(ridx: 0, widx: 158, cap: 158/158,
> unwrapped: UnsafeDirectLittleEndian(PooledUnsafeDirectByteBuf(ridx: 0, widx:
> 0, cap: 417/417.slice(158, 44)
> Fragment 2:0
> [Error Id: 4ee1361d-9877-45eb-bde6-57d5add9fe5e on centos-04.qa.lab:31010]
> (state=,code=0)
> # Apply convert_from function and project original column results in IOB on
> client. (because Error Id is missing)
> 0: jdbc:drill:schema=dfs.tmp> select convert_from(sal, 'int_be'), sal from
> tbl_int_be;
> Error: Unexpected RuntimeException: java.lang.IndexOutOfBoundsException:
> DrillBuf(ridx: 0, widx: 114, cap: 114/114, unwrapped: DrillBuf(ridx: 321,
> widx: 321, cap: 321/321, unwrapped:
> UnsafeDirectLittleEndian(PooledUnsafeDirectByteBuf(ridx: 0, widx: 0, cap:
> 321/321.slice(55, 103) (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3854) IOB Exception : CONVERT_FROM (sal, int_be)
[
https://issues.apache.org/jira/browse/DRILL-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006118#comment-15006118
]
Sean Hsuan-Yi Chu commented on DRILL-3854:
--
I noticed this:
alter session set `planner.slice_target`=1;
, which makes things different.
> IOB Exception : CONVERT_FROM (sal, int_be)
> --
>
> Key: DRILL-3854
> URL: https://issues.apache.org/jira/browse/DRILL-3854
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.2.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Fix For: 1.4.0
>
> Attachments: log, run_time_code.txt
>
>
> CONVERT_FROM function results in IOB Exception
> Drill master commit id : b9afcf8f
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select salary from Emp;
> +-+
> | salary |
> +-+
> | 8 |
> | 9 |
> | 20 |
> | 95000 |
> | 85000 |
> | 9 |
> | 10 |
> | 87000 |
> | 8 |
> | 10 |
> | 99000 |
> +-+
> 11 rows selected (0.535 seconds)
> # create table using above Emp table
> create table tbl_int_be as select convert_to(salary, 'int_be') sal from Emp;
> 0: jdbc:drill:schema=dfs.tmp> alter session set `planner.slice_target`=1;
> +---++
> | ok |summary |
> +---++
> | true | planner.slice_target updated. |
> +---++
> 1 row selected (0.19 seconds)
> # Below query results in IOB on server.
> 0: jdbc:drill:schema=dfs.tmp> select convert_from(sal, 'int_be') from
> tbl_int_be order by sal;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: DrillBuf(ridx: 0, widx: 158,
> cap: 158/158, unwrapped: SlicedByteBuf(ridx: 0, widx: 158, cap: 158/158,
> unwrapped: UnsafeDirectLittleEndian(PooledUnsafeDirectByteBuf(ridx: 0, widx:
> 0, cap: 417/417.slice(158, 44)
> Fragment 2:0
> [Error Id: 4ee1361d-9877-45eb-bde6-57d5add9fe5e on centos-04.qa.lab:31010]
> (state=,code=0)
> # Apply convert_from function and project original column results in IOB on
> client. (because Error Id is missing)
> 0: jdbc:drill:schema=dfs.tmp> select convert_from(sal, 'int_be'), sal from
> tbl_int_be;
> Error: Unexpected RuntimeException: java.lang.IndexOutOfBoundsException:
> DrillBuf(ridx: 0, widx: 114, cap: 114/114, unwrapped: DrillBuf(ridx: 321,
> widx: 321, cap: 321/321, unwrapped:
> UnsafeDirectLittleEndian(PooledUnsafeDirectByteBuf(ridx: 0, widx: 0, cap:
> 321/321.slice(55, 103) (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3854) IOB Exception : CONVERT_FROM (sal, int_be)
[
https://issues.apache.org/jira/browse/DRILL-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006117#comment-15006117
]
Sean Hsuan-Yi Chu commented on DRILL-3854:
--
[~jnadeau], [~cwestin] I see the point. I will try to understand why this bug
can show up in the production mode.
> IOB Exception : CONVERT_FROM (sal, int_be)
> --
>
> Key: DRILL-3854
> URL: https://issues.apache.org/jira/browse/DRILL-3854
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.2.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Fix For: 1.4.0
>
> Attachments: log, run_time_code.txt
>
>
> CONVERT_FROM function results in IOB Exception
> Drill master commit id : b9afcf8f
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select salary from Emp;
> +-+
> | salary |
> +-+
> | 8 |
> | 9 |
> | 20 |
> | 95000 |
> | 85000 |
> | 9 |
> | 10 |
> | 87000 |
> | 8 |
> | 10 |
> | 99000 |
> +-+
> 11 rows selected (0.535 seconds)
> # create table using above Emp table
> create table tbl_int_be as select convert_to(salary, 'int_be') sal from Emp;
> 0: jdbc:drill:schema=dfs.tmp> alter session set `planner.slice_target`=1;
> +---++
> | ok |summary |
> +---++
> | true | planner.slice_target updated. |
> +---++
> 1 row selected (0.19 seconds)
> # Below query results in IOB on server.
> 0: jdbc:drill:schema=dfs.tmp> select convert_from(sal, 'int_be') from
> tbl_int_be order by sal;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: DrillBuf(ridx: 0, widx: 158,
> cap: 158/158, unwrapped: SlicedByteBuf(ridx: 0, widx: 158, cap: 158/158,
> unwrapped: UnsafeDirectLittleEndian(PooledUnsafeDirectByteBuf(ridx: 0, widx:
> 0, cap: 417/417.slice(158, 44)
> Fragment 2:0
> [Error Id: 4ee1361d-9877-45eb-bde6-57d5add9fe5e on centos-04.qa.lab:31010]
> (state=,code=0)
> # Apply convert_from function and project original column results in IOB on
> client. (because Error Id is missing)
> 0: jdbc:drill:schema=dfs.tmp> select convert_from(sal, 'int_be'), sal from
> tbl_int_be;
> Error: Unexpected RuntimeException: java.lang.IndexOutOfBoundsException:
> DrillBuf(ridx: 0, widx: 114, cap: 114/114, unwrapped: DrillBuf(ridx: 321,
> widx: 321, cap: 321/321, unwrapped:
> UnsafeDirectLittleEndian(PooledUnsafeDirectByteBuf(ridx: 0, widx: 0, cap:
> 321/321.slice(55, 103) (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (DRILL-3180) Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and Netezza from Apache Drill
[
https://issues.apache.org/jira/browse/DRILL-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006030#comment-15006030
]
Olav Jordens edited comment on DRILL-3180 at 11/15/15 9:51 PM:
---
Hi Jacques,
I followed Magnus' suggestion to create the storage plugin to Netezza using a
database like so:
{
"type": "jdbc",
"driver": "org.netezza.Driver",
"url": "jdbc:netezza://edw-vip-prod:5480/SYSTEM",
"username": "username",
"password": "password",
"enabled": true
}
and it gives "Success" in version 1.2.0. However, when I try your version 1.3.0
linked above, I get the unable to create/update storage plugin error.
was (Author: olavj):
Hi Jacques,
I followed Magnus' suggestion to create the storage plugin to Netezza using a
database like so:
{
"type": "jdbc",
"driver": "org.netezza.Driver",
"url": "jdbc:netezza://edw-vip-kpr.nzc.co.nz:5480/SYSTEM",
"username": "username",
"password": "password",
"enabled": true
}
and it gives "Success" in version 1.2.0. However, when I try your version 1.3.0
linked above, I get the unable to create/update storage plugin error.
> Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and
> Netezza from Apache Drill
> ---
>
> Key: DRILL-3180
> URL: https://issues.apache.org/jira/browse/DRILL-3180
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Other
>Affects Versions: 1.0.0
>Reporter: Magnus Pierre
>Assignee: Jacques Nadeau
> Labels: Drill, JDBC, plugin
> Fix For: 1.2.0
>
> Attachments: patch.diff, pom.xml, storage-mpjdbc.zip
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> I have developed the base code for a JDBC storage-plugin for Apache Drill.
> The code is primitive but consitutes a good starting point for further
> coding. Today it provides primitive support for SELECT against RDBMS with
> JDBC.
> The goal is to provide complete SELECT support against RDBMS with push down
> capabilities.
> Currently the code is using standard JDBC classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (DRILL-3180) Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and Netezza from Apache Drill
[ https://issues.apache.org/jira/browse/DRILL-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006035#comment-15006035 ] Olav Jordens edited comment on DRILL-3180 at 11/15/15 9:50 PM: --- Another issue I am having in 1.2.0 (running local): When I run SELECT TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE FROM INFORMATION_SCHEMA.`TABLES` ORDER BY TABLE_NAME DESC; I see my storage plugin listed with two 'Tables' NZ_MAT_VALUE_TABLE and NZ_MAT_CONF, but none of the tables in Netezza. I have tried to do a select * ... limit 20 from the tables I know should be there, but each time I get a Table not found error. Also if I issue this query: select * from netezzaplugin.`NZ_MAT_CONF`; I get: org.apache.drill.common.exceptions.UserRemoteException: DATA_READ ERROR: The JDBC storage plugin failed while trying setup the SQL query. sql SELECT * FROM "NZM"."NZ_MAT_CONF" plugin netezzaplugin Fragment 0:0 So I guess my question is: How should I query my tables in Netezza? Thanks for your support - the potential for Drill looks really good to me once I get my head around it. I would ultimately like to query across Netezza and Hadoop. Is this geared towards the MapR distribution, or should all functionality be available in any case? Olav was (Author: olavj): Another issue I am having in 1.2.0 (running local): When I run SELECT TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE FROM INFORMATION_SCHEMA.`TABLES` ORDER BY TABLE_NAME DESC; I see my storage plugin listed with two 'Tables' NZ_MAT_VALUE_TABLE and NZ_MAT_CONF, but none of the tables in Netezza. I have tried to do a select * ... limit 20 from the tables I know should be there, but each time I get a Table not found error. Also if I issue this query: select * from netezzaplugin.`NZ_MAT_VALUE_TABLE`; I get: org.apache.drill.common.exceptions.UserRemoteException: DATA_READ ERROR: The JDBC storage plugin failed while trying setup the SQL query. sql SELECT * FROM "NZM"."NZ_MAT_CONF" plugin netezzaplugin Fragment 0:0 So I guess my question is: How should I query my tables in Netezza? Thanks for your support - the potential for Drill looks really good to me once I get my head around it. I would ultimately like to query across Netezza and Hadoop. Is this geared towards the MapR distribution, or should all functionality be available in any case? Olav > Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and > Netezza from Apache Drill > --- > > Key: DRILL-3180 > URL: https://issues.apache.org/jira/browse/DRILL-3180 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Affects Versions: 1.0.0 >Reporter: Magnus Pierre >Assignee: Jacques Nadeau > Labels: Drill, JDBC, plugin > Fix For: 1.2.0 > > Attachments: patch.diff, pom.xml, storage-mpjdbc.zip > > Original Estimate: 1m > Remaining Estimate: 1m > > I have developed the base code for a JDBC storage-plugin for Apache Drill. > The code is primitive but consitutes a good starting point for further > coding. Today it provides primitive support for SELECT against RDBMS with > JDBC. > The goal is to provide complete SELECT support against RDBMS with push down > capabilities. > Currently the code is using standard JDBC classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (DRILL-3180) Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and Netezza from Apache Drill
[ https://issues.apache.org/jira/browse/DRILL-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006035#comment-15006035 ] Olav Jordens edited comment on DRILL-3180 at 11/15/15 9:49 PM: --- Another issue I am having in 1.2.0 (running local): When I run SELECT TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE FROM INFORMATION_SCHEMA.`TABLES` ORDER BY TABLE_NAME DESC; I see my storage plugin listed with two 'Tables' NZ_MAT_VALUE_TABLE and NZ_MAT_CONF, but none of the tables in Netezza. I have tried to do a select * ... limit 20 from the tables I know should be there, but each time I get a Table not found error. Also if I issue this query: select * from netezzaplugin.`NZ_MAT_VALUE_TABLE`; I get: org.apache.drill.common.exceptions.UserRemoteException: DATA_READ ERROR: The JDBC storage plugin failed while trying setup the SQL query. sql SELECT * FROM "NZM"."NZ_MAT_CONF" plugin netezzaplugin Fragment 0:0 So I guess my question is: How should I query my tables in Netezza? Thanks for your support - the potential for Drill looks really good to me once I get my head around it. I would ultimately like to query across Netezza and Hadoop. Is this geared towards the MapR distribution, or should all functionality be available in any case? Olav was (Author: olavj): Another issue I am having in 1.2.0 (running local): When I run SELECT TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE FROM INFORMATION_SCHEMA.`TABLES` ORDER BY TABLE_NAME DESC; I see my storage plugin listed with two 'Tables' NZ_MAT_VALUE_TABLE and NZ_MAT_CONF, but none of the tables in Netezza. I have tried to do a select * ... limit 20 from the tables I know should be there, but each time I get a Table not found error. How should I query my tables in Netezza? Thanks for your support - the potential for Drill looks really good to me. Is this geared towards the MapR distribution, or should all functionality be available in any case? Olav > Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and > Netezza from Apache Drill > --- > > Key: DRILL-3180 > URL: https://issues.apache.org/jira/browse/DRILL-3180 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Affects Versions: 1.0.0 >Reporter: Magnus Pierre >Assignee: Jacques Nadeau > Labels: Drill, JDBC, plugin > Fix For: 1.2.0 > > Attachments: patch.diff, pom.xml, storage-mpjdbc.zip > > Original Estimate: 1m > Remaining Estimate: 1m > > I have developed the base code for a JDBC storage-plugin for Apache Drill. > The code is primitive but consitutes a good starting point for further > coding. Today it provides primitive support for SELECT against RDBMS with > JDBC. > The goal is to provide complete SELECT support against RDBMS with push down > capabilities. > Currently the code is using standard JDBC classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4056) Avro deserialization corrupts data
[
https://issues.apache.org/jira/browse/DRILL-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006044#comment-15006044
]
Stefán Baxter commented on DRILL-4056:
--
Hi,
Can you possibly make this available to me as patch for the master branch?
Regards,
-Stefán
On Sat, Nov 14, 2015 at 8:17 PM, Stefán Baxter
> Avro deserialization corrupts data
> --
>
> Key: DRILL-4056
> URL: https://issues.apache.org/jira/browse/DRILL-4056
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Other
>Affects Versions: 1.3.0
> Environment: Ubuntu 15.04 - Oracle Java
>Reporter: Stefán Baxter
>Assignee: Jason Altekruse
> Fix For: 1.3.0
>
> Attachments: test.zip
>
>
> I have an Avro file that support the following data/schema:
> {"field":"some", "classification":{"variant":"Gæst"}}
> When I select 10 rows from this file I get:
> +-+
> | EXPR$0|
> +-+
> | Gæst|
> | Voksen |
> | Voksen |
> | Invitation KIF KBH |
> | Invitation KIF KBH |
> | Ordinarie pris KBH |
> | Ordinarie pris KBH |
> | Biljetter 200 krBH |
> | Biljetter 200 krBH |
> | Biljetter 200 krBH |
> +-+
> The bug is that the field values are incorrectly de-serialized and the value
> from the previous row is retained if the subsequent row is shorter.
> The sql query:
> "select s.classification.variant variant from dfs. as s limit 10;"
> That way the "Ordinarie pris" becomes "Ordinarie pris KBH" because the
> previous row had the value "Invitation KIF KBH".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3180) Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and Netezza from Apache Drill
[ https://issues.apache.org/jira/browse/DRILL-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006035#comment-15006035 ] Olav Jordens commented on DRILL-3180: - Another issue I am having in 1.2.0 (running local): When I run SELECT TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE FROM INFORMATION_SCHEMA.`TABLES` ORDER BY TABLE_NAME DESC; I see my storage plugin listed with two 'Tables' NZ_MAT_VALUE_TABLE and NZ_MAT_CONF, but none of the tables in Netezza. I have tried to do a select * ... limit 20 from the tables I know should be there, but each time I get a Table not found error. How should I query my tables in Netezza? Thanks for your support - the potential for Drill looks really good to me. Is this geared towards the MapR distribution, or should all functionality be available in any case? Olav > Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and > Netezza from Apache Drill > --- > > Key: DRILL-3180 > URL: https://issues.apache.org/jira/browse/DRILL-3180 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Affects Versions: 1.0.0 >Reporter: Magnus Pierre >Assignee: Jacques Nadeau > Labels: Drill, JDBC, plugin > Fix For: 1.2.0 > > Attachments: patch.diff, pom.xml, storage-mpjdbc.zip > > Original Estimate: 1m > Remaining Estimate: 1m > > I have developed the base code for a JDBC storage-plugin for Apache Drill. > The code is primitive but consitutes a good starting point for further > coding. Today it provides primitive support for SELECT against RDBMS with > JDBC. > The goal is to provide complete SELECT support against RDBMS with push down > capabilities. > Currently the code is using standard JDBC classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (DRILL-3180) Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and Netezza from Apache Drill
[
https://issues.apache.org/jira/browse/DRILL-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006030#comment-15006030
]
Olav Jordens edited comment on DRILL-3180 at 11/15/15 9:08 PM:
---
Hi Jacques,
I followed Magnus' suggestion to create the storage plugin to Netezza using a
database like so:
{
"type": "jdbc",
"driver": "org.netezza.Driver",
"url": "jdbc:netezza://edw-vip-kpr.nzc.co.nz:5480/SYSTEM",
"username": "username",
"password": "password",
"enabled": true
}
and it gives "Success" in version 1.2.0. However, when I try your version 1.3.0
linked above, I get the unable to create/update storage plugin error.
was (Author: olavj):
Hi Jacques,
I followed Magnus' suggestion to create the storage plugin to Netezza using a
database like so:
{
"type": "jdbc",
"driver": "org.netezza.Driver",
"url": "jdbc:netezza://edw-vip-kpr.nzc.co.nz:5480/SYSTEM",
"username": "admin",
"password": "password",
"enabled": true
}
and it gives "Success" in version 1.2.0. However, when I try your version 1.3.0
linked above, I get the unable to create/update storage plugin error.
> Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and
> Netezza from Apache Drill
> ---
>
> Key: DRILL-3180
> URL: https://issues.apache.org/jira/browse/DRILL-3180
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Other
>Affects Versions: 1.0.0
>Reporter: Magnus Pierre
>Assignee: Jacques Nadeau
> Labels: Drill, JDBC, plugin
> Fix For: 1.2.0
>
> Attachments: patch.diff, pom.xml, storage-mpjdbc.zip
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> I have developed the base code for a JDBC storage-plugin for Apache Drill.
> The code is primitive but consitutes a good starting point for further
> coding. Today it provides primitive support for SELECT against RDBMS with
> JDBC.
> The goal is to provide complete SELECT support against RDBMS with push down
> capabilities.
> Currently the code is using standard JDBC classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3180) Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and Netezza from Apache Drill
[
https://issues.apache.org/jira/browse/DRILL-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006030#comment-15006030
]
Olav Jordens commented on DRILL-3180:
-
Hi Jacques,
I followed Magnus' suggestion to create the storage plugin to Netezza using a
database like so:
{
"type": "jdbc",
"driver": "org.netezza.Driver",
"url": "jdbc:netezza://edw-vip-kpr.nzc.co.nz:5480/SYSTEM",
"username": "admin",
"password": "password",
"enabled": true
}
and it gives "Success" in version 1.2.0. However, when I try your version 1.3.0
linked above, I get the unable to create/update storage plugin error.
> Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and
> Netezza from Apache Drill
> ---
>
> Key: DRILL-3180
> URL: https://issues.apache.org/jira/browse/DRILL-3180
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Other
>Affects Versions: 1.0.0
>Reporter: Magnus Pierre
>Assignee: Jacques Nadeau
> Labels: Drill, JDBC, plugin
> Fix For: 1.2.0
>
> Attachments: patch.diff, pom.xml, storage-mpjdbc.zip
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> I have developed the base code for a JDBC storage-plugin for Apache Drill.
> The code is primitive but consitutes a good starting point for further
> coding. Today it provides primitive support for SELECT against RDBMS with
> JDBC.
> The goal is to provide complete SELECT support against RDBMS with push down
> capabilities.
> Currently the code is using standard JDBC classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3152) Apache Drill 1.0 not able to query MongoDB 3.0.
[
https://issues.apache.org/jira/browse/DRILL-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006020#comment-15006020
]
Jacques Nadeau commented on DRILL-3152:
---
The 1.3 version of Drill has not yet been released. An candidate build can be
found here:
http://people.apache.org/~jacques/apache-drill-1.3.0.rc2/
> Apache Drill 1.0 not able to query MongoDB 3.0.
>
>
> Key: DRILL-3152
> URL: https://issues.apache.org/jira/browse/DRILL-3152
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - MongoDB
>Affects Versions: 0.9.0, 1.0.0
> Environment: The environment is as follows:
> Windows 7
> MongoDB 3 Wiredtiger (installed locally)
> Apache Drill 1.0 (installed locally)
>Reporter: Trent Telfer
> Labels: mongodb, mongodb3, windows7, wiredtiger
> Fix For: Future
>
>
> I have been trying to get Apache Drill 1.0, and previously 0.9 to work with
> MongoDB 3.0 Wiredtiger. I have no problem starting Apache Drill using the
> following, but I am having problems querying MongoDB:
> *./sqlline.bat*
> *!connect jdbc:drill:zk=local*
> *SHOW DATABASES;*
> +-+
> | SCHEMA_NAME |
> +-+
> | INFORMATION_SCHEMA |
> | cp.default |
> | dfs.default |
> | dfs.root|
> | dfs.tmp |
> | mongo.admin |
> | mongo.alliance_db |
> | mongo.local |
> | sys |
> +-+
> *USE mongo.alliance_db;*
> +---++
> | ok |summary |
> +---++
> | true | Default schema changed to [mongo.alliance_db] |
> +---++
> 1 row selected (0.116 seconds)
> *SELECT * FROM price_daily_ngi;*
> May 20, 2015 11:14:40 AM
> org.apache.calcite.sql.validate.SqlValidatorException
> SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table
> 'price_daily_ngi' not found
> May 20, 2015 11:14:40 AM org.apache.calcite.runtime.CalciteException
> SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1,
> column 15 to line 1, column 29: Table 'price_daily_ngi' not found
> Error: PARSE ERROR: From line 1, column 15 to line 1, column 29: Table
> 'price_daily_ngi' not found
> [Error Id: 6414a69d-55a0-4918-8f95-10a920e4dc6b on PCV:31010] (state=,code=0)
> MongoDB storage configuration:
> {
> "type": "mongo",
> "connection": "mongodb://localhost:27017",
> "enabled": true
> }
> The collection price_daily_ngi exists and works with normal MongoDB queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3180) Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and Netezza from Apache Drill
[ https://issues.apache.org/jira/browse/DRILL-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006019#comment-15006019 ] Jacques Nadeau commented on DRILL-3180: --- You should also give the latest 1.3 release candidate a try. There were some issues with multi-schemas in 1.2 that should be resolved in 1.3. I've tested it yet with Netezza but it addressed a bunch of issues with SQL Server and Oracle. You can grab the binary here until the release is finalized: http://people.apache.org/~jacques/apache-drill-1.3.0.rc2/ > Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and > Netezza from Apache Drill > --- > > Key: DRILL-3180 > URL: https://issues.apache.org/jira/browse/DRILL-3180 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Affects Versions: 1.0.0 >Reporter: Magnus Pierre >Assignee: Jacques Nadeau > Labels: Drill, JDBC, plugin > Fix For: 1.2.0 > > Attachments: patch.diff, pom.xml, storage-mpjdbc.zip > > Original Estimate: 1m > Remaining Estimate: 1m > > I have developed the base code for a JDBC storage-plugin for Apache Drill. > The code is primitive but consitutes a good starting point for further > coding. Today it provides primitive support for SELECT against RDBMS with > JDBC. > The goal is to provide complete SELECT support against RDBMS with push down > capabilities. > Currently the code is using standard JDBC classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4047) Select with options
[
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006011#comment-15006011
]
ASF GitHub Bot commented on DRILL-4047:
---
Github user jacques-n commented on a diff in the pull request:
https://github.com/apache/drill/pull/246#discussion_r44876693
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/TestSelectWithOption.java ---
@@ -0,0 +1,203 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import static java.lang.String.format;
+import static org.apache.drill.TestBuilder.listOf;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+
+import org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class TestSelectWithOption extends BaseTestQuery {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(WorkspaceSchemaFactory.class);
+
+ private File genCSVFile(String name, String... rows) throws IOException {
+File file = new File(format("target/%s_%s.csv",
this.getClass().getName(), name));
+try (FileWriter fw = new FileWriter(file)) {
+ for (int i = 0; i < rows.length; i++) {
+fw.append(rows[i] + "\n");
+ }
+}
+return file;
+ }
+
+ private String genCSVTable(String name, String... rows) throws
IOException {
+File f = genCSVFile(name, rows);
+return format("dfs.`${WORKING_PATH}/%s`", f.getPath());
+ }
+
+ private void testWithResult(String query, Object... expectedResult)
throws Exception {
+TestBuilder builder = testBuilder()
+.sqlQuery(query)
+.ordered()
+.baselineColumns("columns");
+for (Object o : expectedResult) {
+ builder = builder.baselineValues(o);
+}
+builder.build().run();
+ }
+
+ @Test
+ public void testTextFieldDelimiter() throws Exception {
+String tableName = genCSVTable("testTextFieldDelimiter",
+"\"b\"|\"0\"",
+"\"b\"|\"1\"",
+"\"b\"|\"2\"");
+
+String queryTemplate =
+"select columns from table(%s (type => 'TeXT', fieldDelimiter =>
'%s'))";
+testWithResult(format(queryTemplate, tableName, ","),
+listOf("b\"|\"0"),
+listOf("b\"|\"1"),
+listOf("b\"|\"2")
+ );
+testWithResult(format(queryTemplate, tableName, "|"),
+listOf("b", "0"),
+listOf("b", "1"),
+listOf("b", "2")
+ );
+ }
+
+ @Test @Ignore // It does not look like lineDelimiter is working
+ public void testTextLineDelimiter() throws Exception {
+String tableName = genCSVTable("testTextLineDelimiter",
+"\"b\"|\"0\"",
+"\"b\"|\"1\"",
+"\"b\"|\"2\"");
+
+testWithResult(format("select columns from table(%s(type => 'TeXT',
lineDelimiter => '|'))", tableName),
+listOf("\"b\""),
+listOf("\"0\"", "\"b\""),
+listOf("\"1\"", "\"b\""),
+listOf("\"2\"")
+ );
+ }
+
+ @Test
+ public void testTextQuote() throws Exception {
+String tableName = genCSVTable("testTextQuote",
+"\"b\"|\"0\"",
+"\"b\"|\"1\"",
+"\"b\"|\"2\"");
+
+testWithResult(format("select columns from table(%s(type => 'TeXT',
fieldDelimiter => '|', quote => '@'))", tableName),
+listOf("\"b\"", "\"0\""),
+listOf("\"b\"", "\"1\""),
+listOf("\"b\"", "\"2\"")
+);
+
+String quoteTableName = genCSVTable("testTextQuote2",
+"@b@|@0@",
+"@b$@c@|@1@");
+// It seems that a parameter can not be called "escape"
+testWithResult(format("select columns from table(%s(`escape` => '$',
type => 'TeXT', fieldDelimiter => '|', quote => '@'))",
