[drill] 02/04: DRILL-7448: Fix warnings when running Drill memory tests
This is an automated email from the ASF dual-hosted git repository.
arina pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/drill.git
commit 120e10bb8d79d99117464404bb6c870a10169d92
Author: Bohdan Kazydub
AuthorDate: Mon Nov 18 19:51:55 2019 +0200
DRILL-7448: Fix warnings when running Drill memory tests
closes #1902
---
common/pom.xml | 8 -
common/src/test/resources/logback-test.xml | 47 ++
2 files changed, 22 insertions(+), 33 deletions(-)
diff --git a/common/pom.xml b/common/pom.xml
index 743374d..47b60fc 100644
--- a/common/pom.xml
+++ b/common/pom.xml
@@ -102,14 +102,6 @@
2.9
-
-
- org.codehaus.janino
- janino
- test
-
-
diff --git a/common/src/test/resources/logback-test.xml
b/common/src/test/resources/logback-test.xml
index 06004a6..f47af26 100644
--- a/common/src/test/resources/logback-test.xml
+++ b/common/src/test/resources/logback-test.xml
@@ -20,26 +20,26 @@
-->
-
-
-
-true
-1
-true
-${LILITH_HOSTNAME:-localhost}
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
[drill] 01/04: DRILL-7388: Kafka improvements
This is an automated email from the ASF dual-hosted git repository. arina pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/drill.git commit ffd3c352586a5884747edcc5a93b0c625a47e100 Author: Arina Ielchiieva AuthorDate: Fri Nov 15 16:01:52 2019 +0200 DRILL-7388: Kafka improvements 1. Upgraded Kafka libraries to 2.3.1 (DRILL-6739). 2. Added new options to support the same features as native JSON reader: a. store.kafka.reader.skip_invalid_records, default: false (DRILL-6723); b. store.kafka.reader.allow_nan_inf, default: true; c. store.kafka.reader.allow_escape_any_char, default: false. 3. Fixed issue when Kafka topic contains only one message (DRILL-7388). 4. Replaced Gson parser with Jackson to parse JSON in the same manner as Drill native Json reader. 5. Performance improvements: Kafka consumers will be closed async, fixed issue with resource leak (DRILL-7290), moved to debug unnecessary info logging. 6. Updated bootstrap-storage-plugins.json to reflect actual Kafka connection properties. 7. Added unit tests. 8. Refactoring and code clean up. closes #1901 --- contrib/storage-kafka/README.md| 12 +- contrib/storage-kafka/pom.xml | 6 +- .../drill/exec/store/kafka/KafkaAsyncCloser.java | 105 + .../drill/exec/store/kafka/KafkaGroupScan.java | 52 --- .../drill/exec/store/kafka/KafkaNodeProcessor.java | 1 - .../exec/store/kafka/KafkaPartitionScanSpec.java | 10 +- .../store/kafka/KafkaPartitionScanSpecBuilder.java | 111 +++-- .../store/kafka/KafkaPushDownFilterIntoScan.java | 25 +-- .../drill/exec/store/kafka/KafkaRecordReader.java | 85 +- .../exec/store/kafka/KafkaScanBatchCreator.java| 14 +- .../drill/exec/store/kafka/KafkaScanSpec.java | 3 +- .../drill/exec/store/kafka/KafkaStoragePlugin.java | 21 +-- .../exec/store/kafka/KafkaStoragePluginConfig.java | 35 ++--- .../drill/exec/store/kafka/KafkaSubScan.java | 2 +- .../drill/exec/store/kafka/MessageIterator.java| 50 -- .../drill/exec/store/kafka/MetaDataField.java | 10 +- .../apache/drill/exec/store/kafka/ReadOptions.java | 95 +++ .../store/kafka/decoders/JsonMessageReader.java| 95 +++ .../exec/store/kafka/decoders/MessageReader.java | 10 +- .../store/kafka/decoders/MessageReaderFactory.java | 4 +- .../store/kafka/schema/KafkaMessageSchema.java | 14 +- .../store/kafka/schema/KafkaSchemaFactory.java | 5 +- .../main/resources/bootstrap-storage-plugins.json | 10 +- .../exec/store/kafka/KafkaFilterPushdownTest.java | 22 +-- .../exec/store/kafka/KafkaMessageGenerator.java| 99 ++-- .../drill/exec/store/kafka/KafkaQueriesTest.java | 173 ++--- .../drill/exec/store/kafka/KafkaTestBase.java | 12 +- .../exec/store/kafka/MessageIteratorTest.java | 5 +- .../drill/exec/store/kafka/TestKafkaSuit.java | 81 +- .../store/kafka/cluster/EmbeddedKafkaCluster.java | 57 --- .../kafka/decoders/MessageReaderFactoryTest.java | 6 +- .../java/org/apache/drill/exec/ExecConstants.java | 10 ++ .../exec/server/options/SystemOptionManager.java | 3 + .../java-exec/src/main/resources/drill-module.conf | 3 + 34 files changed, 807 insertions(+), 439 deletions(-) diff --git a/contrib/storage-kafka/README.md b/contrib/storage-kafka/README.md index a63731f..a26c6e1 100644 --- a/contrib/storage-kafka/README.md +++ b/contrib/storage-kafka/README.md @@ -211,18 +211,22 @@ Note: - store.kafka.record.reader system option can be used for setting record reader and default is org.apache.drill.exec.store.kafka.decoders.JsonMessageReader - Default store.kafka.poll.timeout is set to 200, user has to set this accordingly -- Custom record reader can be implemented by extending org.apache.drill.exec.store.kafka.decoders.MessageReader and setting store.kafka.record.reader accordinlgy +- Custom record reader can be implemented by extending org.apache.drill.exec.store.kafka.decoders.MessageReader and setting store.kafka.record.reader accordingly -In case of JSON message format, following system options can be used accordingly. More details can be found in [Drill Json Model](https://drill.apache.org/docs/json-data-model/) and in [Drill system options configurations](https://drill.apache.org/docs/configuration-options-introduction/) +In case of JSON message format, following system / session options can be used accordingly. More details can be found in [Drill Json Model](https://drill.apache.org/docs/json-data +-model/) and in [Drill system options configurations](https://drill.apache.org/docs/configuration-options-introduction/) - + ALTER SESSION SET `store.kafka.record.reader` = 'org.apache.drill.exec.store.kafka.decoders.JsonMessageReader'; ALTER SESSION SET `store.kafka.poll.timeout` = 200; ALTER SES
[drill] branch master updated (8e6cdbe -> 0230627)
This is an automated email from the ASF dual-hosted git repository. arina pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/drill.git. from 8e6cdbe DRILL-7446: Fix Eclipse compilation issue in AbstractParquetGroupScan new ffd3c35 DRILL-7388: Kafka improvements new 120e10b DRILL-7448: Fix warnings when running Drill memory tests new 5b9380e DRILL-7441: Fix issues with fillEmpties, offset vectors new 0230627 DRILL-7445: Create batch copier based on result set framework The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: common/pom.xml | 8 - .../java/org/apache/drill/common/types/Types.java | 32 +- common/src/test/resources/logback-test.xml | 47 +- .../apache/drill/exec/hive/TestHiveStorage.java| 12 +- contrib/storage-kafka/README.md| 12 +- contrib/storage-kafka/pom.xml | 6 +- .../drill/exec/store/kafka/KafkaAsyncCloser.java | 105 .../drill/exec/store/kafka/KafkaGroupScan.java | 52 +- .../drill/exec/store/kafka/KafkaNodeProcessor.java | 1 - .../exec/store/kafka/KafkaPartitionScanSpec.java | 10 +- .../store/kafka/KafkaPartitionScanSpecBuilder.java | 111 ++-- .../store/kafka/KafkaPushDownFilterIntoScan.java | 25 +- .../drill/exec/store/kafka/KafkaRecordReader.java | 85 ++- .../exec/store/kafka/KafkaScanBatchCreator.java| 14 +- .../drill/exec/store/kafka/KafkaScanSpec.java | 3 +- .../drill/exec/store/kafka/KafkaStoragePlugin.java | 21 +- .../exec/store/kafka/KafkaStoragePluginConfig.java | 35 +- .../drill/exec/store/kafka/KafkaSubScan.java | 2 +- .../drill/exec/store/kafka/MessageIterator.java| 50 +- .../drill/exec/store/kafka/MetaDataField.java | 10 +- .../apache/drill/exec/store/kafka/ReadOptions.java | 95 +++ .../store/kafka/decoders/JsonMessageReader.java| 95 ++- .../exec/store/kafka/decoders/MessageReader.java | 10 +- .../store/kafka/decoders/MessageReaderFactory.java | 4 +- .../store/kafka/schema/KafkaMessageSchema.java | 14 +- .../store/kafka/schema/KafkaSchemaFactory.java | 5 +- .../main/resources/bootstrap-storage-plugins.json | 10 +- .../exec/store/kafka/KafkaFilterPushdownTest.java | 22 +- .../exec/store/kafka/KafkaMessageGenerator.java| 99 ++- .../drill/exec/store/kafka/KafkaQueriesTest.java | 173 +- .../drill/exec/store/kafka/KafkaTestBase.java | 12 +- .../exec/store/kafka/MessageIteratorTest.java | 5 +- .../drill/exec/store/kafka/TestKafkaSuit.java | 81 +-- .../store/kafka/cluster/EmbeddedKafkaCluster.java | 57 +- .../kafka/decoders/MessageReaderFactoryTest.java | 6 +- .../java/org/apache/drill/exec/ExecConstants.java | 10 + .../physical/impl/aggregate/BatchIterator.java | 2 - .../impl/protocol/IndirectContainerAccessor.java | 100 .../impl/protocol/VectorContainerAccessor.java | 41 +- .../physical/impl/validate/BatchValidator.java | 174 +++--- .../exec/physical/resultSet/ResultSetCopier.java | 189 ++ .../exec/physical/resultSet/ResultSetReader.java | 7 + .../resultSet/impl/ResultSetCopierImpl.java| 313 ++ .../resultSet/impl/ResultSetLoaderImpl.java| 1 + .../resultSet/impl/ResultSetReaderImpl.java| 3 + .../exec/physical/rowSet/RowSetFormatter.java | 13 +- .../exec/record/selection/SelectionVector2.java| 13 +- .../selection/SelectionVector2Builder.java}| 35 +- .../exec/server/options/SystemOptionManager.java | 3 + .../drill/exec/vector/complex/fn/JsonReader.java | 1 + .../drill/exec/work/batch/BaseRawBatchBuffer.java | 1 - .../java-exec/src/main/resources/drill-module.conf | 3 + .../test/java/org/apache/drill/TestBugFixes.java | 31 +- .../test/java/org/apache/drill/TestCTASJson.java | 20 +- .../java/org/apache/drill/TestFrameworkTest.java | 12 +- .../java/org/apache/drill/TestJoinNullable.java| 1 - .../exec/compile/TestLargeFileCompilation.java | 22 +- .../physical/impl/TestNestedDateTimeTimestamp.java | 10 +- .../physical/impl/join/TestMergeJoinAdvanced.java | 22 +- .../impl/protocol/TestOperatorRecordBatch.java | 9 +- .../physical/impl/validate/TestBatchValidator.java | 2 +- .../writer/TestCorruptParquetDateCorrection.java | 23 +- .../resultSet/impl/TestResultSetCopier.java| 663 + .../resultSet/impl/TestResultSetReader.java| 13 +- .../exec/physical/rowSet/TestMapAccessors.java | 16 +- .../physical/rowSet/TestOffsetVectorWriter.java| 14 +- .../exec/physical/rowSet/TestScalarAccessors.java | 217 ++- .../drill/exec/planner/sql/TestDrillSQLWorker.java | 8 +- .../org/apache/d
[drill] 04/04: DRILL-7445: Create batch copier based on result set framework
This is an automated email from the ASF dual-hosted git repository.
arina pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/drill.git
commit 02306277bfacaece652ba76025a6f82def9570b6
Author: Paul Rogers
AuthorDate: Sun Nov 10 16:17:49 2019 -0800
DRILL-7445: Create batch copier based on result set framework
The result set framework now provides both a reader and writer.
This PR provides a copier that copies batches using this
framework. Such a copier can:
- Copy selected records
- Copy all records, such as for an SV2 or SV4
The copier uses the result set loader to create uniformly-sized
output batches from input batches of any size. It does this
by merging or splitting input batches as needed.
Since the result set reader handles both SV2 and SV4s, the
copier can filter or reorder rows based on the SV associated
with the input batch.
This version assumes single stream of input batches, and handles
any schema changes in that input by creating output batches
that track the input schema. This would be used in, say, the
selection vector remover. A different design is needed for merging
such as in the merging receiver.
Adds a "copy" method to the column writers. Copy is implemented
by doing a direct memory copy from source to destination vectors.
A unit test verifies functionality for various use cases
and data types.
closes #1899
---
.../java/org/apache/drill/common/types/Types.java | 32 +-
.../physical/impl/aggregate/BatchIterator.java | 2 -
.../impl/protocol/IndirectContainerAccessor.java | 100
.../impl/protocol/VectorContainerAccessor.java | 41 +-
.../exec/physical/resultSet/ResultSetCopier.java | 189 ++
.../exec/physical/resultSet/ResultSetReader.java | 7 +
.../resultSet/impl/ResultSetCopierImpl.java| 313 ++
.../resultSet/impl/ResultSetLoaderImpl.java| 1 +
.../resultSet/impl/ResultSetReaderImpl.java| 3 +
.../exec/physical/rowSet/RowSetFormatter.java | 13 +-
.../exec/record/selection/SelectionVector2.java| 13 +-
.../selection/SelectionVector2Builder.java}| 31 +-
.../impl/protocol/TestOperatorRecordBatch.java | 9 +-
.../resultSet/impl/TestResultSetCopier.java| 663 +
.../resultSet/impl/TestResultSetReader.java| 13 +-
.../org/apache/drill/test/BaseDirTestWatcher.java | 12 +-
.../java/org/apache/drill/test/ClusterFixture.java | 59 +-
.../main/codegen/templates/ColumnAccessors.java| 21 +-
.../drill/exec/record/MaterializedField.java | 1 +
.../drill/exec/vector/accessor/ColumnWriter.java | 9 +
.../drill/exec/vector/accessor/ObjectReader.java | 7 +
.../drill/exec/vector/accessor/ObjectWriter.java | 11 +-
.../accessor/convert/AbstractWriteConverter.java | 6 +
.../accessor/reader/AbstractObjectReader.java | 1 +
.../accessor/reader/AbstractTupleReader.java | 16 +-
.../vector/accessor/reader/ArrayReaderImpl.java| 1 -
.../vector/accessor/reader/BaseScalarReader.java | 12 +
.../vector/accessor/reader/OffsetVectorReader.java | 1 +
.../vector/accessor/reader/UnionReaderImpl.java| 12 +-
.../accessor/writer/AbstractArrayWriter.java | 14 +
.../accessor/writer/AbstractObjectWriter.java | 11 +-
.../accessor/writer/AbstractTupleWriter.java | 22 +-
.../vector/accessor/writer/BaseVarWidthWriter.java | 8 +-
.../vector/accessor/writer/BitColumnWriter.java| 8 +
.../exec/vector/accessor/writer/MapWriter.java | 9 +-
.../accessor/writer/NullableScalarWriter.java | 9 +
.../accessor/writer/OffsetVectorWriterImpl.java| 6 +
.../vector/accessor/writer/UnionWriterImpl.java| 13 +-
.../accessor/writer/dummy/DummyArrayWriter.java| 4 +
.../accessor/writer/dummy/DummyScalarWriter.java | 4 +
40 files changed, 1570 insertions(+), 137 deletions(-)
diff --git a/common/src/main/java/org/apache/drill/common/types/Types.java
b/common/src/main/java/org/apache/drill/common/types/Types.java
index 57a752e..5cdae56 100644
--- a/common/src/main/java/org/apache/drill/common/types/Types.java
+++ b/common/src/main/java/org/apache/drill/common/types/Types.java
@@ -806,16 +806,32 @@ public class Types {
return typeBuilder;
}
- public static boolean isEquivalent(MajorType type1, MajorType type2) {
+ /**
+ * Check if two "core" types are the same, ignoring subtypes and
+ * children. Primarily for non-complex types.
+ *
+ * @param type1 first type
+ * @param type2 second type
+ * @return true if the two types are are the same minor type, mode,
+ * precision and scale
+ */
+
+ public static boolean isSameType(MajorType type1, MajorType type2) {
+return type1.getMinorType() == type2.getMinorType() &&
+ type1.getMode() == type2.getMode() &&
+ type1.getScale() == type2.g
