[jira] [Created] (DRILL-7275) CTAS + CTE query fails with IllegalStateException: Read batch count [%d] should be greater than zero [0]
Khurram Faraaz created DRILL-7275: - Summary: CTAS + CTE query fails with IllegalStateException: Read batch count [%d] should be greater than zero [0] Key: DRILL-7275 URL: https://issues.apache.org/jira/browse/DRILL-7275 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Affects Versions: 1.15.0 Reporter: Khurram Faraaz CTAS + CTE query fails with IllegalStateException: Read batch count [%d] should be greater than zero [0] Precondition check fails on line 47 in VarLenFixedEntryReader.java {noformat} 44 final int expectedDataLen = columnPrecInfo.precision; 45 final int entrySz = 4 + columnPrecInfo.precision; 46 final int readBatch = getFixedLengthMaxRecordsToRead(valuesToRead, entrySz); 47 Preconditions.checkState(readBatch > 0, "Read batch count [%d] should be greater than zero", readBatch); {noformat} Stack trace from drillbit.log, also has the failing query. {noformat} 2019-05-13 14:40:14,090 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query with id 23268c40-ef3a-6349-5901-5762f6888971 issued by scoop_stc: CREATE TABLE TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityProxyDailyReturn AS WITH che AS ( SELECT * FROM TEST_TEMPLATE_SCHEMA_creid.tbl_c_CompositeHierarchyEntry_TimeVarying WHERE CompositeHierarchyName = 'AxiomaRegion/AxiomaSector/VectorUniverse' AND state = 'DupesRemoved' AND CompositeLevel = 'AxiomaRegion_1/AxiomaSector_1/VectorUniverse_0' ), ef AS (SELECT * FROM TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityDailyReturn_FXAdjusted WHERE Status = 'PresentInRawData'), d AS (SELECT * FROM TEST_TEMPLATE_SCHEMA_creid.tbl_r_BusinessDate WHERE IsWeekday), x AS ( SELECT che.CompositeHierarchyName, che.State, che.CompositeNodeName, d.`Date` AS RecordDate, COUNT(che.CompositeNodeName) AS countDistinctConstituents, COUNT(ef.VectorListingId) AS countDataPoints, AVG(ef.DailyReturn) AS AvgReturn, AVG(ef.DailyReturnUSD) AS AvgReturnUSD, AVG(ef.NotionalReturnUSD) AS AvgNotionalReturnUSD FROM d INNER JOIN che ON d.`Date` BETWEEN che.CompositeUltimateChildStartDate AND che.CompositeUltimateChildEndDate LEFT OUTER JOIN ef ON d.`Date` = ef.RecordDate AND 'VectorListingId_' || CAST(ef.VectorListingId AS VARCHAR(100)) = che.UltimateChild GROUP BY che.CompositeHierarchyName, che.State, che.CompositeNodeName, d.`Date`, d.IsWeekday, d.IsHoliday ) SELECT * FROM x 2019-05-13 14:40:16,971 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO o.a.d.e.p.s.h.CreateTableHandler - Creating persistent table [tbl_c_EquityProxyDailyReturn]. ... ... 2019-05-13 14:40:20,036 [23268c40-ef3a-6349-5901-5762f6888971:frag:6:10] INFO o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: Error in parquet record reader. Message: Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet Total records read: 0 Row group index: 0 Records in row group: 3243 Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root { optional int64 VectorListingId; optional int32 RecordDate (DATE); required binary Status (UTF8); required binary CurrencyISO (UTF8); optional double DailyReturn; optional double DailyReturnUSD; optional double NotionalReturnUSD; } , metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks: [BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId] optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4}, ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate (DATE) [RLE, BIT_PACKED, PLAIN], 26021}, ColumnMetaData\{UNCOMPRESSED [Status] required binary Status (UTF8) [BIT_PACKED, PLAIN], 39050}, ColumnMetaData\{UNCOMPRESSED [CurrencyISO] required binary CurrencyISO (UTF8) [BIT_PACKED, PLAIN], 103968}, ColumnMetaData\{UNCOMPRESSED [DailyReturn] optional double DailyReturn [RLE, BIT_PACKED, PLAIN], 126715}, ColumnMetaData\{UNCOMPRESSED [DailyReturnUSD] optional double DailyReturnUSD [RLE, BIT_PACKED, PLAIN], 152732}, ColumnMetaData\{UNCOMPRESSED [NotionalReturnUSD] optional double NotionalReturnUSD [RLE, BIT_PACKED, PLAIN], 178749}]}]} (Error in parquet record reader. ... ... Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet Total records read: 0 Row group index: 0 Records in row group: 3243 Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root { optional int64 VectorListingId; optional int32 RecordDate (DATE); required binary Status (UTF8); required binary CurrencyISO (UTF8); optional double DailyReturn; optional double DailyReturnUSD; optional double NotionalReturnUSD; } , metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks: [BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId] optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4}, ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate (DATE) [RLE,
[jira] [Created] (DRILL-7256) Query over empty Hive tables fails, we will need to print heap usagePercent details in error message
Khurram Faraaz created DRILL-7256: - Summary: Query over empty Hive tables fails, we will need to print heap usagePercent details in error message Key: DRILL-7256 URL: https://issues.apache.org/jira/browse/DRILL-7256 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.15.0 Reporter: Khurram Faraaz The below query from Drill's web UI on Hive tables failed due to not enough heap memory to run this query. It fails intermittently from Drill web UI, and note that the two Hive tables used in the query are empty, meaning they have no data in them. The query does not fail when run from sqlline. The error message does not provide information about the usagePercent of heap. It will be useful to provide heap usagePercent information as part of the error message in QueryWrapper.java when usagePercent > HEAP_MEMORY_FAILURE_THRESHOLD Drill 1.15.0 Failing query. {noformat} SELECT a.event_id FROM hive.cust_bhsf_ce_blob a, hive.t_fct_clinical_event b where a.event_id=b.event_id and a.blob_contents not like '%dd:contenttype="TESTS"%' and b.EVENT_RELATIONSHIP_CD='B' and b.EVENT_CLASS_CD in ('DOC') and b.entry_mode_cd='Web' and b.RECORD_STATUS_CD='Active' and b.RESULT_STATUS_CD ='Auth (Verified)' and substring(b.valid_until_dt_tm,1,10) >='2017-12-30' and substring(b.event_end_date,1,10) >='2018-01-01' {noformat} Stack trace from drillbit.log {noformat} 2019-05-09 16:25:58,472 [qtp1934687-790] ERROR o.a.d.e.server.rest.QueryResources - Query from Web UI Failed org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: There is not enough heap memory to run this query using the web interface. Please try a query with fewer columns or with a filter or limit condition to limit the data returned. You can also try an ODBC/JDBC client. [Error Id: 91668f42-d88e-426b-b1fe-c0d042700500 ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.15.0.5-mapr.jar:1.15.0.5-mapr] at org.apache.drill.exec.server.rest.QueryWrapper.run(QueryWrapper.java:103) ~[drill-java-exec-1.15.0.5-mapr.jar:1.15.0.5-mapr] at org.apache.drill.exec.server.rest.QueryResources.submitQueryJSON(QueryResources.java:72) ~[drill-java-exec-1.15.0.5-mapr.jar:1.15.0.5-mapr] at org.apache.drill.exec.server.rest.QueryResources.submitQuery(QueryResources.java:87) ~[drill-java-exec-1.15.0.5-mapr.jar:1.15.0.5-mapr] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) [jersey-server-2.8.jar:na] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.8.jar:na] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.8.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.8.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.8.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.8.jar:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) [jersey-common-2.8.jar:na] at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:252) [jersey-server-2.8.jar:na] at
[jira] [Created] (DRILL-7163) Join query fails with java.lang.IllegalArgumentException: null
Khurram Faraaz created DRILL-7163: - Summary: Join query fails with java.lang.IllegalArgumentException: null Key: DRILL-7163 URL: https://issues.apache.org/jira/browse/DRILL-7163 Project: Apache Drill Issue Type: Bug Affects Versions: 1.15.0 Reporter: Khurram Faraaz Join query fails with java.lang.IllegalArgumentException: null Drill : 1.15.0 Failing query is {noformat} Select * From ( select convert_from(t.itm.iUUID, 'UTF8') iUUID, convert_from(t.UPC.UPC14, 'UTF8') UPC14, convert_from(t.itm.upcDesc, 'UTF8') upcDesc, convert_from(t.ris.mstBrdOid, 'UTF8') mstBrdOid, convert_from(t.ris.vrfLgyMtch, 'UTF8') vrfLgyMtch, convert_from(t.itm.mtch.cfdMtch, 'UTF8') cfdMtch, convert_from(t.itm.uoM, 'UTF8') uoM, convert_from(t.uomRec.valVal, 'UTF8') uomVal, case when a.iUUID is null then 0 else 1 end as keyind from hbase.`/mapr/tables/item-master` t left outer join ( select distinct convert_from(t.m.iUUID, 'UTF8') iUUID from hbase.`/mapr/tables/items` t ) a on t.itm.iUUID = a.iUUID ) i where (i.mstBrdOid is null or i.vrfLgyMtch is null) and i.keyind=1 {noformat} Stack trace from drillbit.log {noformat} 2019-03-27 11:45:44,563 [23646564-3d23-f32b-6f68-11d7c4dd7a19:frag:1:0] ERROR o.a.d.e.physical.impl.BaseRootExec - Batch dump started: dumping last 2 failed batches 2019-03-27 11:45:44,564 [23646564-3d23-f32b-6f68-11d7c4dd7a19:frag:1:0] ERROR o.a.d.e.p.i.p.ProjectRecordBatch - ProjectRecordBatch[projector=Projector[vector2=null, selectionVectorMode=NONE], hasRemainder=false, remainderIndex=0, recordCount=0, container=org.apache.drill.exec.record.VectorContainer@2133fd0e[recordCount = 0, schemaChanged = false, schema = BatchSchema [fields=[[`row_key` (VARBINARY:REQUIRED)], [`clnDesc` (MAP:REQUIRED), children=([`bndlCnt` (VARBINARY:OPTIONAL)], [`by` (VARBINARY:OPTIONAL)], [`desc` (VARBINARY:OPTIONAL)], [`dt` (VARBINARY:OPTIONAL)], [`descExt` (VARBINARY:OPTIONAL)])], [`dup` (MAP:REQUIRED), children=([`dupBy` (VARBINARY:OPTIONAL)], [`dupDt` (VARBINARY:OPTIONAL)], [`duplicate` (VARBINARY:OPTIONAL)], [`preferred` (VARBINARY:OPTIONAL)])], [`itm` (MAP:REQUIRED), children=([`iUUID` (VARBINARY:OPTIONAL)], [`cfdLgyMtch` (VARBINARY:OPTIONAL)], [`uoM` (VARBINARY:OPTIONAL)], [`upcCd` (VARBINARY:OPTIONAL)], [`upcDesc` (VARBINARY:OPTIONAL)], [`promo` (VARBINARY:OPTIONAL)])], [`lckSts` (MAP:REQUIRED), children=([`lckBy` (VARBINARY:OPTIONAL)], [`lckDt` (VARBINARY:OPTIONAL)])], [`lgy` (MAP:REQUIRED), children=([`lgyBr` (VARBINARY:OPTIONAL)])], [`obs` (MAP:REQUIRED), children=([`POSFile` (VARBINARY:OPTIONAL)])], [`prmRec` (MAP:REQUIRED)], [`ris` (MAP:REQUIRED), children=([`UPC` (VARBINARY:OPTIONAL)], [`brdDesc` (VARBINARY:OPTIONAL)], [`brdExtDesc` (VARBINARY:OPTIONAL)], [`brdFamDesc` (VARBINARY:OPTIONAL)], [`brdTypeCd` (VARBINARY:OPTIONAL)], [`flvDesc` (VARBINARY:OPTIONAL)], [`mfgDesc` (VARBINARY:OPTIONAL)], [`modBy` (VARBINARY:OPTIONAL)], [`modDt` (VARBINARY:OPTIONAL)], [`msaCatCd` (VARBINARY:OPTIONAL)])], [`rjr` (MAP:REQUIRED)], [`uomRec` (MAP:REQUIRED), children=([`valBy` (VARBINARY:OPTIONAL)], [`valDt` (VARBINARY:OPTIONAL)], [`valVal` (VARBINARY:OPTIONAL)], [`recBy` (VARBINARY:OPTIONAL)], [`recDt` (VARBINARY:OPTIONAL)], [`recRat` (VARBINARY:OPTIONAL)], [`recVal` (VARBINARY:OPTIONAL)])], [`upc` (MAP:REQUIRED), children=([`UPC14` (VARBINARY:OPTIONAL)], [`allUPCVar` (VARBINARY:OPTIONAL)])], [`$f12` (VARBINARY:OPTIONAL)], [`iUUID` (VARCHAR:OPTIONAL)]], selectionVector=NONE], wrappers = [org.apache.drill.exec.vector.VarBinaryVector@b23a384[field = [`row_key` (VARBINARY:REQUIRED)], ...], org.apache.drill.exec.vector.complex.MapVector@61c779ff, org.apache.drill.exec.vector.complex.MapVector@575c0f96, org.apache.drill.exec.vector.complex.MapVector@69b943fe, org.apache.drill.exec.vector.complex.MapVector@7f90e2ce, org.apache.drill.exec.vector.complex.MapVector@25c27442, org.apache.drill.exec.vector.complex.MapVector@12d5ffd3, org.apache.drill.exec.vector.complex.MapVector@3150f8c4, org.apache.drill.exec.vector.complex.MapVector@49aefab2, org.apache.drill.exec.vector.complex.MapVector@7f78e7a1, org.apache.drill.exec.vector.complex.MapVector@426ea4fa, org.apache.drill.exec.vector.complex.MapVector@74cee2ab, org.apache.drill.exec.vector.NullableVarBinaryVector@4a0bfdea[field = [`$f12` (VARBINARY:OPTIONAL)], ...], org.apache.drill.exec.vector.NullableVarCharVector@72f64ee5[field = [`iUUID` (VARCHAR:OPTIONAL)], ...]], ...]] 2019-03-27 11:45:44,565 [23646564-3d23-f32b-6f68-11d7c4dd7a19:frag:1:0] ERROR o.a.d.e.p.impl.join.HashJoinBatch - HashJoinBatch[container=org.apache.drill.exec.record.VectorContainer@45887d35[recordCount = 0, schemaChanged = false, schema = BatchSchema [fields=[[`row_key` (VARBINARY:REQUIRED)], [`clnDesc` (MAP:REQUIRED), children=([`bndlCnt` (VARBINARY:OPTIONAL)], [`by` (VARBINARY:OPTIONAL)], [`desc` (VARBINARY:OPTIONAL)],
[jira] [Created] (DRILL-7144) sqlline option : !set useLineContinuation false, fails with ParseException
Khurram Faraaz created DRILL-7144: - Summary: sqlline option : !set useLineContinuation false, fails with ParseException Key: DRILL-7144 URL: https://issues.apache.org/jira/browse/DRILL-7144 Project: Apache Drill Issue Type: Bug Affects Versions: 1.15.0, 1.13.0 Reporter: Khurram Faraaz Assignee: Arina Ielchiieva sqlline option does not work as intended. Returns ParseException instead. !set useLineContinuation false On mapr-drill-1.13.0 we hit the below Exception. {noformat} 0: jdbc:drill:drillbit=drill-abcd-dev.dev.schw> !set useLineContinuation false Error setting configuration: useLineContinuation: java.lang.IllegalArgumentException: No method matching "setuseLineContinuation" was found in sqlline.SqlLineOpts. {noformat} It does not work on drill-1.15.0-mapr-r1 git.branch=drill-1.15.0-mapr-r1 git.commit.id=ebc9fe49d4477b04701fdd81884d5a0b748a13ae {noformat} [test@test-ab bin]# ./sqlline -u "jdbc:drill:schema=dfs.tmp;auth=MAPRSASL;drillbit=test-ab.qa.lab" -n mapr -p mapr Apache Drill 1.15.0.3-mapr "Start your SQL engine." 0: jdbc:drill:schema=dfs.tmp> !set useLineContinuation false 0: jdbc:drill:schema=dfs.tmp> select * from sys.version > select * from sys.memory Error: PARSE ERROR: Encountered "select" at line 2, column 1. Was expecting one of: "ORDER" ... "LIMIT" ... "OFFSET" ... "FETCH" ... "NATURAL" ... "JOIN" ... "INNER" ... "LEFT" ... "RIGHT" ... "FULL" ... "CROSS" ... "," ... "OUTER" ... "EXTEND" ... "(" ... "MATCH_RECOGNIZE" ... "AS" ... ... ... ... ... ... "TABLESAMPLE" ... "WHERE" ... "GROUP" ... "HAVING" ... "WINDOW" ... "UNION" ... "INTERSECT" ... "EXCEPT" ... "MINUS" ... "." ... "[" ... SQL Query select * from sys.version select * from sys.memory ^ [Error Id: 067d5402-b965-4660-8981-34491ab5a051 on test-ab.qa.lab:31010] (state=,code=0) {noformat} {noformat} [Error Id: 067d5402-b965-4660-8981-34491ab5a051 ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.15.0.3-mapr.jar:1.15.0.3-mapr] at org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:185) [drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr] at org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:138) [drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr] at org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:110) [drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr] at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76) [drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr] at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584) [drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr] at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272) [drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] Caused by: org.apache.calcite.sql.parser.SqlParseException: Encountered "select" at line 2, column 1. Was expecting one of: "ORDER" ... "LIMIT" ... "OFFSET" ... "FETCH" ... ... "[" ... at org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.convertException(DrillParserImpl.java:350) ~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr] at org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.normalizeException(DrillParserImpl.java:131) ~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr] at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:137) ~[calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2] at org.apache.calcite.sql.parser.SqlParser.parseStmt(SqlParser.java:162) ~[calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2] at org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:177) [drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr] ... 8 common frames omitted Caused by: org.apache.drill.exec.planner.sql.parser.impl.ParseException: Encountered "select" at line 2, column 1. Was expecting one of: "ORDER" ... "LIMIT" ... "OFFSET" ... "FETCH" ... "NATURAL" ... ... ... "[" ... at org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.generateParseException(DrillParserImpl.java:24076) ~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr] at org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_consume_token(DrillParserImpl.java:23893) ~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr] at org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.SqlStmtEof(DrillParserImpl.java:899) ~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr] at
[jira] [Created] (DRILL-7128) IllegalStateException: Read batch count [0] should be greater than zero
Khurram Faraaz created DRILL-7128: - Summary: IllegalStateException: Read batch count [0] should be greater than zero Key: DRILL-7128 URL: https://issues.apache.org/jira/browse/DRILL-7128 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Affects Versions: 1.15.0 Reporter: Khurram Faraaz Source table is a Hive table stored as parquet. Issue is seen only when querying datacapturekey column, which is of VARCHAR type. Hive 2.3 MapR Drill : 1.15.0.0-mapr commit id : 951ef156fb1025677a2ca2dcf84e11002bf4b513 {noformat} 0: jdbc:drill:drillbit=test.a.node1> describe bt_br_cc_invalid_leads ; +-++--+ | COLUMN_NAME | DATA_TYPE | IS_NULLABLE | +-++--+ | wrapup | CHARACTER VARYING | YES | | datacapturekey | CHARACTER VARYING | YES | | leadgendate | CHARACTER VARYING | YES | | crla1 | CHARACTER VARYING | YES | | crla2 | CHARACTER VARYING | YES | | invalid_lead | INTEGER | YES | | destination_advertiser_vendor_name | CHARACTER VARYING | YES | | source_program_key | CHARACTER VARYING | YES | | publisher_publisher | CHARACTER VARYING | YES | | areaname | CHARACTER VARYING | YES | | data_abertura_ficha | CHARACTER VARYING | YES | +-++--+ 11 rows selected (1.85 seconds) 0: jdbc:drill:drillbit=test.a.node1> // from the view definition, note that column datacapturekey is of type VARVCHAR with precision 2000 { "name" : "bt_br_cc_invalid_leads", "sql" : "SELECT CAST(`wrapup` AS VARCHAR(2000)) AS `wrapup`, CAST(`datacapturekey` AS VARCHAR(2000)) AS `datacapturekey`, CAST(`leadgendate` AS VARCHAR(2000)) AS `leadgendate`, CAST(`crla1` AS VARCHAR(2000)) AS `crla1`, CAST(`crla2` AS VARCHAR(2000)) AS `crla2`, CAST(`invalid_lead` AS INTEGER) AS `invalid_lead`, CAST(`destination_advertiser_vendor_name` AS VARCHAR(2000)) AS `destination_advertiser_vendor_name`, CAST(`source_program_key` AS VARCHAR(2000)) AS `source_program_key`, CAST(`publisher_publisher` AS VARCHAR(2000)) AS `publisher_publisher`, CAST(`areaname` AS VARCHAR(2000)) AS `areaname`, CAST(`data_abertura_ficha` AS VARCHAR(2000)) AS `data_abertura_ficha`\nFROM `dfs`.`root`.`/user/bigtable/logs/hive/warehouse/bt_br_cc_invalid_leads`", "fields" : [ { "name" : "wrapup", "type" : "VARCHAR", "precision" : 2000, "isNullable" : true }, { "name" : "datacapturekey", "type" : "VARCHAR", "precision" : 2000, "isNullable" : true ... ... // total number of rows in bt_br_cc_invalid_leads 0: jdbc:drill:drillbit=test.a.node1> select count(*) from bt_br_cc_invalid_leads ; +-+ | EXPR$0 | +-+ | 20599 | +-+ 1 row selected (0.173 seconds) {noformat} Stack trace from drillbit.log {noformat} 2019-03-18 12:19:01,610 [237010da-6eda-a913-0424-32f63fbe01be:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query with id 237010da-6eda-a913-0424-32f63fbe01be issued by bigtable: SELECT `bt_br_cc_invalid_leads`.`datacapturekey` AS `datacapturekey` FROM `dfs.drill_views`.`bt_br_cc_invalid_leads` `bt_br_cc_invalid_leads` GROUP BY `bt_br_cc_invalid_leads`.`datacapturekey` 2019-03-18 12:19:02,495 [237010da-6eda-a913-0424-32f63fbe01be:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 237010da-6eda-a913-0424-32f63fbe01be:0:0: State change requested AWAITING_ALLOCATION --> RUNNING 2019-03-18 12:19:02,495 [237010da-6eda-a913-0424-32f63fbe01be:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 237010da-6eda-a913-0424-32f63fbe01be:0:0: State to report: RUNNING 2019-03-18 12:19:02,502 [237010da-6eda-a913-0424-32f63fbe01be:frag:0:0] INFO o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: Error in parquet record reader. Message: Hadoop path: /user/bigtable/logs/hive/warehouse/bt_br_cc_invalid_leads/08_0 Total records read: 0 Row group index: 0 Records in row group: 1551 Parquet Metadata: ParquetMetaData{FileMetaData{schema: message hive_schema { optional binary wrapup (UTF8); optional binary datacapturekey (UTF8); optional binary leadgendate (UTF8); optional binary crla1 (UTF8); optional binary crla2 (UTF8); optional binary invalid_lead (UTF8); optional binary destination_advertiser_vendor_name (UTF8); optional binary source_program_key (UTF8); optional binary publisher_publisher (UTF8); optional binary areaname (UTF8); optional binary data_abertura_ficha (UTF8); } , metadata: {}}, blocks: [BlockMetaData\{1551, 139906 [ColumnMetaData{UNCOMPRESSED [wrapup] optional binary wrapup (UTF8) [PLAIN_DICTIONARY, RLE, BIT_PACKED], 4}, ColumnMetaData\{UNCOMPRESSED [datacapturekey] optional binary datacapturekey (UTF8) [RLE, PLAIN, BIT_PACKED], 656}, ColumnMetaData\{UNCOMPRESSED [leadgendate] optional binary leadgendate (UTF8) [PLAIN_DICTIONARY, RLE, BIT_PACKED], 23978},
[jira] [Created] (DRILL-7100) parquet RecordBatchSizerManager : IllegalArgumentException: the requested size must be non-negative
Khurram Faraaz created DRILL-7100: - Summary: parquet RecordBatchSizerManager : IllegalArgumentException: the requested size must be non-negative Key: DRILL-7100 URL: https://issues.apache.org/jira/browse/DRILL-7100 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Affects Versions: 1.15.0 Reporter: Khurram Faraaz Table has string columns that can range from 1024 bytes to 32MB in length, we should be able to handle such wide string columns in parquet, when querying from Drill. Hive Version 2.3.3 Drill Version 1.15 {noformat} CREATE TABLE temp.cust_bhsf_ce_blob_parquet ( event_id DECIMAL, valid_until_dt_tm string, blob_seq_num DECIMAL, valid_from_dt_tm string, blob_length DECIMAL, compression_cd DECIMAL, blob_contents string, updt_dt_tm string, updt_id DECIMAL, updt_task DECIMAL, updt_cnt DECIMAL, updt_applctx DECIMAL, last_utc_ts string, ccl_load_dt_tm string, ccl_updt_dt_tm string ) STORED AS PARQUET; {noformat} The source table is stored as ORC format. Failing query. {noformat} SELECT event_id, BLOB_CONTENTS FROM hive.temp.cust_bhsf_ce_blob_parquet WHERE event_id = 3443236037 2019-03-07 14:40:17,886 [237e8c79-0e9b-45d6-9134-0da95dba462f:frag:1:269] INFO o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: the requested size must be non-negative (the requested size must be non-negative) org.apache.drill.common.exceptions.UserException: INTERNAL_ERROR ERROR: the requested size must be non-negative {noformat} Snippet from drillbit.log file {noformat} [Error Id: 41a4d597-f54d-42a6-be6d-5dbeb7f642ba ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:293) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:69) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:297) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:284) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_181] at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_181] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669) [hadoop-common-2.7.0-mapr-1808.jar:na] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:284) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.15.0.0-mapr.jar:1.15.0.0-mapr] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] Caused by: java.lang.IllegalArgumentException: the requested size must be non-negative at org.apache.drill.shaded.guava.com.google.common.base.Preconditions.checkArgument(Preconditions.java:135) ~[drill-shaded-guava-23.0.jar:23.0] at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:224)
[jira] [Created] (DRILL-7061) Selecting option to limit results to 1000 on web UI causes parse error
Khurram Faraaz created DRILL-7061: - Summary: Selecting option to limit results to 1000 on web UI causes parse error Key: DRILL-7061 URL: https://issues.apache.org/jira/browse/DRILL-7061 Project: Apache Drill Issue Type: Bug Components: Web Server Affects Versions: 1.16.0 Reporter: Khurram Faraaz Attachments: image-2019-02-27-14-17-24-348.png Selecting option to Limit results to 1,000 causes a parse error on web UI, screen shot is attached. Browser used was Chrome. Drill version => 1.16.0-SNAPSHOT commit = e342ff5 Error reported on web UI when we press Submit button on web UI {noformat} Query Failed: An Error Occurred org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: 'LIMIT start, count' is not allowed under the current SQL conformance level SQL Query -- [autoLimit: 1,000 rows] select * from ( select length(varStr) from dfs.`/root/many_json_files` ) limit 1,000 [Error Id: e252d1cc-54d4-4530-837c-a1726a5be89f on qa102-45.qa.lab:31010]{noformat} !image-2019-02-27-14-17-24-348.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7023) Query fails with IndexOutOfBoundsException after upgrade from drill 1.13.0 to drill 1.14.0
Khurram Faraaz created DRILL-7023: - Summary: Query fails with IndexOutOfBoundsException after upgrade from drill 1.13.0 to drill 1.14.0 Key: DRILL-7023 URL: https://issues.apache.org/jira/browse/DRILL-7023 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.14.0 Reporter: Khurram Faraaz Query fails with IndexOutOfBoundsException after upgrade from drill 1.13.0 to drill 1.14.0 {noformat} 2018-12-06 21:43:00,538 [23f5f79c-3777-eb37-ee46-f73be74381ef:frag:2:1] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IndexOutOfBoundsException Fragment 2:1 [Error Id: 3b653503-b6da-4853-a395-317a169468ce on am1397.test.net:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IndexOutOfBoundsException Fragment 2:1 [Error Id: 3b653503-b6da-4853-a395-317a169468ce on am1397.test.net:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361) [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216) [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327) [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.14.0-mapr.jar:1.14.0-mapr] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_152] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_152] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152] Caused by: java.lang.IndexOutOfBoundsException: null at io.netty.buffer.DrillBuf.getBytes(DrillBuf.java:677) ~[drill-memory-base-1.14.0-mapr.jar:4.0.48.Final] at org.apache.drill.exec.vector.BigIntVector.copyEntry(BigIntVector.java:389) ~[vector-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.test.generated.HashJoinProbeGen480.appendProbe(HashJoinProbeTemplate.java:190) ~[na:na] at org.apache.drill.exec.test.generated.HashJoinProbeGen480.outputOuterRow(HashJoinProbeTemplate.java:223) ~[na:na] at org.apache.drill.exec.test.generated.HashJoinProbeGen480.executeProbePhase(HashJoinProbeTemplate.java:357) ~[na:na] at org.apache.drill.exec.test.generated.HashJoinProbeGen480.probeAndProject(HashJoinProbeTemplate.java:400) ~[na:na] at org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext(HashJoinBatch.java:465) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:69) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:142) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at
[jira] [Created] (DRILL-6994) TIMESTAMP type DOB column in Spark parquet is treated as VARBINARY in Drill
Khurram Faraaz created DRILL-6994: - Summary: TIMESTAMP type DOB column in Spark parquet is treated as VARBINARY in Drill Key: DRILL-6994 URL: https://issues.apache.org/jira/browse/DRILL-6994 Project: Apache Drill Issue Type: Bug Affects Versions: 1.14.0 Reporter: Khurram Faraaz A timestamp type column in a parquet file created from Spark is treated as VARBINARY by Drill 1.14.0., Trying to cast DOB column to DATE results in an Exception, although the monthOfYear field is in the allowed range. Data used in the test {noformat} [test@md123 spark_data]# cat inferSchema_example.csv Name,Department,years_of_experience,DOB Sam,Software,5,1990-10-10 Alex,Data Analytics,3,1992-10-10 {noformat} Create the parquet file using the above CSV file {noformat} [test@md123 bin]# ./spark-shell 19/01/22 21:21:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://md123.qa.lab:4040 Spark context available as 'sc' (master = local[*], app id = local-1548192099796). Spark session available as 'spark'. Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.3.1-mapr-SNAPSHOT /_/ Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191) Type in expressions to have them evaluated. Type :help for more information. scala> import org.apache.spark.sql.\{DataFrame, SQLContext} import org.apache.spark.sql.\{DataFrame, SQLContext} scala> import org.apache.spark.\{SparkConf, SparkContext} import org.apache.spark.\{SparkConf, SparkContext} scala> val sqlContext: SQLContext = new SQLContext(sc) warning: there was one deprecation warning; re-run with -deprecation for details sqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@2e0163cb scala> val df = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").option("inferSchema", "true").load("/apps/inferSchema_example.csv") df: org.apache.spark.sql.DataFrame = [Name: string, Department: string ... 2 more fields] scala> df.printSchema test |-- Name: string (nullable = true) |-- Department: string (nullable = true) |-- years_of_experience: integer (nullable = true) |-- DOB: timestamp (nullable = true) scala> df.write.parquet("/apps/infer_schema_example.parquet") // Read the parquet file scala> val data = sqlContext.read.parquet("/apps/infer_schema_example.parquet") data: org.apache.spark.sql.DataFrame = [Name: string, Department: string ... 2 more fields] // Print the schema of the parquet file from Spark scala> data.printSchema test |-- Name: string (nullable = true) |-- Department: string (nullable = true) |-- years_of_experience: integer (nullable = true) |-- DOB: timestamp (nullable = true) // Display the contents of parquet file on spark-shell // register temp table and do a show on all records,to display. scala> data.registerTempTable("employee") warning: there was one deprecation warning; re-run with -deprecation for details scala> val allrecords = sqlContext.sql("SELeCT * FROM employee") allrecords: org.apache.spark.sql.DataFrame = [Name: string, Department: string ... 2 more fields] scala> allrecords.show() ++--+---+---+ |Name| Department|years_of_experience| DOB| ++--+---+---+ | Sam| Software| 5|1990-10-10 00:00:00| |Alex|Data Analytics| 3|1992-10-10 00:00:00| ++--+---+---+ {noformat} Querying the parquet file from Drill 1.14.0-mapr, results in the DOB column (timestamp type in Spark) being treated as VARBINARY. {noformat} apache drill 1.14.0-mapr "a little sql for your nosql" 0: jdbc:drill:schema=dfs.tmp> select * from dfs.`/apps/infer_schema_example.parquet`; +---+-+--+--+ | Name | Department | years_of_experience | DOB | +---+-+--+--+ | Sam | Software | 5 | [B@2bef51f2 | | Alex | Data Analytics | 3 | [B@650eab8 | +---+-+--+--+ 2 rows selected (0.229 seconds) // typeof(DOB) column returns a VARBINARY type, whereas the parquet schema in Spark for DOB: timestamp (nullable = true) 0: jdbc:drill:schema=dfs.tmp> select typeof(DOB) from dfs.`/apps/infer_schema_example.parquet`; ++ | EXPR$0 | ++ | VARBINARY | | VARBINARY | ++ 2 rows selected (0.199 seconds) {noformat} // CAST to DATE type results in Exception, though the monthOfYear is in the range [1,12] {noformat} 0: jdbc:drill:schema=dfs.tmp> select cast(DOB as DATE) from dfs.`/apps/infer_schema_example.parquet`; Error: SYSTEM ERROR: IllegalFieldValueException: Value 0 for monthOfYear must be in the range [1,12] Fragment
[jira] [Created] (DRILL-6990) IllegalStateException: The current reader doesn't support getting next information
Khurram Faraaz created DRILL-6990: - Summary: IllegalStateException: The current reader doesn't support getting next information Key: DRILL-6990 URL: https://issues.apache.org/jira/browse/DRILL-6990 Project: Apache Drill Issue Type: Bug Affects Versions: 1.14.0 Reporter: Khurram Faraaz Attachments: parqt_nestedArray.parquet.tar Reading a parquet file created from Spark, returns IllegalStateException: The current reader doesn't support getting next information Drill 1.14.0, parquet file created from Spark is attached here. //Steps to create parquet file from Spark 2.3.1 [root@ba102-495 ~]# cd /opt/mapr/spark/spark-2.3.1 [root@ba102-495 spark-2.3.1]# cd bin [root@ba102-495 bin]# ./spark-shell 19/01/21 22:57:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://qa102-45.qa.lab:4040 Spark context available as 'sc' (master = local[*], app id = local-1548111430809). Spark session available as 'spark'. Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.3.1-mapr-SNAPSHOT /_/ Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191) Type in expressions to have them evaluated. Type :help for more information. scala> import spark.implicits._ import spark.implicits._ scala> val df = spark.read.json("/apps/nestedDataJson.json") df: org.apache.spark.sql.DataFrame = [id: bigint, nested_array: array>] scala> df.write.parquet("/apps/parqt_nestedArray.parquet") Data used in test {noformat} [root@ba102-495 ~]# cat nestedDataJson.json {"id":19,"nested_array":[[1,2,3,4],[5,6,7,8],[9,10,12]]} {"id":14121,"nested_array":[[1,3,4],[5,6,8],[9,11,12]]} {"id":18894,"nested_array":[[1,3,4],[5,6,7,8],[9,10,11,12]]} {"id":12499,"nested_array":[[1,4],[5,7,8],[9,11,12]]} {"id":120,"nested_array":[[1,4],[5,7,8],[9,10,11,12]]} {"id":12,"nested_array":[[1,2,3,4],[5,6,7,8],[11,12]]} {"id":13,"nested_array":[[1,2,3,4],[5,8],[9,10,11,12]]} {"id":14,"nested_array":[[1,2,3,4],[5,68],[9,10,11,12]]} {"id":123,"nested_array":[[1,2,3,4],[5,8],[9,10,11,12]]} {"id":124,"nested_array":[[1,2,4],[5,6,7,8],[9,10,11,12]]} {"id":134,"nested_array":[[1,4],[5,8],[9,12]]} {noformat} >From drillbit.log {noformat} Query Failed: An Error Occurred org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalStateException: The current reader doesn't support getting next information. Fragment 0:0 [Error Id: c16c70dd-6565-463f-83a7-118ccd8442e2 on ba102-495.qa.lab:31010] ... ... 2019-01-21 23:08:11,268 [23b9af24-10b9-ad11-5583-ecc3e0c562e6:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: The current reader doesn't support getting next information. Fragment 0:0 [Error Id: c16c70dd-6565-463f-83a7-118ccd8442e2 on ba102-495.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: The current reader doesn't support getting next information. Fragment 0:0 [Error Id: c16c70dd-6565-463f-83a7-118ccd8442e2 on ba102-495.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361) [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216) [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327) [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.14.0-mapr.jar:1.14.0-mapr] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] Caused by: java.lang.IllegalStateException: The current reader doesn't support getting next information. at org.apache.drill.exec.vector.complex.impl.AbstractBaseReader.next(AbstractBaseReader.java:64) ~[vector-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.vector.complex.impl.SingleMapReaderImpl.next(SingleMapReaderImpl.java:31) ~[vector-1.14.0-mapr.jar:1.14.0-mapr] at org.apache.drill.exec.test.generated.ProjectorGen971.doEval(ProjectorTemplate.java:35) ~[na:na] at org.apache.drill.exec.test.generated.ProjectorGen971.projectRecords(ProjectorTemplate.java:67) ~[na:na] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:231) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr] at
[jira] [Created] (DRILL-6979) Add autofocus attribute to username on login page, and to query textbox on Query tab.
Khurram Faraaz created DRILL-6979: - Summary: Add autofocus attribute to username on login page, and to query textbox on Query tab. Key: DRILL-6979 URL: https://issues.apache.org/jira/browse/DRILL-6979 Project: Apache Drill Issue Type: Improvement Components: Web Server Affects Versions: 1.16.0 Reporter: Khurram Faraaz Assignee: Khurram Faraaz Add autofocus attribute to username on login page, and to query textbox on Query tab. The two text boxes that need the change are in these files ./exec/java-exec/src/main/resources/rest/query/query.ftl ./exec/java-exec/src/main/resources/rest/login.ftl -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6961) Error Occurred: Cannot connect to the db. query INFORMATION_SCHEMA.VIEWS : Maybe you have incorrect connection params or db unavailable now (timeout)
Khurram Faraaz created DRILL-6961: - Summary: Error Occurred: Cannot connect to the db. query INFORMATION_SCHEMA.VIEWS : Maybe you have incorrect connection params or db unavailable now (timeout) Key: DRILL-6961 URL: https://issues.apache.org/jira/browse/DRILL-6961 Project: Apache Drill Issue Type: Improvement Components: Storage - Information Schema Affects Versions: 1.13.0 Reporter: Khurram Faraaz Trying to query drill information_schema.views table returns error. Disabling openTSDB plugin resolves the problem. Drill 1.13.0 Failing query : {noformat} SELECT TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, VIEW_DEFINITION FROM INFORMATION_SCHEMA.`VIEWS` where VIEW_DEFINITION not like 'kraken'; {noformat} Stack Trace from drillbit.log {noformat} 2019-01-07 15:36:21,975 [23cc39aa-2618-e9f0-e77e-4fafa6edc314:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 23cc39aa-2618-e9f0-e77e-4fafa6edc314: SELECT TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, VIEW_DEFINITION FROM INFORMATION_SCHEMA.`VIEWS` where VIEW_DEFINITION not like 'kraken' 2019-01-07 15:36:35,221 [23cc39aa-2618-e9f0-e77e-4fafa6edc314:frag:0:0] INFO o.a.d.e.s.o.c.services.ServiceImpl - User Error Occurred: Cannot connect to the db. Maybe you have incorrect connection params or db unavailable now (timeout) org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Cannot connect to the db. Maybe you have incorrect connection params or db unavailable now [Error Id: f8b4c074-ba62-4691-b142-a8ea6e4f6b2a ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl.getTableNames(ServiceImpl.java:107) [drill-opentsdb-storage-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl.getAllMetricNames(ServiceImpl.java:70) [drill-opentsdb-storage-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory$OpenTSDBSchema.getTableNames(OpenTSDBSchemaFactory.java:78) [drill-opentsdb-storage-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.calcite.jdbc.SimpleCalciteSchema.addImplicitTableToBuilder(SimpleCalciteSchema.java:106) [calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0] at org.apache.calcite.jdbc.CalciteSchema.getTableNames(CalciteSchema.java:318) [calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0] at org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getTableNames(CalciteSchema.java:587) [calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0] at org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getTableNames(CalciteSchema.java:548) [calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0] at org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.visitTables(InfoSchemaRecordGenerator.java:227) [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:216) [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:209) [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:196) [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.store.ischema.InfoSchemaTableType.getRecordReader(InfoSchemaTableType.java:58) [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:34) [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:30) [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.physical.impl.ImplCreator$2.run(ImplCreator.java:146) [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.physical.impl.ImplCreator$2.run(ImplCreator.java:142) [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr] at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_144] at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_144] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1633) [hadoop-common-2.7.0-mapr-1710.jar:na] at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:142) [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182) [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:137) [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182)
[jira] [Created] (DRILL-6942) Provide ability to sort list of profiles on Drill web UI
Khurram Faraaz created DRILL-6942: - Summary: Provide ability to sort list of profiles on Drill web UI Key: DRILL-6942 URL: https://issues.apache.org/jira/browse/DRILL-6942 Project: Apache Drill Issue Type: Improvement Components: Web Server Affects Versions: 1.15.0 Reporter: Khurram Faraaz We need to provide a way to sort the query profiles, on the profiles tab on Drill web UI. The use case is when users run many queries (several hundred queries), they want a way to list the queries that have taken the longest time (i.e. duration) to complete. All queries that completed, failed or were canceled and took very long, all such queries should be listed on the top in the profiles tab page on Drill web ui. An option to list the query profiles based on total duration should be provided to user. That way user can easily identify such long running queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6937) sys.functions table needs a fix in the names column
Khurram Faraaz created DRILL-6937: - Summary: sys.functions table needs a fix in the names column Key: DRILL-6937 URL: https://issues.apache.org/jira/browse/DRILL-6937 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.15.0 Reporter: Khurram Faraaz The function names in the name column of sys.functions in some cases, are the operators, this is not the expected behavior, the name column should have actual names and not the operators. I am on Drill 1.15.0 commit : 8743e8f1e8d5bca4d67c94d07a8560ad356ff2b6 {noformat} Apache Drill 1.15.0 "Data is the new oil. Ready to Drill some?" 0: jdbc:drill:schema=dfs.tmp> select count(*) from sys.functions; +-+ | EXPR$0 | +-+ | 2846 | +-+ 1 row selected (0.327 seconds) 0: jdbc:drill:schema=dfs.tmp> {noformat} {noformat} 0: jdbc:drill:schema=dfs.tmp> select distinct name from sys.functions limit 12; ++ | name | ++ | != | | $sum0 | | && | | - | | /int | | < | | <= | | <> | | = | | == | | > | | >= | ++ 12 rows selected (0.175 seconds) {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6816) NPE - Concurrent query execution using PreparedStatement
Khurram Faraaz created DRILL-6816: - Summary: NPE - Concurrent query execution using PreparedStatement Key: DRILL-6816 URL: https://issues.apache.org/jira/browse/DRILL-6816 Project: Apache Drill Issue Type: Bug Affects Versions: 1.14.0 Reporter: Khurram Faraaz Concurrent query execution from JDBC program using PreparedStatement results in NPE. Queries that were executed concurrently are (part of a query file), {noformat} select id from `test_tbl.json` select count(id) from `test_tbl.json` select count(*) from `test_tbl.json` select * from `test_tbl.json` {noformat} Drill 1.14.0 git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454 MapR version => 6.1.0.20180911143226.GA (secure cluster) Executing the above queries concurrently using a Statement object results in successful query execution. {noformat} Statement stmt = conn.createStatement(); ResultSet rs = stmt.executeQuery(query); {noformat} However, when the same queries listed above are executed using a PreparedStatement object we see an NPE {noformat} PreparedStatement prdstmnt = conn.prepareStatement(query); prdstmnt.executeUpdate(); {noformat} Stack trace from drillbit.log {noformat} java.lang.NullPointerException at java.util.Objects.requireNonNull(Objects.java:203) at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577) at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143) at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150) at org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511) at org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146) at org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512) at org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142) at RunQuery.executeQuery(RunQuery.java:61) at RunQuery.run(RunQuery.java:30) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-6563) TPCDS query 10 has regressed
[ https://issues.apache.org/jira/browse/DRILL-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz resolved DRILL-6563. --- Resolution: Fixed > TPCDS query 10 has regressed > - > > Key: DRILL-6563 > URL: https://issues.apache.org/jira/browse/DRILL-6563 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning Optimization >Affects Versions: 1.14.0 >Reporter: Khurram Faraaz >Assignee: Pritesh Maker >Priority: Major > Fix For: 1.15.0 > > Attachments: 24ca3c6c-90e1-a4bf-6c6f-3f981fa2d043.sys.drill, > query10.fast_plan_old_commit, tpcds_query_10_plan_slow_140d09e.pdf, > tpcds_query_plan_10_140d09e.txt > > > TPC-DS query 10 has regressed in performance from taking 3.5 seconds to > execute on Apache Drill 1.14.0 commit b92f599 , to 07 min 51.851 sec to > complete execution on Apache Drill 1.14.0 commit 140d09e. Query was executed > over SF1 parquet views on a 4 node cluster. > Query plan from old and newer commit is attached here, with the query profile > from newer commit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6602) Drill query cancellation should log the originator
Khurram Faraaz created DRILL-6602: - Summary: Drill query cancellation should log the originator Key: DRILL-6602 URL: https://issues.apache.org/jira/browse/DRILL-6602 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Reporter: Khurram Faraaz The Drill query Cancel functionality does not log the origin of the Cancel request. (1). security issue, we don't know who (which user) canceled the query. (2). debugging such a canceled query can be a pain to find the origin and the root cause of why a query was canceled. On Apache Drill 1.14.0 git.commit.id.abbrev=b0314a3, we have observed this problem, and it can be consistently reproduced. There is no information at all about the origin of why the query was canceled in the drillbit.log -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6595) IllegalAccessError: tried to access field org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files from class org.apache.drill.exec.store.pa
Khurram Faraaz created DRILL-6595: - Summary: IllegalAccessError: tried to access field org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files from class org.apache.drill.exec.store.parquet.metadata.Metadata_V3 Key: DRILL-6595 URL: https://issues.apache.org/jira/browse/DRILL-6595 Project: Apache Drill Issue Type: Bug Affects Versions: 1.14.0 Reporter: Khurram Faraaz java.lang.IllegalAccessError reported in drillbit.out file Apache Drill 1.14.0 git.commit.id.abbrev=b0314a3 {noformat} Jul 11, 2018 1:33:39 PM com.fasterxml.jackson.module.afterburner.deser.OptimizedSettableBeanProperty _reportProblem WARNING: Disabling Afterburner deserialization for class org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3 (field #1; mutator com.fasterxml.jackson.module.afterburner.deser.SettableObjectFieldProperty), due to access error (type java.lang.IllegalAccessError, message=tried to access field org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files from class org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer48257508) java.lang.IllegalAccessError: tried to access field org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files from class org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer48257508 at org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer48257508.objectField(org/apache/drill/exec/store/parquet/metadata/Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer.java) at com.fasterxml.jackson.module.afterburner.deser.SettableObjectFieldProperty.deserializeAndSet(SettableObjectFieldProperty.java:50) at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288) at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:189) at com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:120) at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:130) at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97) at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:254) at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:68) at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3058) at org.apache.drill.exec.store.parquet.metadata.Metadata.readBlockMeta(Metadata.java:617) at org.apache.drill.exec.store.parquet.metadata.Metadata.readBlockMeta(Metadata.java:156) at org.apache.drill.exec.store.parquet.ParquetGroupScan.expandSelectionFromMetadataCache(ParquetGroupScan.java:374) at org.apache.drill.exec.store.parquet.ParquetGroupScan.expandIfNecessary(ParquetGroupScan.java:337) at org.apache.drill.exec.store.parquet.ParquetGroupScan.(ParquetGroupScan.java:121) at org.apache.drill.exec.store.parquet.ParquetGroupScan.(ParquetGroupScan.java:102) at org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:180) at org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:70) at org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan(FileSystemPlugin.java:136) at org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:114) at org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:109) at org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:99) at org.apache.drill.exec.planner.logical.DrillPushProjectIntoScanRule.canPushProjectIntoScan(DrillPushProjectIntoScanRule.java:125) at org.apache.drill.exec.planner.logical.DrillPushProjectIntoScanRule.onMatch(DrillPushProjectIntoScanRule.java:81) at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212) at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:652) at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:368) at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:426) at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:366) at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel(DefaultSqlHandler.java:255) at
[jira] [Created] (DRILL-6590) DATA_WRITE ERROR: Hash Join failed to write to output file: /tmp/drill/spill/24bac407
Khurram Faraaz created DRILL-6590: - Summary: DATA_WRITE ERROR: Hash Join failed to write to output file: /tmp/drill/spill/24bac407 Key: DRILL-6590 URL: https://issues.apache.org/jira/browse/DRILL-6590 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.14.0 Reporter: Khurram Faraaz Apache Drill 1.14.0 git.commit.id.abbrev=eb946b0 There was enough space on /tmp, however Hash Join failed to write to spill file [test@qa102-45 drill-1.14.0]# clush -a df -h /tmp : Filesystem Size Used Avail Use% Mounted on : /dev/mapper/vg_root-lv_root 500G 150G 351G 30% / : Filesystem Size Used Avail Use% Mounted on : /dev/mapper/vg_root-lv_root 500G 17G 484G 4% / : Filesystem Size Used Avail Use% Mounted on : /dev/mapper/vg_root-lv_root 500G 14G 487G 3% / : Filesystem Size Used Avail Use% Mounted on : /dev/mapper/vg_root-lv_root 500G 13G 488G 3% / Stack trace from drillbit.log {noformat} 2018-07-10 18:17:51,953 [BitServer-10] WARN o.a.d.exec.rpc.control.WorkEventBus - A fragment message arrived but there was no registered listener for that message: profile { state: FAILED error { error_id: "6e258de2-2d4f-4b48-967d-df1b329955cd" endpoint { address: "qa102-48.qa.lab" user_port: 31010 control_port: 31011 data_port: 31012 version: "1.14.0-SNAPSHOT" state: STARTUP } error_type: DATA_WRITE message: "DATA_WRITE ERROR: Hash Join failed to write to output file: /tmp/drill/spill/24bac407-2adb-5763-ed08-cb5714dca2c0_HashJoin_4-22-53/spill15_outer\n\nFragment 4:53\n\n[Error Id: 6e258de2-2d4f-4b48-967d-df1b329955cd on qa102-48.qa.lab:31010]" exception { exception_class: "java.nio.channels.ClosedByInterruptException" stack_trace { class_name: "..." line_number: 0 method_name: "..." is_native_method: false } stack_trace { class_name: "com.google.protobuf.CodedOutputStream" file_name: "CodedOutputStream.java" line_number: 833 method_name: "refreshBuffer" is_native_method: false } stack_trace { class_name: "com.google.protobuf.CodedOutputStream" file_name: "CodedOutputStream.java" line_number: 843 method_name: "flush" is_native_method: false } stack_trace { class_name: "com.google.protobuf.AbstractMessageLite" file_name: "AbstractMessageLite.java" line_number: 91 method_name: "writeDelimitedTo" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.cache.VectorSerializer$Writer" file_name: "VectorSerializer.java" line_number: 97 method_name: "write" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.common.HashPartition" file_name: "HashPartition.java" line_number: 346 method_name: "spillThisPartition" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.common.HashPartition" file_name: "HashPartition.java" line_number: 263 method_name: "completeABatch" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.common.HashPartition" file_name: "HashPartition.java" line_number: 237 method_name: "completeAnOuterBatch" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.common.HashPartition" file_name: "HashPartition.java" line_number: 232 method_name: "appendOuterRow" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.test.generated.HashJoinProbeGen49" file_name: "HashJoinProbeTemplate.java" line_number: 306 method_name: "executeProbePhase" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.test.generated.HashJoinProbeGen49" file_name: "HashJoinProbeTemplate.java" line_number: 393 method_name: "probeAndProject" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.join.HashJoinBatch" file_name: "HashJoinBatch.java" line_number: 357 method_name: "innerNext" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.record.AbstractRecordBatch" file_name: "AbstractRecordBatch.java" line_number: 172 method_name: "next" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.record.AbstractRecordBatch" file_name: "AbstractRecordBatch.java" line_number: 119 method_name: "next" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.join.HashJoinBatch" file_name: "HashJoinBatch.java" line_number: 274 method_name: "sniffNonEmptyBatch" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.join.HashJoinBatch" file_name: "HashJoinBatch.java" line_number: 236 method_name: "prefetchFirstBatchFromBothSides" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.join.HashJoinBatch" file_name: "HashJoinBatch.java" line_number: 216 method_name: "buildSchema" is_native_method: false
[jira] [Created] (DRILL-6583) UI usability issue
Khurram Faraaz created DRILL-6583: - Summary: UI usability issue Key: DRILL-6583 URL: https://issues.apache.org/jira/browse/DRILL-6583 Project: Apache Drill Issue Type: Bug Components: Web Server Affects Versions: 1.14.0 Reporter: Khurram Faraaz Attachments: UI_usability_issue_AD_1_14_0.png When a query is under execution, on the web UI we see this text which is actually a set of different links that help navigate to different pages on the UI, below that query's profile. Apache Drill 1.14.0 git.commit.id.abbrev=f481a7c They all appear on a single line with no spacing and a typo, the formatting of the text for those links needs to be changed / improved. Attached is a screenshot for the issue, look for "FirstPrevious1NextLast" on the bottom left of the screenshot. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6563) TPCDS query 10 has regressed
Khurram Faraaz created DRILL-6563: - Summary: TPCDS query 10 has regressed Key: DRILL-6563 URL: https://issues.apache.org/jira/browse/DRILL-6563 Project: Apache Drill Issue Type: Bug Components: Query Planning Optimization Affects Versions: 1.14.0 Reporter: Khurram Faraaz Attachments: 24ca3c6c-90e1-a4bf-6c6f-3f981fa2d043.sys.drill, query10.fast_plan_old_commit, tpcds_query_10_plan_slow_140d09e.pdf, tpcds_query_plan_10_140d09e.txt TPC-DS query 10 has regressed in performance from taking 3.5 seconds to execute on Apache Drill 1.14.0 commit b92f599 , to 07 min 51.851 sec to complete execution on Apache Drill 1.14.0 commit 140d09e. Query was executed over SF1 parquet views on a 4 node cluster. Query plan from old and newer commit is attached here, with the query profile from newer commit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6548) IllegalStateException: Unexpected EMIT outcome received in buildSchema phase
Khurram Faraaz created DRILL-6548: - Summary: IllegalStateException: Unexpected EMIT outcome received in buildSchema phase Key: DRILL-6548 URL: https://issues.apache.org/jira/browse/DRILL-6548 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.14.0 Reporter: Khurram Faraaz Assignee: Sorabh Hamirwasia On a four node Apache Drill 1.14.0 master branch against TPC-DS SF1 parquet data (parquet views) git.commit.id.abbrev=b92f599 TPC-DS query 69 fails with IllegalStateException: Unexpected EMIT outcome received in buildSchema phase Failing query is, {noformat} 2018-06-27 15:24:39,493 [24cbf157-e95c-42ab-7307-f75f5943a277:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 24cbf157-e95c-42ab-7307-f75f5943a277: SELECT cd_gender, cd_marital_status, cd_education_status, Count(*) cnt1, cd_purchase_estimate, Count(*) cnt2, cd_credit_rating, FROM customer c, customer_address ca, customer_demographics WHERE c.c_current_addr_sk = ca.ca_address_sk AND ca_state IN ( 'KS', 'AZ', 'NE' ) AND cd_demo_sk = c.c_current_cdemo_sk AND EXISTS (SELECT * FROM store_sales, date_dim WHERE c.c_customer_sk = ss_customer_sk AND ss_sold_date_sk = d_date_sk AND d_year = 2004 AND d_moy BETWEEN 3 AND 3 + 2) AND ( NOT EXISTS (SELECT * FROM web_sales, date_dim WHERE c.c_customer_sk = ws_bill_customer_sk AND ws_sold_date_sk = d_date_sk AND d_year = 2004 AND d_moy BETWEEN 3 AND 3 + 2) AND NOT EXISTS (SELECT * FROM catalog_sales, date_dim WHERE c.c_customer_sk = cs_ship_customer_sk AND cs_sold_date_sk = d_date_sk AND d_year = 2004 AND d_moy BETWEEN 3 AND 3 + 2) ) GROUP BY cd_gender, cd_marital_status, cd_education_status, cd_purchase_estimate, cd_credit_rating ORDER BY cd_gender, cd_marital_status, cd_education_status, cd_purchase_estimate, cd_credit_rating cd_credit_rating LIMIT 100 {noformat} Stack trace from drillbit.log {noformat} 2018-06-27 15:24:42,130 [24cbf157-e95c-42ab-7307-f75f5943a277:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: Unexpected EMIT outcome received in buildSchema phase Fragment 0:0 [Error Id: ba1a35e0-807e-4bab-b820-8aa6aad80e87 on qa102-45.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Unexpected EMIT outcome received in buildSchema phase Fragment 0:0 [Error Id: ba1a35e0-807e-4bab-b820-8aa6aad80e87 on qa102-45.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] Caused by: java.lang.IllegalStateException: Unexpected EMIT outcome received in buildSchema phase at org.apache.drill.exec.physical.impl.TopN.TopNBatch.buildSchema(TopNBatch.java:178) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:152) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at
[jira] [Created] (DRILL-6518) DESCRIBE command on Drill created parquet table does not return results
Khurram Faraaz created DRILL-6518: - Summary: DESCRIBE command on Drill created parquet table does not return results Key: DRILL-6518 URL: https://issues.apache.org/jira/browse/DRILL-6518 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Affects Versions: 1.14.0 Reporter: Khurram Faraaz Attachments: 0_0_0.parquet, item.drill.parquet_metadata Describe command on a Drill (1.14.0) created parquet table, does not return the table description. Parquet file and parquet metadata cache file for item table is attached here, it has the details of column types in the metadata cache file. {noformat} Apache Drill 1.14.0-SNAPSHOT commit : b447260e49dc4a8c906f5b310c037fe6dd77166f {noformat} {noformat} // DESCRIBE commands returns no information about the table. 0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> describe dfs.`/drill/testdata/tpcds_sf1/parquet/item`; +--++--+ | COLUMN_NAME | DATA_TYPE | IS_NULLABLE | +--++--+ +--++--+ No rows selected (0.221 seconds) 0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> refresh table metadata dfs.`/drill/testdata/tpcds_sf1/parquet/item`; +---+--+ | ok | summary | +---+--+ | true | Successfully updated metadata for table /drill/testdata/tpcds_sf1/parquet/item. | +---+--+ 1 row selected (0.173 seconds) 0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> describe dfs.`/drill/testdata/tpcds_sf1/parquet/item`; +--++--+ | COLUMN_NAME | DATA_TYPE | IS_NULLABLE | +--++--+ +--++--+ No rows selected (0.229 seconds) {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6517) IllegalStateException: Record count not set for this vector container
Khurram Faraaz created DRILL-6517: - Summary: IllegalStateException: Record count not set for this vector container Key: DRILL-6517 URL: https://issues.apache.org/jira/browse/DRILL-6517 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.14.0 Reporter: Khurram Faraaz Assignee: Padma Penumarthy Attachments: 24d7b377-7589-7928-f34f-57d02061acef.sys.drill TPC-DS query is Canceled after 2 hrs and 47 mins and we see an IllegalStateException: Record count not set for this vector container, in drillbit.log Steps to reproduce the problem, query profile (24d7b377-7589-7928-f34f-57d02061acef) is attached here. {noformat} In drill-env.sh set max direct memory to 12G on all 4 nodes in cluster export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"12G"} and set these options from sqlline, alter system set `planner.memory.max_query_memory_per_node` = 10737418240; alter system set `drill.exec.hashagg.fallback.enabled` = true; To run the query (replace IP-ADDRESS with your foreman node's IP address) cd /opt/mapr/drill/drill-1.14.0/bin ./sqlline -u "jdbc:drill:schema=dfs.tpcds_sf1_parquet_views;drillbit=" -f /root/query72.sql {noformat} Stack trace from drillbit.log {noformat} 2018-06-18 20:08:51,912 [24d7b377-7589-7928-f34f-57d02061acef:frag:4:49] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: Record count not set for this vector container Fragment 4:49 [Error Id: 73177a1c-f7aa-4c9e-99e1-d6e1280e3f27 on qa102-45.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Record count not set for this vector container Fragment 4:49 [Error Id: 73177a1c-f7aa-4c9e-99e1-d6e1280e3f27 on qa102-45.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] Caused by: java.lang.IllegalStateException: Record count not set for this vector container at com.google.common.base.Preconditions.checkState(Preconditions.java:173) ~[guava-18.0.jar:na] at org.apache.drill.exec.record.VectorContainer.getRecordCount(VectorContainer.java:394) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.getRecordCount(RemovingRecordBatch.java:49) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.record.RecordBatchSizer.(RecordBatchSizer.java:690) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.record.RecordBatchSizer.(RecordBatchSizer.java:662) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.record.JoinBatchMemoryManager.update(JoinBatchMemoryManager.java:73) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.record.JoinBatchMemoryManager.update(JoinBatchMemoryManager.java:79) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.join.HashJoinBatch.prefetchFirstBatchFromBothSides(HashJoinBatch.java:242) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.join.HashJoinBatch.buildSchema(HashJoinBatch.java:218) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:152) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.join.HashJoinBatch.sniffNonEmptyBatch(HashJoinBatch.java:276) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.join.HashJoinBatch.prefetchFirstBatchFromBothSides(HashJoinBatch.java:238) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at
[jira] [Created] (DRILL-6453) TPC-DS query 72 has regressed
Khurram Faraaz created DRILL-6453: - Summary: TPC-DS query 72 has regressed Key: DRILL-6453 URL: https://issues.apache.org/jira/browse/DRILL-6453 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.14.0 Reporter: Khurram Faraaz Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill TPC-DS query 72 seems to have regressed, query profile for the case where it Canceled after 2 hours on Drill 1.14.0 is attached here. {noformat} On, Drill 1.14.0-SNAPSHOT commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took around 55 seconds to execute) SF1 parquet data on 4 nodes; planner.memory.max_query_memory_per_node = 10737418240. drill.exec.hashagg.fallback.enabled = true TPC-DS query 72 executed successfully & took 47 seconds to complete execution. {noformat} {noformat} TPC-DS data in the below run has date values stored as DATE datatype and not VARCHAR type On, Drill 1.14.0-SNAPSHOT commit : 82e1a12 SF1 parquet data on 4 nodes; planner.memory.max_query_memory_per_node = 10737418240. drill.exec.hashagg.fallback.enabled = true and alter system set `exec.hashjoin.num_partitions` = 1; TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to Cancel it by stopping the Foreman drillbit. As a result several minor fragments are reported to be in CANCELLATION_REQUESTED state on UI. {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6452) document steps to execute SQL queries from Postman (chrome extension) on Drill
Khurram Faraaz created DRILL-6452: - Summary: document steps to execute SQL queries from Postman (chrome extension) on Drill Key: DRILL-6452 URL: https://issues.apache.org/jira/browse/DRILL-6452 Project: Apache Drill Issue Type: Bug Components: Documentation Affects Versions: 1.14.0 Reporter: Khurram Faraaz We need documentation to list the steps with screen shots about executing SQL queries from Postman (chrome extension) on Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6441) IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.
Khurram Faraaz created DRILL-6441: - Summary: IllegalStateException: Allocator[ROOT] closed with outstanding child allocators. Key: DRILL-6441 URL: https://issues.apache.org/jira/browse/DRILL-6441 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.14.0 Reporter: Khurram Faraaz This is seen when the drillbit is shutdown using $DRILL_HOME/bin/drillbit.sh stop, it happens when BaseAllocator tries to close. Apache Drill 1.14.0 commit c6c5d27d91468a29656bee2acba55d3321978aab {noformat} 2018-05-11 15:19:44,510 [2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:0:0: State to report: FINISHED Wed May 16 10:17:13 PDT 2018 Terminating drillbit pid 32076 2018-05-16 10:17:13,793 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Received shutdown request. 2018-05-16 10:17:14,876 [Drillbit-ShutdownHook#0] INFO o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 20, cache miss count: 6, hit rate: 70% 2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] ERROR o.a.d.exec.server.BootStrapContext - Error while closing java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child allocators. Allocator(ROOT) 0/0/9577600/34359738368 (res/actual/peak/limit) child allocators: 8 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit) child allocators: 0 ledgers: 0 reservations: 0 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit) child allocators: 0 ledgers: 0 reservations: 0 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit) child allocators: 0 ledgers: 0 reservations: 0 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit) child allocators: 0 ledgers: 0 reservations: 0 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit) child allocators: 0 ledgers: 0 reservations: 0 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit) child allocators: 0 ledgers: 0 reservations: 0 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit) child allocators: 0 ledgers: 0 reservations: 0 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit) child allocators: 0 ledgers: 0 reservations: 0 ledgers: 0 reservations: 0 at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496) ~[drill-memory-base-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:259) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:263) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:363) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] 2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Shutdown completed (1095 ms). {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6427) outputBatchSize is missing from the DEBUG output for HashJoinBatch operator
Khurram Faraaz created DRILL-6427: - Summary: outputBatchSize is missing from the DEBUG output for HashJoinBatch operator Key: DRILL-6427 URL: https://issues.apache.org/jira/browse/DRILL-6427 Project: Apache Drill Issue Type: Bug Reporter: Khurram Faraaz Assignee: Padma Penumarthy Drill 1.14.0-SNAPSHOT commit : f99d1f1323c0a5bac99842d6283d3025f3cb527f outputBatchSize is missing from the DEBUG output for HashJoinBatch operator Query used in test, {noformat} select count(*) from `twovarchar_asc_128MB.parquet` t1, `twovarchar_asc_16MB.parquet` t2 WHERE t1.Varbinaryvalue = t2.Varbinaryvalue {noformat} Snippet from drillbit.log {noformat} 2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG o.a.d.e.p.impl.join.HashJoinBatch - left input: batch count : 1, avg batch bytes : 90145920, avg row bytes : 8166, record count : 11040 2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG o.a.d.e.p.impl.join.HashJoinBatch - right input: batch count : 1, avg batch bytes : 10567808, avg row bytes : 7506, record count : 1408 2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG o.a.d.e.p.impl.join.HashJoinBatch - output: batch count : 166, avg batch bytes : 15951453, avg row bytes : 15672, record count : 168960 {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6330) TPC-DS query 95 failed to plan
Khurram Faraaz created DRILL-6330: - Summary: TPC-DS query 95 failed to plan Key: DRILL-6330 URL: https://issues.apache.org/jira/browse/DRILL-6330 Project: Apache Drill Issue Type: Bug Components: Query Planning Optimization Affects Versions: 1.14.0 Reporter: Khurram Faraaz TPC-DS query 95 planned and executed successfully on Drill 1.10.0 (git.commit id : bbcf4b76), it fails to plan on 1.14.0 master on 4 node cluster, SF1 parquet data. TPC-DS query 95 fails to plan on Drill 1.14.0 git.commit.id.abbrev=da24113 git.branch=master {noformat} [root@qa102-45 bin]# ./sqlline -u "jdbc:drill:schema=dfs.tpcds_sf1_parquet_views;drillbit=" -f ~/tpcds_drill_1_14_0/query95.sql 1/2 WITH ws_wh AS ( SELECT ws1.ws_order_number, ws1.ws_warehouse_sk wh1, ws2.ws_warehouse_sk wh2 FROM web_sales ws1, web_sales ws2 WHERE ws1.ws_order_number = ws2.ws_order_number AND ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk) SELECT Count(DISTINCT ws_order_number) AS `order count` , Sum(ws_ext_ship_cost) AS `total shipping cost` , Sum(ws_net_profit) AS `total net profit` FROM web_sales ws1 , date_dim , customer_address , web_site WHERE d_date BETWEEN '2000-4-01' AND ( Cast('2000-4-01' AS DATE) + INTERVAL '60' day) AND ws1.ws_ship_date_sk = d_date_sk AND ws1.ws_ship_addr_sk = ca_address_sk AND ca_state = 'IN' AND ws1.ws_web_site_sk = web_site_sk AND web_company_name = 'pri' AND ws1.ws_order_number IN ( SELECT ws_order_number FROM ws_wh) AND ws1.ws_order_number IN ( SELECT wr_order_number FROM web_returns, ws_wh WHERE wr_order_number = ws_wh.ws_order_number) ORDER BY count(DISTINCT ws_order_number) LIMIT 100; Error: UNSUPPORTED_OPERATION ERROR: This query cannot be planned possibly due to either a cartesian join or an inequality join [Error Id: d2ec3faf-1253-492e-bc56-80689e62335c on qa102-45.qa.lab:31010] (org.apache.drill.exec.work.foreman.UnsupportedRelOperatorException) This query cannot be planned possibly due to either a cartesian join or an inequality join org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToPrel():450 org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():179 org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():146 org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():84 org.apache.drill.exec.work.foreman.Foreman.runSQL():567 org.apache.drill.exec.work.foreman.Foreman.run():264 java.util.concurrent.ThreadPoolExecutor.runWorker():1149 java.util.concurrent.ThreadPoolExecutor$Worker.run():624 java.lang.Thread.run():748 (state=,code=0) {noformat} >From the profile file under log/profiles directory, for TPC-DS query 95. {noformat} {"id":\{"part1":2679297541583898541,"part2":5048058602276109542},"type":1,"start":1523661076935,"end":1523661077389,"query":"WITH ws_wh AS \n( \nSELECT ws1.ws_order_number, \nws1.ws_warehouse_sk wh1, \nws2.ws_warehouse_sk wh2 \nFROM web_sales ws1, \nweb_sales ws2 \nWHERE ws1.ws_order_number = ws2.ws_order_number \nAND ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk) \nSELECT \nCount(DISTINCT ws_order_number) AS `order count` , \nSum(ws_ext_ship_cost) AS `total shipping cost` , \nSum(ws_net_profit) AS `total net profit` \nFROM web_sales ws1 , \ndate_dim , \ncustomer_address , \nweb_site \nWHERE d_date BETWEEN '2000-4-01' AND ( \nCast('2000-4-01' AS DATE) + INTERVAL '60' day) \nAND ws1.ws_ship_date_sk = d_date_sk \nAND ws1.ws_ship_addr_sk = ca_address_sk \nAND ca_state = 'IN' \nAND ws1.ws_web_site_sk = web_site_sk \nAND web_company_name = 'pri' \nAND ws1.ws_order_number IN \n( \nSELECT ws_order_number \nFROM ws_wh) \nAND ws1.ws_order_number IN \n( \nSELECT wr_order_number \nFROM web_returns, \nws_wh \nWHERE wr_order_number = ws_wh.ws_order_number) \nORDER BY count(DISTINCT ws_order_number) \nLIMIT 100","foreman":\{"address":"qa102-45.qa.lab","userPort":31010,"controlPort":31011,"dataPort":31012,"version":"1.14.0-SNAPSHOT","state":0},"state":4,"totalFragments":0,"finishedFragments":0,"user":"anonymous","error":"UNSUPPORTED_OPERATION ERROR: This query cannot be planned possibly due to either a cartesian join or an inequality join\n\n","verboseError":"UNSUPPORTED_OPERATION ERROR: This query cannot be planned possibly due to either a cartesian join or an inequality join\n\n\n\n (org.apache.drill.exec.work.foreman.UnsupportedRelOperatorException) This query cannot be planned possibly due to either a cartesian join or an inequality join\n org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToPrel():450\n org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():179\n org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():146\n org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():84\n org.apache.drill.exec.work.foreman.Foreman.runSQL():567\n org.apache.drill.exec.work.foreman.Foreman.run():264\n
[jira] [Created] (DRILL-6329) TPC-DS Query 66 failed due to OOM
Khurram Faraaz created DRILL-6329: - Summary: TPC-DS Query 66 failed due to OOM Key: DRILL-6329 URL: https://issues.apache.org/jira/browse/DRILL-6329 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.14.0 Reporter: Khurram Faraaz Attachments: 252f0f20-2774-43d7-ec31-911ee0f5f330.sys.drill, TPCDS_Query_66.sql, TPCDS_Query_66_PLAN.txt TPC-DS Query 66 failed after 27 minutes on Drill 1.14.0 on a 4 node cluster against SF1 parquet data (dfs.tpcds_sf1_parquet_views). Query 66 and the query profile and the query plan are attached here. This seems to be a regression, the same query worked fine on 1.10.0 On Drill 1.10.0 ( git.commit id : bbcf4b76) => 9.026 seconds (completed successfully). On Drill 1.14.0 ( git.commit.id.abbrev=da24113 ) query 66 failed after running for 27 minutes, due to OutOfMemoryException Stack trace from sqlline console, no stack trace was written to drillbit.log {noformat} Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the query. Too little memory available Fragment 2:0 [Error Id: 5636a939-a318-4b59-b3e8-9eb93f6b82f3 on qa102-45.qa.lab:31010] (org.apache.drill.exec.exception.OutOfMemoryException) Too little memory available org.apache.drill.exec.test.generated.HashAggregatorGen7120.delayedSetup():409 org.apache.drill.exec.test.generated.HashAggregatorGen7120.doWork():579 org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():176 org.apache.drill.exec.record.AbstractRecordBatch.next():164 org.apache.drill.exec.record.AbstractRecordBatch.next():119 org.apache.drill.exec.record.AbstractRecordBatch.next():109 org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51 org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():134 org.apache.drill.exec.record.AbstractRecordBatch.next():164 org.apache.drill.exec.physical.impl.BaseRootExec.next():105 org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():93 org.apache.drill.exec.physical.impl.BaseRootExec.next():95 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():292 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():279 java.security.AccessController.doPrivileged():-2 javax.security.auth.Subject.doAs():422 org.apache.hadoop.security.UserGroupInformation.doAs():1595 org.apache.drill.exec.work.fragment.FragmentExecutor.run():279 org.apache.drill.common.SelfCleaningRunnable.run():38 java.util.concurrent.ThreadPoolExecutor.runWorker():1149 java.util.concurrent.ThreadPoolExecutor$Worker.run():624 java.lang.Thread.run():748 (state=,code=0) java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query. Too little memory available Fragment 2:0 [Error Id: 5636a939-a318-4b59-b3e8-9eb93f6b82f3 on qa102-45.qa.lab:31010] (org.apache.drill.exec.exception.OutOfMemoryException) Too little memory available org.apache.drill.exec.test.generated.HashAggregatorGen7120.delayedSetup():409 org.apache.drill.exec.test.generated.HashAggregatorGen7120.doWork():579 org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():176 org.apache.drill.exec.record.AbstractRecordBatch.next():164 org.apache.drill.exec.record.AbstractRecordBatch.next():119 org.apache.drill.exec.record.AbstractRecordBatch.next():109 org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51 org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():134 org.apache.drill.exec.record.AbstractRecordBatch.next():164 org.apache.drill.exec.physical.impl.BaseRootExec.next():105 org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():93 org.apache.drill.exec.physical.impl.BaseRootExec.next():95 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():292 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():279 java.security.AccessController.doPrivileged():-2 javax.security.auth.Subject.doAs():422 org.apache.hadoop.security.UserGroupInformation.doAs():1595 org.apache.drill.exec.work.fragment.FragmentExecutor.run():279 org.apache.drill.common.SelfCleaningRunnable.run():38 java.util.concurrent.ThreadPoolExecutor.runWorker():1149 java.util.concurrent.ThreadPoolExecutor$Worker.run():624 java.lang.Thread.run():748 ... Caused by: org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query. Too little memory available Fragment 2:0 [Error Id: 5636a939-a318-4b59-b3e8-9eb93f6b82f3 on qa102-45.qa.lab:31010] (org.apache.drill.exec.exception.OutOfMemoryException) Too little memory available org.apache.drill.exec.test.generated.HashAggregatorGen7120.delayedSetup():409 org.apache.drill.exec.test.generated.HashAggregatorGen7120.doWork():579
[jira] [Created] (DRILL-6308) Exception handling - OOM
Khurram Faraaz created DRILL-6308: - Summary: Exception handling - OOM Key: DRILL-6308 URL: https://issues.apache.org/jira/browse/DRILL-6308 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.14.0 Reporter: Khurram Faraaz Assignee: Sorabh Hamirwasia For RpcHandlers in Drill, when there is an OutOfMemory condition ,then all the handlers call OutOfMemoryHandler with the caught exception. Every channel, except for DataServer the default instance of MemoryHandler is used which throws UnsupportedOperationException. This is misleading since actual exception is for OutOfMemory not UnsupportedOperationException. Here is an example where it is not handled the right way. {noformat} Caused by: java.lang.UnsupportedOperationException: null at org.apache.drill.exec.rpc.OutOfMemoryHandler$1.handle(OutOfMemoryHandler.java:25) ~[drill-rpc-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.rpc.SaslEncryptionHandler.encode(SaslEncryptionHandler.java:170) ~[drill-rpc-1.13.0-mapr.jar:1.13.0-mapr] at org.apache.drill.exec.rpc.SaslEncryptionHandler.encode(SaslEncryptionHandler.java:44) ~[drill-rpc-1.13.0-mapr.jar:1.13.0-mapr] at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) [netty-codec-4.0.48.Final.jar:4.0.48.Final] ... 25 common frames omitted{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6302) NPE in Drillbit.java in close method
Khurram Faraaz created DRILL-6302: - Summary: NPE in Drillbit.java in close method Key: DRILL-6302 URL: https://issues.apache.org/jira/browse/DRILL-6302 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.13.0 Environment: git.commit.id=bb07ebbb9ba8742f44689f8bd8efb5853c5edea0 Reporter: Khurram Faraaz Assignee: Sorabh Hamirwasia registrationHandle in the close method of Drillbit.java (line 228) is null, which causes an NPE and as a result the server does not close. [https://github.com/mapr/private-drill/blob/drill-1.12.0-mapr/exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java#L228] registrationHandle = coord.update(registrationHandle, State.QUIESCENT); Stack trace from drillbit.log {noformat} /opt/mapr/drill/drill-1.12.0/logs/drillbit.out {noformat} OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 Exception in thread "main" java.lang.NullPointerException at org.apache.drill.exec.coord.zk.ZKClusterCoordinator.update(ZKClusterCoordinator.java:223) at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:228) at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:401) at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:372) at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:368) {noformat}{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6285) Failure while attempting to fail rpc response.
Khurram Faraaz created DRILL-6285: - Summary: Failure while attempting to fail rpc response. Key: DRILL-6285 URL: https://issues.apache.org/jira/browse/DRILL-6285 Project: Apache Drill Issue Type: Bug Components: Execution - RPC Affects Versions: 1.14.0 Reporter: Khurram Faraaz When a query is marked as Canceled on Web UI, there is this Exception in drillbit.log Drill 1.13.0 commit cac2882d5a9e22fbc251e4caf622fe30242ad557 {noformat} 2018-03-21 15:35:43,198 [UserServer-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: State change requested RUNNING --> FAILED 2018-03-21 15:35:43,198 [254d2796-7744-cc2a-d77e-5ec030b04211:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: State change requested FAILED --> FAILED 2018-03-21 15:35:43,199 [254d2796-7744-cc2a-d77e-5ec030b04211:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: State change requested FAILED --> FAILED 2018-03-21 15:35:43,202 [254d2796-7744-cc2a-d77e-5ec030b04211:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: State change requested FAILED --> FAILED 2018-03-21 15:35:43,202 [UserServer-1] WARN o.apache.drill.exec.rpc.RequestIdMap - Failure while attempting to fail rpc response. java.lang.IllegalArgumentException: Self-suppression not permitted at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[na:1.8.0_161] at org.apache.drill.common.DeferredException.addException(DeferredException.java:88) ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97) ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:412) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.access$700(FragmentExecutor.java:55) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor$ExecutorStateImpl.fail(FragmentExecutor.java:426) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.ops.FragmentContextImpl.fail(FragmentContextImpl.java:233) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:100) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:97) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.ops.StatusHandler.failed(StatusHandler.java:42) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:139) ~[drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:76) [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:66) [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692) [hppc-0.7.1.jar:na] at org.apache.drill.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:62) [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.rpc.AbstractRemoteConnection.channelClosed(AbstractRemoteConnection.java:192) [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.rpc.AbstractServerConnection.channelClosed(AbstractServerConnection.java:165) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:167) [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:144) [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at
[jira] [Created] (DRILL-6254) IllegalArgumentException: the requested size must be non-negative
Khurram Faraaz created DRILL-6254: - Summary: IllegalArgumentException: the requested size must be non-negative Key: DRILL-6254 URL: https://issues.apache.org/jira/browse/DRILL-6254 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.13.0 Reporter: Khurram Faraaz Assignee: Padma Penumarthy Attachments: genAllTypesJSN.py Flatten query fails due to IllegalArgumentException: the requested size must be non-negative. Script to generate JSON data file is attached here. {noformat} 0: jdbc:drill:schema=dfs.tmp> CREATE TABLE tbl_all_types_jsn_to_parquet AS . . . . . . . . . . . . . . > SELECT . . . . . . . . . . . . . . > CAST( col_int AS INT) col_int, . . . . . . . . . . . . . . > CAST( col_bigint AS BIGINT) col_bigint, . . . . . . . . . . . . . . > CAST( col_char AS CHAR(10)) col_char, . . . . . . . . . . . . . . > CAST( col_fxdln_str AS VARCHAR(256)) col_fxdln_str, . . . . . . . . . . . . . . > CAST( col_varln_str AS VARCHAR(256)) col_varln_str, . . . . . . . . . . . . . . > CAST( col_float AS FLOAT) col_float, . . . . . . . . . . . . . . > CAST( col_double AS DOUBLE PRECISION) col_double, . . . . . . . . . . . . . . > CAST( col_date AS DATE) col_date, . . . . . . . . . . . . . . > CAST( col_time AS TIME) col_time, . . . . . . . . . . . . . . > CAST( col_tmstmp AS TIMESTAMP) col_tmstmp, . . . . . . . . . . . . . . > CAST( col_boolean AS BOOLEAN) col_boolean, . . . . . . . . . . . . . . > col_binary, . . . . . . . . . . . . . . > array_of_ints from `all_supported_types.json`; +---++ | Fragment | Number of records written | +---++ | 0_0 | 9 | +---++ 1 row selected (0.29 seconds) {noformat} Reset all options and set slice_target=1 alter system reset all; alter system set `planner.slice_target`=1; output_batch_size was set to its default value drill.exec.memory.operator.output_batch_size = 16777216 {noformat} select *, flatten(array_of_ints) from tbl_all_types_jsn_to_parquet; Error: SYSTEM ERROR: IllegalArgumentException: the requested size must be non-negative Fragment 0:0 [Error Id: 480bae96-ae89-45a7-b937-011c0f87c14d on qa102-45.qa.lab:31010] (state=,code=0) 0: jdbc:drill:schema=dfs.tmp> {noformat} Stack trace from drillbit.log {noformat} 2018-03-15 12:19:43,916 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 255538af-bcd5-98ee-32e0-68d98fc4a6fa: select *, flatten(array_of_ints) from tbl_all_types_jsn_to_parquet 2018-03-15 12:19:43,952 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2018-03-15 12:19:43,953 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2018-03-15 12:19:43,966 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO o.a.d.exec.store.parquet.Metadata - Took 0 ms to get file statuses 2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1 using 1 threads. Time: 2ms total, 2.927366ms avg, 2ms max. 2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1 using 1 threads. Earliest start: 2.829000 μs, Latest start: 2.829000 μs, Average start: 2.829000 μs . 2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO o.a.d.exec.store.parquet.Metadata - Took 3 ms to read file metadata 2018-03-15 12:19:44,000 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: State change requested AWAITING_ALLOCATION --> RUNNING 2018-03-15 12:19:44,000 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: State to report: RUNNING 2018-03-15 12:19:44,905 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: State change requested RUNNING --> FAILED 2018-03-15 12:19:44,927 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: State change requested FAILED --> FINISHED 2018-03-15 12:19:44,928 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalArgumentException: the requested size must be non-negative Fragment 0:0 [Error Id: 480bae96-ae89-45a7-b937-011c0f87c14d on qa102-45.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
[jira] [Created] (DRILL-6247) Minor fragments remain in CANCELLATION_REQUESTED state after query failure
Khurram Faraaz created DRILL-6247: - Summary: Minor fragments remain in CANCELLATION_REQUESTED state after query failure Key: DRILL-6247 URL: https://issues.apache.org/jira/browse/DRILL-6247 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.13.0 Reporter: Khurram Faraaz Attachments: cancellation_requested_march_14.png, drillbit_snippet.log, jstack_cancellation_requested.txt Once a query fails, in this case due to an OOM in RPC we see many minor fragments are reported to be in CANCELLATION_REQUESTED state on Web UI after query has failed. The problem is reproducible. drillbit.log for this failure and jstack output are attached here. To reproduce the problem on a 4 node cluster. alter system reset all; alter system set `planner.slice_target`=1; Failing query => SELECT * , FLATTEN(arr) FROM many_json_files; Drill 1.13.0-SNAPSHOT, commit id: 766315ea17377199897d685ab801edd38394fe01 Stack trace from output of jstack, fragment 0:0 is reported to be in CANCELLATION_REQUESTED state on Drill Web UI jstack -l 13488 > jstack_DRILL_6235.txt {noformat} "25569e98-10f9-2fe2-9dec-0a42f3ad45fa:frag:0:0" #87 daemon prio=10 os_prio=0 tid=0x7f9d01374360 nid=0x2ff5 waiting on condition [0x7f9cd5536000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0007a388b300> (a org.apache.drill.exec.rpc.ResettableBarrier$InternalSynchronizer) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at org.apache.drill.exec.rpc.ResettableBarrier.await(ResettableBarrier.java:70) at org.apache.drill.exec.rpc.AbstractRemoteConnection$WriteManager.waitForWritable(AbstractRemoteConnection.java:114) at org.apache.drill.exec.rpc.AbstractRemoteConnection.blockOnNotWritable(AbstractRemoteConnection.java:76) at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:108) at org.apache.drill.exec.rpc.user.UserServer$BitToUserConnection.sendData(UserServer.java:275) at org.apache.drill.exec.ops.AccountingUserConnection.sendData(AccountingUserConnection.java:42) at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:120) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:233) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226) at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} Stack trace from drillbit.log {noformat} 2018-03-14 10:52:44,545 [25569e98-10f9-2fe2-9dec-0a42f3ad45fa:frag:1:49] INFO o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes ran out of memory while executing the query. (Failure allocating buffer.) org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query. Failure allocating buffer. [Error Id: b83884df-af31-411a-9b28-554c294a7357 ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:243) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Failure allocating buffer. at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:67) ~[drill-memory-base-1.13.0-SNAPSHOT.jar:4.0.48.Final] at
[jira] [Created] (DRILL-6235) Flatten query leads to out of memory in RPC layer.
Khurram Faraaz created DRILL-6235: - Summary: Flatten query leads to out of memory in RPC layer. Key: DRILL-6235 URL: https://issues.apache.org/jira/browse/DRILL-6235 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.12.0 Reporter: Khurram Faraaz Assignee: Padma Penumarthy Attachments: 25593391-512d-23ab-7c84-3651006931e2.sys.drill Flatten query leads to out of memory in RPC layer. Query profile is attached here. Total number of JSON files = 4095 Each JSON file has nine rows And each row in the JSON has an array with 1024 integer values, and there are other string values outside of the array. Two major fragments and eighty eight minor fragments were created On a 4 node CentOS cluster number of CPU cores [root@qa102-45 ~]# grep -c ^processor /proc/cpuinfo 32 Details of memory {noformat} 0: jdbc:drill:schema=dfs.tmp> select * from sys.memory; +--++---+-+-+-+-+ | hostname | user_port | heap_current | heap_max | direct_current | jvm_direct_current | direct_max | +--++---+-+-+-+-+ | qa102-45.qa.lab | 31010 | 1130364912 | 4294967296 | 0 | 170528 | 8589934592 | | qa102-47.qa.lab | 31010 | 171823104 | 4294967296 | 0 | 21912 | 8589934592 | | qa102-48.qa.lab | 31010 | 201326576 | 4294967296 | 0 | 21912 | 8589934592 | | qa102-46.qa.lab | 31010 | 214780896 | 4294967296 | 0 | 21912 | 8589934592 | +--++---+-+-+-+-+ 4 rows selected (0.166 seconds) {noformat} Reset all options and set slice_target=1 alter system reset all; alter system set `planner.slice_target`=1; {noformat} SELECT * , FLATTEN(arr) FROM many_json_files ... Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the query. Failure allocating buffer. Fragment 1:38 [Error Id: cf4fd273-d8a2-45e8-8d72-15c738e53b0f on qa102-45.qa.lab:31010] (state=,code=0) {noformat} Stack trace from drillbit.log fir above failing query. {noformat} 2018-03-12 11:52:33,849 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 25593391-512d-23ab-7c84-3651006931e2:0:0: State change requested AWAITING_ALLOCATION --> RUNNING 2018-03-12 11:52:33,849 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 25593391-512d-23ab-7c84-3651006931e2:0:0: State to report: RUNNING 2018-03-12 11:52:33,854 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO o.a.d.e.c.ClassCompilerSelector - Java compiler policy: DEFAULT, Debug option: true 2018-03-12 11:52:35,929 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming stream due to memory limits. Current Allocation: 92340224. 2018-03-12 11:52:35,929 [BitServer-3] WARN o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming stream due to memory limits. Current Allocation: 92340224. 2018-03-12 11:52:35,930 [BitServer-3] ERROR o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer. 2018-03-12 11:52:35,930 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer. 2018-03-12 11:52:35,930 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming stream due to memory limits. Current Allocation: 83886080. 2018-03-12 11:52:35,930 [BitServer-3] WARN o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming stream due to memory limits. Current Allocation: 83886080. 2018-03-12 11:52:35,930 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer. 2018-03-12 11:52:35,930 [BitServer-3] ERROR o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer. 2018-03-12 11:52:35,931 [BitServer-3] WARN o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming stream due to memory limits. Current Allocation: 83886080. 2018-03-12 11:52:35,931 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming stream due to memory limits. Current Allocation: 83886080. 2018-03-12 11:52:35,931 [BitServer-3] ERROR o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer. 2018-03-12 11:52:35,931 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer. ... ... 2018-03-12 11:52:35,939 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming stream due to memory limits. Current Allocation: 67174400. 2018-03-12 11:52:35,939 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer. 2018-03-12 11:52:35,939 [BitServer-2] WARN
[jira] [Created] (DRILL-6229) memory leak when output_batch_size is set to 4096
Khurram Faraaz created DRILL-6229: - Summary: memory leak when output_batch_size is set to 4096 Key: DRILL-6229 URL: https://issues.apache.org/jira/browse/DRILL-6229 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.12.0 Reporter: Khurram Faraaz When drill.exec.memory.operator.output_batch_size` = 4096 (4KB), we continuously see in each test run, the memory leak when tests are run from test framework. Drill 1.13.0 git.commit.id=766315ea17377199897d685ab801edd38394fe01 alter system set `drill.exec.memory.operator.output_batch_size` = 4096; To run flatten tests from drill test framework bin/run_tests -s Functional/flatten_operators -g functional -d -t 500 -n 40 The below two tests time out and we see the memory leak Timeouts: /root/drillTEST_FRAMEWORK/drill-test-framework/framework/resources/Functional/flatten_operators/10rows/filter24.q Timeouts: /root/drillTEST_FRAMEWORK/drill-test-framework/framework/resources/Functional/flatten_operators/10rows/filter19.q oadd.org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Connection ... (user client) closed unexpectedly. Drillbit down? Stack trace printed in drill test framework log {noformat} [Error Id: 54a16569-6555-4e92-aa5f-f76a8c11b31f ] at oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:376) at oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at oadd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) at oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1003) at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:632) at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:610) at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:553) at oadd.io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1236) at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618) at oadd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:602) at oadd.io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618) at oadd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:602) at oadd.io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73) at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618) at oadd.io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:35) at oadd.io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:607) at oadd.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399) at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) at java.lang.Thread.run(Thread.java:748) Channel is closed, discarding remaining 908 byte(s) in buffer. Exception while closing client. java.lang.RuntimeException: Exception while closing at oadd.org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46) at oadd.org.apache.drill.exec.client.DrillClient.close(DrillClient.java:459) at org.apache.drill.jdbc.impl.DrillConnectionImpl.closeOrWarn(DrillConnectionImpl.java:810) at org.apache.drill.jdbc.impl.DrillConnectionImpl.cleanup(DrillConnectionImpl.java:823) at org.apache.drill.jdbc.impl.DrillHandler.onConnectionClose(DrillHandler.java:36) at oadd.org.apache.calcite.avatica.AvaticaConnection.close(AvaticaConnection.java:205) at org.apache.drill.test.framework.ConnectionPool.close(ConnectionPool.java:102) at org.apache.drill.test.framework.TestDriver.runTests(TestDriver.java:562) at org.apache.drill.test.framework.TestDriver.main(TestDriver.java:100) Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (4198400) Allocator(ROOT) 0/4096/6713344/28631367680 (res/actual/peak/limit) at
[jira] [Created] (DRILL-6215) Use prepared statement instead of Statement in JdbcRecordReader class
Khurram Faraaz created DRILL-6215: - Summary: Use prepared statement instead of Statement in JdbcRecordReader class Key: DRILL-6215 URL: https://issues.apache.org/jira/browse/DRILL-6215 Project: Apache Drill Issue Type: Bug Components: Storage - JDBC Affects Versions: 1.12.0 Reporter: Khurram Faraaz Use prepared statement instead of Statement in JdbcRecordReader class, which is more efficient and less vulnerable to SQL injection attacks. Apache Drill 1.13.0-SNAPSHOT, commit : 9073aed67d89e8b2188870d6c812706085c9c41b Findbugs reports the below bug and suggests that we use prepared statement instead of Statement. {noformat} In class org.apache.drill.exec.store.jdbc.JdbcRecordReader In method org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(OperatorContext, OutputMutator) At JdbcRecordReader.java:[line 170] org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(OperatorContext, OutputMutator) passes a nonconstant String to an execute method on an SQL statement The method invokes the execute method on an SQL statement with a String that seems to be dynamically generated. Consider using a prepared statement instead. It is more efficient and less vulnerable to SQL injection attacks. {noformat} LOC - https://github.com/apache/drill/blob/a9ea4ec1c5645ddab4b7aef9ac060ff5f109b696/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcRecordReader.java#L170 {noformat} To run with findbugs: mvn clean install -Pfindbugs -DskipTests Findbugs will wirite the output to finbugsXml.html in the target directory of each module. For example the java-exec module report is located at: ./exec/java-exec/target/findbugs/findbugsXml.html Use find . -name "findbugsXml.html" to locate the files. {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6214) DrillRuntimeException: Unexpected byte 0xfe at position 0 encountered while decoding UTF8 string.
Khurram Faraaz created DRILL-6214: - Summary: DrillRuntimeException: Unexpected byte 0xfe at position 0 encountered while decoding UTF8 string. Key: DRILL-6214 URL: https://issues.apache.org/jira/browse/DRILL-6214 Project: Apache Drill Issue Type: Bug Components: Functions - Drill Affects Versions: 1.12.0 Reporter: Khurram Faraaz binary_string function returns DrillRuntimeException. Apache Drill 1.13.0-SNAPSHOT, commit id: 9073aed67d89e8b2188870d6c812706085c9c41b {noformat} 0: jdbc:drill:schema=dfs.tmp> select binary_string(convert_to('abcdefghi','UTF16')) binStr from (values(1)); Error: SYSTEM ERROR: DrillRuntimeException: Unexpected byte 0xfe at position 0 encountered while decoding UTF8 string. Fragment 1:0 [Error Id: 6233b0fe-19f2-40ff-9c9e-c4e317ff19d0 on qa102-48.qa.lab:31010] (state=,code=0) Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Unexpected byte 0xfe at position 0 encountered while decoding UTF8 string. at org.apache.drill.exec.expr.fn.impl.StringFunctionUtil.utf8CharLen(StringFunctionUtil.java:88) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.expr.fn.impl.StringFunctionUtil.getUTF8CharLength(StringFunctionUtil.java:32) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.test.generated.ProjectorGen346.doSetup(ProjectorTemplate.java:116) ~[na:na] at org.apache.drill.exec.test.generated.ProjectorGen346.setup(ProjectorTemplate.java:101) ~[na:na] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchemaFromInput(ProjectRecordBatch.java:490) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:498) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:82) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:233) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_131] at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_131] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) ~[hadoop-common-2.7.0-mapr-1707.jar:na] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] ... 4 common frames omitted {noformat} {noformat} 0: jdbc:drill:schema=dfs.tmp> select binary_string(convert_to('123456','UTF16')) binStr from (values(1)); Error: SYSTEM ERROR: DrillRuntimeException: Unexpected byte 0xfe at position 0 encountered while decoding UTF8 string. Fragment 1:0 [Error Id: 8a1bb0f0-7a9a-46da-b5e9-46e184690e2b on qa102-46.qa.lab:31010] (state=,code=0) Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Unexpected byte 0xfe at position 0 encountered while decoding UTF8 string. at org.apache.drill.exec.expr.fn.impl.StringFunctionUtil.utf8CharLen(StringFunctionUtil.java:88) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.expr.fn.impl.StringFunctionUtil.getUTF8CharLength(StringFunctionUtil.java:32) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.test.generated.ProjectorGen272.doSetup(ProjectorTemplate.java:116) ~[na:na] at org.apache.drill.exec.test.generated.ProjectorGen272.setup(ProjectorTemplate.java:101) ~[na:na] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchemaFromInput(ProjectRecordBatch.java:490) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:498)
[jira] [Created] (DRILL-6181) CTAS should support writing nested structures (nested lists) to parquet.
Khurram Faraaz created DRILL-6181: - Summary: CTAS should support writing nested structures (nested lists) to parquet. Key: DRILL-6181 URL: https://issues.apache.org/jira/browse/DRILL-6181 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Affects Versions: 1.12.0 Reporter: Khurram Faraaz Both Parquet and Hive support writing nested structures into parquet https://issues.apache.org/jira/browse/HIVE-8909 https://issues.apache.org/jira/browse/PARQUET-113 A CTAS from Drill fails when there is a nested list of lists, in one of the columns in the project. JSON data used in the test, note that "arr" is a nested list of lists {noformat} [root@qa102-45 ~]# cat jsonToParquet_02.json {"id":"123","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]} {"id":"3","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]} {"id":"13","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]} {"id":"12","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]} {"id":"2","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]} {"id":"1","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]} {"id":"230","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]} {"id":"1230","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]} {"id":"1123","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]} {"id":"2123","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]} {"id":"1523","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]} [root@qa102-45 ~]# {noformat} CTAS fails with UnsupportedOperationException on Drill 1.12.0-mapr commit id bb07ebbb9ba8742f44689f8bd8efb5853c5edea0 {noformat} 0: jdbc:drill:schema=dfs.tmp> CREATE TABLE tbl_prq_from_json_02 as select id, arr from `jsonToParquet_02.json`; Error: SYSTEM ERROR: UnsupportedOperationException: Unsupported type LIST Fragment 0:0 [Error Id: 7e5b3c2d-9cf1-4e87-96c8-e7e7e8055ddf on qa102-45.qa.lab:31010] (state=,code=0) {noformat} Stack trace from drillbit.log {noformat} 2018-02-22 09:56:54,368 [2570fb99-62da-a516-2c1f-0381e21723ae:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: UnsupportedOperationException: Unsupported type LIST Fragment 0:0 [Error Id: 7e5b3c2d-9cf1-4e87-96c8-e7e7e8055ddf on qa102-45.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: UnsupportedOperationException: Unsupported type LIST Fragment 0:0 [Error Id: 7e5b3c2d-9cf1-4e87-96c8-e7e7e8055ddf on qa102-45.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586) ~[drill-common-1.12.0-mapr.jar:1.12.0-mapr] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301) [drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267) [drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.12.0-mapr.jar:1.12.0-mapr] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] Caused by: java.lang.UnsupportedOperationException: Unsupported type LIST at org.apache.drill.exec.store.parquet.ParquetRecordWriter.getType(ParquetRecordWriter.java:253) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr] at org.apache.drill.exec.store.parquet.ParquetRecordWriter.newSchema(ParquetRecordWriter.java:205) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr] at org.apache.drill.exec.store.parquet.ParquetRecordWriter.updateSchema(ParquetRecordWriter.java:190) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr] at org.apache.drill.exec.physical.impl.WriterRecordBatch.setupNewSchema(WriterRecordBatch.java:157) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr] at org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:103) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134)
[jira] [Created] (DRILL-6155) support for pagination of results on Drill's web UI
Khurram Faraaz created DRILL-6155: - Summary: support for pagination of results on Drill's web UI Key: DRILL-6155 URL: https://issues.apache.org/jira/browse/DRILL-6155 Project: Apache Drill Issue Type: Bug Components: Client - Java Affects Versions: 1.12.0 Reporter: Khurram Faraaz We need support for pagination of results on Drill's web UI. Today if a query returns say a million results, those results are not paginated on Drill's web UI. Such large number of results should be paginated as required on the web UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6146) UNION with empty input on any one side returns incorrect results
Khurram Faraaz created DRILL-6146: - Summary: UNION with empty input on any one side returns incorrect results Key: DRILL-6146 URL: https://issues.apache.org/jira/browse/DRILL-6146 Project: Apache Drill Issue Type: Bug Components: Query Planning Optimization Affects Versions: 1.12.0 Reporter: Khurram Faraaz When any one side of the UNION has an empty file as input, Drill returns incorrect results. table t3 does not have any data inserted into its rows. Postgress returns 1 as the result for both the queries, whereas Drill does not. {noformat} postgres=# create table t3(id int, name varchar(25)); CREATE TABLE postgres=# select * from (values(1)) t union select id from t3; 1 postgres=# select id from t3 union select * from (values(1)) t; 1 {noformat} Results from Drill 1.12.0-mapr, note we return result 1 as result after the union. We have a directory named empty_JSON_f , and it has a single empty JSON file (that JSON file has no content in it, it is empty). {noformat} 0: jdbc:drill:schema=dfs.tmp> select * from (values(1)) UNION select id from empty_JSON_f; +-+ | EXPR$0 | +-+ | 1 | +-+ 1 row selected (2.272 seconds){noformat} However, in this query we return null and loose the value 1 from the right hand side, after the union, this doesn't seem correct {noformat} 0: jdbc:drill:schema=dfs.tmp> select id from empty_JSON_f UNION select * from (values(1)); +---+ | id | +---+ | null | +---+ 1 row selected (0.33 seconds){noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6141) JOIN query that uses USING clause returns incorrect results
Khurram Faraaz created DRILL-6141: - Summary: JOIN query that uses USING clause returns incorrect results Key: DRILL-6141 URL: https://issues.apache.org/jira/browse/DRILL-6141 Project: Apache Drill Issue Type: Bug Components: Query Planning Optimization Affects Versions: 1.12.0 Reporter: Khurram Faraaz Join query that uses USING clause returns incorrect results. Postgres 9.2.23 returns only one occurrence of the "id" column {noformat} postgres=# create table t1(id int, name varchar(30)); CREATE TABLE postgres=# create table t2(id int, name varchar(30)); CREATE TABLE postgres=# select * from t1; id | name +--- 10 | John 13 | Kevin 15 | Susan (3 rows) postgres=# select * from t2; id | name +--- 19 | Kyle 13 | Kevin 1 | Bob 17 | Kumar (4 rows) postgres=# select * from t1 JOIN t2 USING(id); id | name | name +---+--- 13 | Kevin | Kevin (1 row) {noformat} results from Drill 1.12.0-mapr commit : 2de42491be795721bcb4059bd46e27fc33272309 {noformat} 0: jdbc:drill:schema=dfs.tmp> create table t1 as select cast(columns[0] as int) c1, cast(columns[1] as varchar(30)) c2 from `t1.csv`; +---++ | Fragment | Number of records written | +---++ | 0_0 | 3 | +---++ 1 row selected (0.213 seconds) 0: jdbc:drill:schema=dfs.tmp> create table t2 as select cast(columns[0] as int) c1, cast(columns[1] as varchar(30)) c2 from `t2.csv`; +---++ | Fragment | Number of records written | +---++ | 0_0 | 4 | +---++ 1 row selected (0.168 seconds) 0: jdbc:drill:schema=dfs.tmp> select * from t1; +-++ | c1 | c2 | +-++ | 10 | John | | 13 | Kevin | | 15 | Susan | +-++ 3 rows selected (0.15 seconds) 0: jdbc:drill:schema=dfs.tmp> select * from t2; +-++ | c1 | c2 | +-++ | 19 | Kyle | | 13 | Kevin | | 1 | Bob | | 17 | Kumar | +-++ 4 rows selected (0.171 seconds) ## Note that Drill returns an extra column, unlike Postgres, for the same query over same data 0: jdbc:drill:schema=dfs.tmp> select * from t1 JOIN t2 USING(c1); +-++--++ | c1 | c2 | c10 | c20 | +-++--++ | 13 | Kevin | 13 | Kevin | +-++--++ 1 row selected (0.256 seconds) ## explain plan for above query 0: jdbc:drill:schema=dfs.tmp> explain plan for select * from t1 JOIN t2 USING(c1); +--+--+ | text | json | +--+--+ | 00-00 Screen 00-01 ProjectAllowDup(*=[$0], *0=[$1]) 00-02 Project(T49¦¦*=[$0], T48¦¦*=[$2]) 00-03 Project(T49¦¦*=[$2], c10=[$3], T48¦¦*=[$0], c1=[$1]) 00-04 HashJoin(condition=[=($3, $1)], joinType=[inner]) 00-06 Project(T48¦¦*=[$0], c1=[$1]) 00-08 Scan(table=[[dfs, tmp, t2]], groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:///tmp/t2]], selectionRoot=maprfs:/tmp/t2, numFiles=1, numRowGroups=1, usedMetadataFile=false, columns=[`*`]]]) 00-05 Project(T49¦¦*=[$0], c10=[$1]) 00-07 Project(T49¦¦*=[$0], c1=[$1]) 00-09 Scan(table=[[dfs, tmp, t1]], groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:///tmp/t1]], selectionRoot=maprfs:/tmp/t1, numFiles=1, numRowGroups=1, usedMetadataFile=false, columns=[`*`]]]) {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-5965) Array index access returns an empty array
Khurram Faraaz created DRILL-5965: - Summary: Array index access returns an empty array Key: DRILL-5965 URL: https://issues.apache.org/jira/browse/DRILL-5965 Project: Apache Drill Issue Type: Bug Components: Storage - JSON Affects Versions: 1.12.0 Reporter: Khurram Faraaz Accessing an array with [] from JSON data, returns an empty string, whereas it should return the actual data from the array at the index and not an empty array. Drill 1.11.0-mapr commit: 065d72ba48c7af6b389b763753ecb6bf7d229ce8 {noformat} 0: jdbc:drill:schema=dfs.tmp> select t.structured_rep[0] from `cornell_nlvr_train.json` t limit 1; +-+ | EXPR$0 | +-+ | [] | +-+ 1 row selected (0.249 seconds) {noformat} Where as accessing the elements of the array returns correct results. {noformat} 0: jdbc:drill:schema=dfs.tmp> select t.structured_rep[0][0] from `cornell_nlvr_train.json` t limit 1; +---+ |EXPR$0 | +---+ | {"y_loc":21,"size":20,"type":"triangle","x_loc":27,"color":"Yellow"} | +---+ 1 row selected (0.325 seconds) 0: jdbc:drill:schema=dfs.tmp> select t.structured_rep[0][1] from `cornell_nlvr_train.json` t limit 1; +-+ | EXPR$0| +-+ | {"y_loc":60,"size":10,"type":"circle","x_loc":59,"color":"Yellow"} | +-+ 1 row selected (0.247 seconds) {noformat} Data used in the test {noformat} { "sentence": "There is a circle closely touching a corner of a box.", "label": "true", "identifier": "1304-0", "directory": "74", "evals": { "r0": "true" }, "structured_rep": [ [{ "y_loc": 21, "size": 20, "type": "triangle", "x_loc": 27, "color": "Yellow" }, { "y_loc": 60, "size": 10, "type": "circle", "x_loc": 59, "color": "Yellow" }], [{ "y_loc": 81, "size": 10, "type": "triangle", "x_loc": 48, "color": "Yellow" }, { "y_loc": 64, "size": 20, "type": "circle", "x_loc": 77, "color": "#0099ff" }], [{ "y_loc": 2, "size": 20, "type": "triangle", "x_loc": 62, "color": "Yellow" }, { "y_loc": 70, "size": 30, "type": "circle", "x_loc": 70, "color": "Black" }, { "y_loc": 51, "size": 20, "type": "circle", "x_loc": 30, "color": "#0099ff" }, { "y_loc": 42, "size": 20, "type": "circle", "x_loc": 67, "color": "Yellow" }, { "y_loc": 73, "size": 20, "type": "circle", "x_loc": 37, "color": "Black" }, { "y_loc": 14, "size": 30, "type": "triangle", "x_loc": 7, "color": "Yellow" }, { "y_loc": 27, "size": 10, "type": "circle", "x_loc": 48, "color": "Black" }] ] } {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
Khurram Faraaz created DRILL-5963: - Summary: Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever. Key: DRILL-5963 URL: https://issues.apache.org/jira/browse/DRILL-5963 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.12.0 Environment: Drill 1.12.0-SNAPSHOT, commit: 4a718a0bd728ae02b502ac93620d132f0f6e1b6c Reporter: Khurram Faraaz Priority: Critical Canceling the below query that is hung in planning state, leaves the query in ENQUEUED state for ever. Here is the query that is hung in planning state {noformat} 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA from (values(1)); +--+ | | +--+ +--+ No rows selected (304.291 seconds) {noformat} Explain plan for that query also just hangs. {noformat} explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA from (values(1)); ... {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5947) COUNT() on empty JSON results in schema change error
Khurram Faraaz created DRILL-5947: - Summary: COUNT() on empty JSON results in schema change error Key: DRILL-5947 URL: https://issues.apache.org/jira/browse/DRILL-5947 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Reporter: Khurram Faraaz Count over a field, from a non-empty and another empty JSON file results in schema change error, "Streaming aggregate does not support schema changes" {noformat} OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 apache drill 1.11.0-mapr "drill baby drill" 0: jdbc:drill:schema=dfs.tmp> select count(id) from `diff_types`; Error: UNSUPPORTED_OPERATION ERROR: Streaming aggregate does not support schema changes Prior schema : BatchSchema [fields=[id(BIGINT:OPTIONAL)], selectionVector=NONE] New schema : BatchSchema [fields=[id(INT:OPTIONAL)], selectionVector=NONE] Fragment 0:0 [Error Id: c1c77a0e-c7cf-494e-b8d2-ef2cc7e816e0 on centos-01.qa.lab:31010] (state=,code=0) 0: jdbc:drill:schema=dfs.tmp> select id from `dif`Display all 872 possibilities? (y or n) 0: jdbc:drill:schema=dfs.tmp> select id from `diff_types/m1.json`; +---+ | id | +---+ | 1234 | +---+ 1 row selected (0.157 seconds) 0: jdbc:drill:schema=dfs.tmp> select id from `diff_types/m2.json`; +---+ | id | +---+ | null | +---+ 1 row selected (0.169 seconds) 0: jdbc:drill:schema=dfs.tmp> select * from `diff_types`; +---+ | id | +---+ | 1234 | | null | +---+ 2 rows selected (0.209 seconds) 0: jdbc:drill:schema=dfs.tmp> select count(*) from `diff_types`; +-+ | EXPR$0 | +-+ | 2 | +-+ 1 row selected (0.208 seconds) {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5945) Streaming aggregate does not support schema changes
Khurram Faraaz created DRILL-5945: - Summary: Streaming aggregate does not support schema changes Key: DRILL-5945 URL: https://issues.apache.org/jira/browse/DRILL-5945 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Reporter: Khurram Faraaz There are two JSON files, one of the arrays "arr" in f2.json is empty, whereas array "arr" in f1.json is not empty, a count on arr results in, "Streaming aggregate does not support schema changes" {noformat} 0: jdbc:drill:schema=dfs.tmp> select count(arr) from `f*.json`; Error: UNSUPPORTED_OPERATION ERROR: Streaming aggregate does not support schema changes Prior schema : BatchSchema [fields=[arr(VARCHAR:REPEATED) [$data$(VARCHAR:REQUIRED)]], selectionVector=NONE] New schema : BatchSchema [fields=[arr(INT:REPEATED) [$data$(INT:REQUIRED)]], selectionVector=NONE] Fragment 0:0 [Error Id: 9081b02f-0de8-486a-b58b-f28ec3b84d98 on centos-01.qa.lab:31010] (state=,code=0) {noformat} Stack trace from drillbit.log {noformat} 2017-11-08 13:36:48,159 [25fc870f-e4d6-7107-c6cc-6e68ba92c4ba:frag:0:0] INFO o.a.d.e.p.i.a.StreamingAggBatch - User Error Occurred: Streaming aggregate does not support schema changes Prior schema : BatchSchema [fields=[arr(VARCHAR:REPEATED) [$data$(VARCHAR:REQUIRED)]], selectionVector=NONE] New schema : BatchSchema [fields=[arr(INT:REPEATED) [$data$(INT:REQUIRED)]], selectionVector=NONE] org.apache.drill.common.exceptions.UserException: UNSUPPORTED_OPERATION ERROR: Streaming aggregate does not support schema changes Prior schema : BatchSchema [fields=[arr(VARCHAR:REPEATED) [$data$(VARCHAR:REQUIRED)]], selectionVector=NONE] New schema : BatchSchema [fields=[arr(INT:REPEATED) [$data$(INT:REQUIRED)]], selectionVector=NONE] [Error Id: 9081b02f-0de8-486a-b58b-f28ec3b84d98 ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586) ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr] at org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext(StreamingAggBatch.java:207) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr] at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr] at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_144] at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_144] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) [hadoop-common-2.7.0-mapr-1607.jar:na] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:227) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.11.0-mapr.jar:1.11.0-mapr] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144] {noformat} Data used in the query {noformat} [root@centos-01 json_data]# cat f1.json { "id":1, "arr":["str1","str2","str3","str4"] } [root@centos-01 json_data]# cat f2.json { "id":2, "arr":[] } >From sqlline OpenJDK 64-Bit Server VM warning: ignoring option
[jira] [Created] (DRILL-5909) need new JMX metrics for (FAILED and CANCELED) queries
Khurram Faraaz created DRILL-5909: - Summary: need new JMX metrics for (FAILED and CANCELED) queries Key: DRILL-5909 URL: https://issues.apache.org/jira/browse/DRILL-5909 Project: Apache Drill Issue Type: Bug Components: Execution - Monitoring Affects Versions: 1.11.0, 1.12.0 Reporter: Khurram Faraaz we have these JMX metrics today {noformat} drill.queries.running drill.queries.completed {noformat} we need these new JMX metrics {noformat} drill.queries.failed drill.queries.canceled {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5806) DrillRuntimeException: Interrupted but context.shouldContinue() is true
Khurram Faraaz created DRILL-5806: - Summary: DrillRuntimeException: Interrupted but context.shouldContinue() is true Key: DRILL-5806 URL: https://issues.apache.org/jira/browse/DRILL-5806 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.12.0 Environment: Drill 1.12.0 commit : aaff1b35b7339fb4e6ab480dd517994ff9f0a5c5 Reporter: Khurram Faraaz On a three node cluster 1. run concurrent queries (TPC-DS query 11) from a Java program. 2. stop the drillbit (foreman drillbit) this way, /opt/mapr/drill/drill-1.12.0/bin/drillbit.sh stop 3. InterruptedException: null, is written to the drillbit.log Stack trace from drillbit.log {noformat} 2017-09-19 21:49:20,867 [263e6f48-0ace-0c0d-4f90-55ae2f0d778b:frag:5:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: InterruptedException Fragment 5:0 [Error Id: 63ce8c18-040a-47f9-9643-e826de9a1a27 on centos-01.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: InterruptedException Fragment 5:0 [Error Id: 63ce8c18-040a-47f9-9643-e826de9a1a27 on centos-01.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550) ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:298) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Interrupted but context.shouldContinue() is true at org.apache.drill.exec.work.batch.BaseRawBatchBuffer.getNext(BaseRawBatchBuffer.java:178) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.getNextBatch(UnorderedReceiverBatch.java:141) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.next(UnorderedReceiverBatch.java:164) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:141) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.test.generated.HashAggregatorGen498.doWork(HashAggTemplate.java:581) ~[na:na] at org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:168) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[jira] [Resolved] (DRILL-3241) Query with window function runs out of direct memory and does not report back to client that it did
[ https://issues.apache.org/jira/browse/DRILL-3241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz resolved DRILL-3241. --- Resolution: Fixed > Query with window function runs out of direct memory and does not report back > to client that it did > --- > > Key: DRILL-3241 > URL: https://issues.apache.org/jira/browse/DRILL-3241 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.0.0 >Reporter: Victoria Markman >Assignee: Deneche A. Hakim >Priority: Critical > Fix For: 1.12.0 > > > Even though query run out of memory and was cancelled on the server, client > (sqlline) was never notified of the event and it appears to the user that > query is hung. > Configuration: > Single drillbit configured with: > DRILL_MAX_DIRECT_MEMORY="2G" > DRILL_HEAP="1G" > TPCDS100 parquet files > Query: > {code} > select > sum(ss_quantity) over(partition by ss_store_sk order by ss_sold_date_sk) > from store_sales; > {code} > drillbit.log > {code} > 2015-06-01 21:42:29,514 [BitServer-5] ERROR > o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication. > Connection: /10.10.88.133:31012 <--> /10.10.88.133:38887 (data server). > Closing connection. > io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Direct > buffer memory > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:233) > ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at > io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at > io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:618) > [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at > io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:329) > [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:250) > [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > [netty-common-4.0.27.Final.jar:4.0.27.Final] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] > Caused by: java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_71] > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) > ~[na:1.7.0_71] > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > ~[na:1.7.0_71] > at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:437) > ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] > at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:179) > ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] > at io.netty.buffer.PoolArena.allocate(PoolArena.java:168) > ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] > at io.netty.buffer.PoolArena.reallocate(PoolArena.java:280) > ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] > at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:110) > ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] > at > io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251) > ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] > at > io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849) > ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] > at > io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:841) > ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] > at > io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:831) > ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] > at
[jira] [Created] (DRILL-5793) NPE on close
Khurram Faraaz created DRILL-5793: - Summary: NPE on close Key: DRILL-5793 URL: https://issues.apache.org/jira/browse/DRILL-5793 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.12.0 Environment: Drill 1.12.0 commit : aaff1b35b7339fb4e6ab480dd517994ff9f0a5c5 Reporter: Khurram Faraaz The code looks wrong: {noformat} @Override public void close() throws Exception { options.close(); } If the shutdown occurs to early, options is not yet assigned and an NPE results. {noformat} {noformat} 2017-09-14 20:16:39,551 [main] DEBUG o.apache.drill.exec.server.Drillbit - Shutdown begun. 2017-09-14 20:16:41,560 [pool-5-thread-1] INFO o.a.drill.exec.rpc.user.UserServer - closed eventLoopGroup io.netty.channel.nio.NioEventLoopGroup@71a84ff4 in 1006 ms 2017-09-14 20:16:41,560 [pool-5-thread-2] INFO o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup io.netty.channel.nio.NioEventLoopGroup@f711283 in 1005 ms 2017-09-14 20:16:41,561 [pool-5-thread-1] INFO o.a.drill.exec.service.ServiceEngine - closed userServer in 1007 ms 2017-09-14 20:16:41,562 [pool-5-thread-2] DEBUG o.a.drill.exec.memory.BaseAllocator - closed allocator[rpc:bit-data]. 2017-09-14 20:16:41,562 [pool-5-thread-2] INFO o.a.drill.exec.service.ServiceEngine - closed dataPool in 1008 ms 2017-09-14 20:16:41,563 [main] DEBUG o.a.drill.exec.memory.BaseAllocator - closed allocator[rpc:user]. 2017-09-14 20:16:41,563 [main] DEBUG o.a.drill.exec.memory.BaseAllocator - closed allocator[rpc:bit-control]. 2017-09-14 20:16:41,593 [main] DEBUG o.a.drill.exec.memory.BaseAllocator - closed allocator[ROOT]. 2017-09-14 20:16:41,593 [main] WARN o.apache.drill.exec.server.Drillbit - Failure on close() java.lang.NullPointerException: null at org.apache.drill.exec.server.options.SystemOptionManager.close(SystemOptionManager.java:369) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.server.DrillbitContext.close(DrillbitContext.java:241) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.work.WorkManager.close(WorkManager.java:154) ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:173) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:314) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:290) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:286) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT] {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (DRILL-4296) Query hangs in CANCELLATION_REQUESTED when cancelled after it starts returning results
[ https://issues.apache.org/jira/browse/DRILL-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz resolved DRILL-4296. --- Resolution: Fixed > Query hangs in CANCELLATION_REQUESTED when cancelled after it starts > returning results > -- > > Key: DRILL-4296 > URL: https://issues.apache.org/jira/browse/DRILL-4296 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.5.0 > Environment: commit.id=c9dbfbd > 2 nodes with 32 cores and 32GB of max direct memory for drill >Reporter: Deneche A. Hakim > Attachments: 295eefc3-8d15-d63b-a721-3fde365b639c.sys.drill, > data.tar.gz, node1_jstack.txt, node2_jstack.txt > > > After running the following queries (it's the same reproduction from > DRILL-2274): > {noformat} > set planner.memory.max_query_memory_per_node=8589934592; > select sub1.uid from `all2274.json` sub1 inner join `all2274.json` sub2 on > sub1.uid = sub2.uid order by sub1.uid; > {noformat} > After the query starts returning results, I cancelled the query from sqlline. > This caused the query to hang in a CANCELLATION_REQUESTED state. > Looking at jstack (attached) the root fragment is blocked waiting for Ack > from the client. > The the foreman node (which also runs Zookeeper) runs out of disk space once > the query finishes spilling, which seems to contribute to the issue. Once I > changed the spill directory to nfs I no longer so the issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5591) non-ASCII characters in text file result in MalformedInputException
Khurram Faraaz created DRILL-5591: - Summary: non-ASCII characters in text file result in MalformedInputException Key: DRILL-5591 URL: https://issues.apache.org/jira/browse/DRILL-5591 Project: Apache Drill Issue Type: Bug Components: Storage - Text & CSV Affects Versions: 1.11.0 Reporter: Khurram Faraaz I am on Drill 1.11.0 commit id: 874bf62 To repro the issue: wget http://cfdisat.blob.core.windows.net/lco/l_RFC_2017_05_11_2.txt.gz gunzip l_RFC_2017_05_11_2.txt.gz hadoop fs -put l_RFC_2017_05_11_2.txt /tmp There are some non-ASCII characters at the beginning and end of the file used in the test. {noformat} [root@centos-01 drill_5590]# head l_RFC_2017_05_11_2.txt ����0���1�� ��� ��RFC|SNCF|SUBCONTRATACION CUBB910321AC1|0|0 CUBB9104187K9|0|0 CUBB910709KD0|0|0 CUBB910817CE8|0|0 CUBB9111286YA|0|0 CUBB920408J69|0|0 {noformat} Failing query {noformat} 0: jdbc:drill:schema=dfs.tmp> select count(1) from `l_RFC_2017_05_11_2.txt` t where columns[0] like 'CUBA7706%'; Error: SYSTEM ERROR: MalformedInputException: Input length = 1 Fragment 0:0 [Error Id: cdfa704c-0bc8-4791-95ae-d05b4c63beab on centos-01.qa.lab:31010] (state=,code=0) {noformat} Stack trace from drillbit.log {noformat} Caused by: java.lang.RuntimeException: java.nio.charset.MalformedInputException: Input length = 1 at org.apache.drill.exec.expr.fn.impl.CharSequenceWrapper.decodeUT8(CharSequenceWrapper.java:185) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.expr.fn.impl.CharSequenceWrapper.setBuffer(CharSequenceWrapper.java:119) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.test.generated.FiltererGen15.doEval(FilterTemplate2.java:50) ~[na:na] at org.apache.drill.exec.test.generated.FiltererGen15.filterBatchNoSV(FilterTemplate2.java:100) ~[na:na] at org.apache.drill.exec.test.generated.FiltererGen15.filterBatch(FilterTemplate2.java:73) ~[na:na] at org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.doWork(FilterRecordBatch.java:81) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:93) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:133) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
[jira] [Created] (DRILL-5581) Query with CASE statement returns wrong results
Khurram Faraaz created DRILL-5581: - Summary: Query with CASE statement returns wrong results Key: DRILL-5581 URL: https://issues.apache.org/jira/browse/DRILL-5581 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Reporter: Khurram Faraaz A query that uses case statement, returns wrong results. {noformat} Apache Drill 1.11.0-SNAPSHOT, commit id: 874bf629 [test@centos-101 ~]# cat order_sample.csv 202634342,2101,20160301 apache drill 1.11.0-SNAPSHOT "this isn't your grandfather's sql" 0: jdbc:drill:schema=dfs.tmp> ALTER SESSION SET `store.format`='csv'; +---++ | ok |summary | +---++ | true | store.format updated. | +---++ 1 row selected (0.245 seconds) 0: jdbc:drill:schema=dfs.tmp> CREATE VIEW `vw_order_sample_csv` as . . . . . . . . . . . . . . > SELECT . . . . . . . . . . . . . . > `columns`[0] AS `ND`, . . . . . . . . . . . . . . > CAST(`columns`[1] AS BIGINT) AS `col1`, . . . . . . . . . . . . . . > CAST(`columns`[2] AS BIGINT) AS `col2` . . . . . . . . . . . . . . > FROM `order_sample.csv`; +---+--+ | ok | summary| +---+--+ | true | View 'vw_order_sample_csv' created successfully in 'dfs.tmp' schema | +---+--+ 1 row selected (0.253 seconds) 0: jdbc:drill:schema=dfs.tmp> select . . . . . . . . . . . . . . > case . . . . . . . . . . . . . . > when col1 > col2 then col1 . . . . . . . . . . . . . . > else col2 . . . . . . . . . . . . . . > end as temp_col, . . . . . . . . . . . . . . > case . . . . . . . . . . . . . . > when col1 = 2101 and (20170302 - col2) > 1 then 'D' . . . . . . . . . . . . . . > when col2 = 2101 then 'P' . . . . . . . . . . . . . . > when col1 - col2 > 1 then '0' . . . . . . . . . . . . . . > else 'A' . . . . . . . . . . . . . . > end as status . . . . . . . . . . . . . . > from `vw_order_sample_csv`; +---+-+ | temp_col | status | +---+-+ | 20160301 | A | +---+-+ 1 row selected (0.318 seconds) 0: jdbc:drill:schema=dfs.tmp> explain plan for . . . . . . . . . . . . . . > select . . . . . . . . . . . . . . > case . . . . . . . . . . . . . . > when col1 > col2 then col1 . . . . . . . . . . . . . . > else col2 . . . . . . . . . . . . . . > end as temp_col, . . . . . . . . . . . . . . > case . . . . . . . . . . . . . . > when col1 = 2101 and (20170302 - col2) > 1 then 'D' . . . . . . . . . . . . . . > when col2 = 2101 then 'P' . . . . . . . . . . . . . . > when col1 - col2 > 1 then '0' . . . . . . . . . . . . . . > else 'A' . . . . . . . . . . . . . . > end as status . . . . . . . . . . . . . . > from `vw_order_sample_csv`; +--+--+ | text | json | +--+--+ | 00-00Screen 00-01 Project(temp_col=[CASE(>(CAST(ITEM($0, 1)):BIGINT, CAST(ITEM($0, 2)):BIGINT), CAST(ITEM($0, 1)):BIGINT, CAST(ITEM($0, 2)):BIGINT)], status=[CASE(AND(=(CAST(ITEM($0, 1)):BIGINT, 2101), >(-(20170302, CAST(ITEM($0, 2)):BIGINT), 1)), 'D', =(CAST(ITEM($0, 2)):BIGINT, 2101), 'P', >(-(CAST(ITEM($0, 1)):BIGINT, CAST(ITEM($0, 2)):BIGINT), 1), '0', 'A')]) 00-02Scan(groupscan=[EasyGroupScan [selectionRoot=maprfs:/tmp/order_sample.csv, numFiles=1, columns=[`columns`[1], `columns`[2]], files=[maprfs:///tmp/order_sample.csv]]]) // Details of Java compiler from sys.options 0: jdbc:drill:schema=dfs.tmp> select name, status from sys.options where name like '%java_compiler%'; ++--+ | name | status | ++--+ | exec.java.compiler.exp_in_method_size | DEFAULT | | exec.java_compiler | DEFAULT | | exec.java_compiler_debug | DEFAULT | | exec.java_compiler_janino_maxsize | DEFAULT | ++--+ 4 rows selected (0.21 seconds) {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5576) OutOfMemoryException when some CPU cores are taken offline while concurrent queries are under execution
Khurram Faraaz created DRILL-5576: - Summary: OutOfMemoryException when some CPU cores are taken offline while concurrent queries are under execution Key: DRILL-5576 URL: https://issues.apache.org/jira/browse/DRILL-5576 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Environment: 3 nodes CentOS cluster Reporter: Khurram Faraaz When we reduce the number of available CPU cores while concurrent queries are under execution we see an OOM. Drill 1.11.0 commit ID: d11aba2 three node CentOS 6.8 cluster On each of the nodes Drill's direct memory was set to export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"16G"} There are 24 cores on the node where foreman Drillbit is under execution. {noformat} [root@centos-01 logs]# lscpu Architecture: x86_64 CPU op-mode(s):32-bit, 64-bit Byte Order:Little Endian CPU(s):24 On-line CPU(s) list: 0,2,4,5,8,9,12,14,15,18,20,22 Off-line CPU(s) list: 1,3,6,7,10,11,13,16,17,19,21,23 Thread(s) per core:1 Core(s) per socket:4 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family:6 Model: 44 Model name:Intel(R) Xeon(R) CPU E5645 @ 2.40GHz Stepping: 2 CPU MHz: 1600.000 BogoMIPS: 4799.86 Virtualization:VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 12288K NUMA node0 CPU(s): 0,2,4,5,12,14,15 NUMA node1 CPU(s): 8,9,18,20,22 {noformat} Java code snippet that creates threads and executes TPC-DS query 11 concurrently {noformat} ExecutorService executor = Executors.newFixedThreadPool(48); try { for (int i = 1; i <= 48; i++) { executor.submit(new ConcurrentQuery(conn)); } } catch (Exception e) { System.out.println(e.getMessage()); e.printStackTrace(); } {noformat} While the TPC-DS Query 11 is under execution using above program, we take half of the available CPU cores offline {noformat} [root@centos-01 ~]# sh turnCPUCoresOffline.sh OFFLINE cores are : 1,3,6-7,10-11,13,16-17,19,21,23 ONLINE cores are : 0,2,4-5,8-9,12,14-15,18,20,22 {noformat} The result is we see an OutOfMemoryException, drillbit.log files are attached. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5573) Query profiles missing from Web UI and physical *.sys.drill file missing for query that was in CANCELLATION_REQUESTED state
Khurram Faraaz created DRILL-5573: - Summary: Query profiles missing from Web UI and physical *.sys.drill file missing for query that was in CANCELLATION_REQUESTED state Key: DRILL-5573 URL: https://issues.apache.org/jira/browse/DRILL-5573 Project: Apache Drill Issue Type: Bug Components: Web Server Affects Versions: 1.11.0 Environment: 3 node CentOS cluster Reporter: Khurram Faraaz A query that was in CANCELLATION_REQUESTED state after the Java concurrent program was terminated by issuing Ctrl-C from the command line. Next, stop and start Drillbits (Drill 1.11.0 commit id: d11aba2) on all nodes, and we see the below error message on Drill's Web UI, and we do not find the physical file .sys.drill under drill-1.11.0/logs/profiles folder clush -a /opt/mapr/drill/drill-1.11.0/bin/drillbit.sh stop clush -a /opt/mapr/drill/drill-1.11.0/bin/drillbit.sh start http://:8047/profiles/26c9073d-a8ff-17b0-5a86-34042bdfb494 { "errorMessage" : "VALIDATION ERROR: No profile with given query id '26c9073d-a8ff-17b0-5a86-34042bdfb494' exists. Please verify the query id.\n\n\n[Error Id: c8f1ed5c-d93f-42b7-840d-54a1ed8da132 ]" } Stack trace from drillbit.log from foreman node, for the above error id. {noformat} 2017-06-07 10:18:37,612 [main] INFO o.a.drill.exec.server.rest.WebServer - Setting up HTTP connector for web server 2017-06-07 10:18:38,601 [main] INFO o.apache.drill.exec.server.Drillbit - Startup completed (4149 ms). 2017-06-07 10:20:03,718 [qtp1575043618-89] INFO o.a.d.e.s.r.profile.ProfileResources - User Error Occurred: No profile with given query id '26c9073d-a8ff-17b0-5a86-34042bdfb494' exists. Please verify the query id. org.apache.drill.common.exceptions.UserException: VALIDATION ERROR: No profile with given query id '26c9073d-a8ff-17b0-5a86-34042bdfb494' exists. Please verify the query id. [Error Id: c8f1ed5c-d93f-42b7-840d-54a1ed8da132 ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.server.rest.profile.ProfileResources.getQueryProfile(ProfileResources.java:288) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.server.rest.profile.ProfileResources.getProfile(ProfileResources.java:310) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_91] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_91] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) [jersey-server-2.8.jar:na] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.8.jar:na] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.8.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.8.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.8.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.8.jar:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) [jersey-common-2.8.jar:na] at
[jira] [Created] (DRILL-5569) NullPointerException
Khurram Faraaz created DRILL-5569: - Summary: NullPointerException Key: DRILL-5569 URL: https://issues.apache.org/jira/browse/DRILL-5569 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Affects Versions: 1.11.0 Reporter: Khurram Faraaz The below Exception was seen when TPC-DS Query 4 was executed against Drill 1.11.0 Drill 1.11.0 git commit ID: d11aba2 [root@centos-01 mapr]# cat MapRBuildVersion 5.2.1.42646.GA Stack trace from drillbit.log {noformat} 2017-06-06 07:46:43,160 [Drillbit-ShutdownHook#0] WARN o.apache.drill.exec.work.WorkManager - Closing WorkManager but there are 80 running fragments. 2017-06-06 07:46:43,207 [Drillbit-ShutdownHook#0] INFO o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 959, cache miss count: 12, hit rate: 99% 2017-06-06 07:46:43,504 [scan-3] ERROR o.a.d.e.u.f.BufferedDirectBufInputStream - Error reading from stream 1_1_0.parquet. Error was : Error reading out of an FSDataInputStream using the Hadoop 2 ByteBuffer based read method. 2017-06-06 07:46:43,510 [scan-8] ERROR o.a.d.e.u.f.BufferedDirectBufInputStream - Error reading from stream 1_1_0.parquet. Error was : Error reading out of an FSDataInputStream using the Hadoop 2 ByteBuffer based read method. 2017-06-06 07:46:43,514 [scan-8] INFO o.a.d.e.s.p.c.AsyncPageReader - User Error Occurred: Exception occurred while reading from disk. (java.io.IOException: Error reading out of an FSDataInputStream using the Hadoop 2 ByteBuffer based read method.) org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: Exception occurred while reading from disk. File: /drill/testdata/tpcds_sf1/parquet/store_sales/1_1_0.parquet Column: ss_ext_list_price Row Group Start: 75660513 [Error Id: 3a758095-fcc4-4364-a50b-33a027c1beb6 ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.handleAndThrowException(AsyncPageReader.java:199) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.access$600(AsyncPageReader.java:81) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$AsyncPageReaderTask.call(AsyncPageReader.java:483) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$AsyncPageReaderTask.call(AsyncPageReader.java:392) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_65] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] Caused by: java.io.IOException: java.io.IOException: Error reading out of an FSDataInputStream using the Hadoop 2 ByteBuffer based read method. at org.apache.drill.exec.util.filereader.BufferedDirectBufInputStream.getNextBlock(BufferedDirectBufInputStream.java:185) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.util.filereader.BufferedDirectBufInputStream.readInternal(BufferedDirectBufInputStream.java:212) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.util.filereader.BufferedDirectBufInputStream.read(BufferedDirectBufInputStream.java:277) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.util.filereader.DirectBufInputStream.getNext(DirectBufInputStream.java:111) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$AsyncPageReaderTask.call(AsyncPageReader.java:437) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] ... 5 common frames omitted Caused by: java.io.IOException: Error reading out of an FSDataInputStream using the Hadoop 2 ByteBuffer based read method. at org.apache.parquet.hadoop.util.CompatibilityUtil.getBuf(CompatibilityUtil.java:99) ~[parquet-hadoop-1.8.1-drill-r0.jar:1.8.1-drill-r0] at org.apache.drill.exec.util.filereader.BufferedDirectBufInputStream.getNextBlock(BufferedDirectBufInputStream.java:182) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] ... 9 common frames omitted Caused by: java.lang.NullPointerException: null at com.mapr.fs.MapRFsInStream.readIntoDirectByteBuffer(MapRFsInStream.java:219) ~[maprfs-5.2.1-mapr.jar:5.2.1-mapr] at com.mapr.fs.MapRFsInStream.read(MapRFsInStream.java:333)
[jira] [Created] (DRILL-5570) InterruptedException: null
Khurram Faraaz created DRILL-5570: - Summary: InterruptedException: null Key: DRILL-5570 URL: https://issues.apache.org/jira/browse/DRILL-5570 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Environment: 3 node CentOS cluster Reporter: Khurram Faraaz When TPC-DS query11 was executed concurrently and one of the non-foreman Drillbits was stopped (./bin/drillbit.sh stop) we see the below system error InterruptedException in the drillbit.log of the non-foreman node Drill 1.11.0 git commit ID: d11aba2 [root@centos-01 mapr]# cat MapRBuildVersion 5.2.1.42646.GA {noformat} 2017-06-06 07:46:44,288 [26c9a242-dfa1-35be-b5f1-ff6b4fa66086:frag:11:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: InterruptedException Fragment 11:0 [Error Id: 40723399-8983-4777-a2bb-dc9d55ae338e on centos-02.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: InterruptedException Fragment 11:0 [Error Id: 40723399-8983-4777-a2bb-dc9d55ae338e on centos-02.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_65] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Interrupted but context.shouldContinue() is true at org.apache.drill.exec.work.batch.BaseRawBatchBuffer.getNext(BaseRawBatchBuffer.java:178) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.getNextBatch(UnorderedReceiverBatch.java:141) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.next(UnorderedReceiverBatch.java:159) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext(PartitionSenderRootExec.java:144) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_65] at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_65] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) ~[hadoop-common-2.7.0-mapr-1607.jar:na] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:227) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] ... 4 common frames omitted Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[na:1.8.0_65] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[na:1.8.0_65] at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492) ~[na:1.8.0_65] at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680) ~[na:1.8.0_65] at
[jira] [Created] (DRILL-5566) AssertionError: Internal error: invariant violated: call to wrong operator
Khurram Faraaz created DRILL-5566: - Summary: AssertionError: Internal error: invariant violated: call to wrong operator Key: DRILL-5566 URL: https://issues.apache.org/jira/browse/DRILL-5566 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Reporter: Khurram Faraaz CHARACTER_LENGTH is a non-reserved keyword as per the SQL specification. It is a monadic function that accepts exactly one operand or parameter. {noformat} ::= | | | | ... ... ::= | ::= { CHAR_LENGTH | CHARACTER_LENGTH } [ USING ] ... ... ::= CHARACTERS | OCTETS {noformat} Drill reports an assertion error in drillbit.log when character_length function is used in a SQL query. {noformat} 0: jdbc:drill:schema=dfs.tmp> select character_length(cast('hello' as varchar(10))) col1 from (values(1)); Error: SYSTEM ERROR: AssertionError: Internal error: invariant violated: call to wrong operator [Error Id: 49198839-5a1b-4786-9257-59739b27d2a8 on centos-01.qa.lab:31010] (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception during fragment initialization: Internal error: invariant violated: call to wrong operator org.apache.drill.exec.work.foreman.Foreman.run():297 java.util.concurrent.ThreadPoolExecutor.runWorker():1145 java.util.concurrent.ThreadPoolExecutor$Worker.run():615 java.lang.Thread.run():745 Caused By (java.lang.AssertionError) Internal error: invariant violated: call to wrong operator org.apache.calcite.util.Util.newInternal():777 org.apache.calcite.util.Util.permAssert():885 org.apache.calcite.sql2rel.ReflectiveConvertletTable$3.convertCall():219 org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall():59 org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit():4148 org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit():3581 org.apache.calcite.sql.SqlCall.accept():130 org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression():4040 org.apache.calcite.sql2rel.StandardConvertletTable$8.convertCall():185 org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall():59 org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit():4148 org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit():3581 org.apache.calcite.sql.SqlCall.accept():130 org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression():4040 org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectList():3411 org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl():612 org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect():568 org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive():2773 org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery():522 org.apache.drill.exec.planner.sql.SqlConverter.toRel():269 org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRel():623 org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert():195 org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():164 org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():131 org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79 org.apache.drill.exec.work.foreman.Foreman.runSQL():1050 org.apache.drill.exec.work.foreman.Foreman.run():280 java.util.concurrent.ThreadPoolExecutor.runWorker():1145 java.util.concurrent.ThreadPoolExecutor$Worker.run():615 java.lang.Thread.run():745 (state=,code=0) {noformat} Calcite supports character_length function {noformat} [root@centos-0170 csv]# ./sqlline sqlline version 1.1.9 sqlline> !connect jdbc:calcite:model=target/test-classes/model.json admin admin SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. 0: jdbc:calcite:model=target/test-classes/mod> select character_length(cast('hello' as varchar(10))) col1 from (values(1)); ++ |COL1| ++ | 5 | ++ 1 row selected (1.379 seconds) {noformat} Postgres 9.3 also supports character_length function {noformat} postgres=# select character_length(cast('hello' as varchar(10))) col1 from (values(1)) foo; col1 -- 5 (1 row) {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5564) IllegalStateException: allocator[op:21:1:5:HashJoinPOP]: buffer space (16674816) + prealloc space (0) + child space (0) != allocated (16740352)
Khurram Faraaz created DRILL-5564: - Summary: IllegalStateException: allocator[op:21:1:5:HashJoinPOP]: buffer space (16674816) + prealloc space (0) + child space (0) != allocated (16740352) Key: DRILL-5564 URL: https://issues.apache.org/jira/browse/DRILL-5564 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Environment: 3 node CentOS cluster Reporter: Khurram Faraaz Run a concurrent Java program that executes TPCDS query11 while the above concurrent java program is under execution stop foreman Drillbit (from another shell, using below command) ./bin/drillbit.sh stop and you will see the IllegalStateException: allocator[op:21:1:5:HashJoinPOP]: and another assertion error, in the drillbit.log AssertionError: Failure while stopping processing for operator id 10. Currently have states of processing:false, setup:false, waiting:true. Drill 1.11.0 git commit ID: d11aba2 (with assertions enabled) details from drillbit.log from the foreman Drillbit node. {noformat} 2017-06-05 18:38:33,838 [26ca5afa-7f6d-991b-1fdf-6196faddc229:frag:23:1] INFO o.a.d.e.w.fragment.FragmentExecutor - 26ca5afa-7f6d-991b-1fdf-6196faddc229:23:1: State change requested RUNNING --> FAILED 2017-06-05 18:38:33,849 [26ca5afa-7f6d-991b-1fdf-6196faddc229:frag:23:1] INFO o.a.d.e.w.fragment.FragmentExecutor - 26ca5afa-7f6d-991b-1fdf-6196faddc229:23:1: State change requested FAILED --> FINISHED 2017-06-05 18:38:33,852 [26ca5afa-7f6d-991b-1fdf-6196faddc229:frag:23:1] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: AssertionError: Failure while stopping processing for operator id 10. Currently have states of processing:false, setup:false, waiting:true. Fragment 23:1 [Error Id: a116b326-43ed-4569-a20e-a10ba03d215e on centos-01.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: AssertionError: Failure while stopping processing for operator id 10. Currently have states of processing:false, setup:false, waiting:true. Fragment 23:1 [Error Id: a116b326-43ed-4569-a20e-a10ba03d215e on centos-01.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] Caused by: java.lang.RuntimeException: java.lang.AssertionError: Failure while stopping processing for operator id 10. Currently have states of processing:false, setup:false, waiting:true. at org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:101) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:409) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:250) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] ... 4 common frames omitted Caused by: java.lang.AssertionError: Failure while stopping processing for operator id 10. Currently have states of processing:false, setup:false, waiting:true. at org.apache.drill.exec.ops.OperatorStats.stopProcessing(OperatorStats.java:167) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:255) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at
[jira] [Created] (DRILL-5563) Stop non foreman Drillbit results in IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.
Khurram Faraaz created DRILL-5563: - Summary: Stop non foreman Drillbit results in IllegalStateException: Allocator[ROOT] closed with outstanding child allocators. Key: DRILL-5563 URL: https://issues.apache.org/jira/browse/DRILL-5563 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Environment: 3 node CentOS cluster Reporter: Khurram Faraaz Stopping the non-foreman Drillbit normally (as shown below) results in IllegalStateException: Allocator[ROOT] closed with outstanding child allocators. /opt/mapr/drill/drill-1.11.0/bin/drillbit.sh stop Drill 1.11.0 commit ID: d11aba2 Details from drillbit.log {noformat} Mon Jun 5 09:29:09 UTC 2017 Terminating drillbit pid 28182 2017-06-05 09:29:09,651 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Received shutdown request. 2017-06-05 09:29:11,691 [pool-6-thread-1] INFO o.a.drill.exec.rpc.user.UserServer - closed eventLoopGroup io.netty.channel.nio.NioEventLoopGroup@55511dc2 in 1004 ms 2017-06-05 09:29:11,691 [pool-6-thread-2] INFO o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup io.netty.channel.nio.NioEventLoopGroup@4078d750 in 1004 ms 2017-06-05 09:29:11,692 [pool-6-thread-1] INFO o.a.drill.exec.service.ServiceEngine - closed userServer in 1005 ms 2017-06-05 09:29:11,692 [pool-6-thread-2] INFO o.a.drill.exec.service.ServiceEngine - closed dataPool in 1005 ms 2017-06-05 09:29:11,701 [Drillbit-ShutdownHook#0] INFO o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 21, cache miss count: 7, hit rate: 67% 2017-06-05 09:29:11,709 [Drillbit-ShutdownHook#0] ERROR o.a.d.exec.server.BootStrapContext - Error while closing java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child allocators. Allocator(ROOT) 0/800/201359872/17179869184 (res/actual/peak/limit) child allocators: 4 Allocator(frag:3:2) 200/0/0/200 (res/actual/peak/limit) child allocators: 0 ledgers: 0 reservations: 0 Allocator(frag:4:2) 200/0/0/200 (res/actual/peak/limit) child allocators: 0 ledgers: 0 reservations: 0 Allocator(frag:1:2) 200/0/0/200 (res/actual/peak/limit) child allocators: 0 ledgers: 0 reservations: 0 Allocator(frag:2:2) 200/0/0/200 (res/actual/peak/limit) child allocators: 0 ledgers: 0 reservations: 0 ledgers: 0 reservations: 0 at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:492) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) [drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) [drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:247) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) [drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) [drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:159) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:253) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] 2017-06-05 09:29:11,709 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Shutdown completed (2057 ms). {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5561) IllegalArgumentException: Self-suppression not permitted, seen when foreman Drillbit is killed
Khurram Faraaz created DRILL-5561: - Summary: IllegalArgumentException: Self-suppression not permitted, seen when foreman Drillbit is killed Key: DRILL-5561 URL: https://issues.apache.org/jira/browse/DRILL-5561 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Environment: 3 node Centos cluster Reporter: Khurram Faraaz Priority: Minor Drill 1.11.0 running on three nodes kill foreman Drillbit, [root@centos-01 ~]# kill -9 15852; while a long running SQL query (TPCDS Query 11) is under execution drillbit.log from centos-01.qa.lab (foreman Drillbit) has no Exceptions or Errors logged in it. snippet of drillbit.log from centos-02.qa.lab ( non foreman Drillbit ) {noformat} 2017-05-30 08:21:32,834 [BitClient-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 26d2d481-d949-1de8-9497-c1492f086501:12:0: State change requested RUNNING --> FAILED 2017-05-30 08:21:32,834 [BitServer-9] WARN o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed channel. Connection: /10.10.100.202:31012 <--> /10.10.100.201:50773 (data server) java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_65] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_65] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_65] at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_65] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_65] at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311) ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] at io.netty.buffer.WrappedByteBuf.setBytes(WrappedByteBuf.java:407) ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] at io.netty.buffer.UnsafeDirectLittleEndian.setBytes(UnsafeDirectLittleEndian.java:30) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final] at io.netty.buffer.DrillBuf.setBytes(DrillBuf.java:770) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final] at io.netty.buffer.MutableWrappedByteBuf.setBytes(MutableWrappedByteBuf.java:280) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final] at io.netty.buffer.ExpandableByteBuf.setBytes(ExpandableByteBuf.java:26) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881) ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] 2017-05-30 08:21:32,848 [BitClient-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 26d2d481-d949-1de8-9497-c1492f086501:16:0: State change requested RUNNING --> FAILED 2017-05-30 08:21:32,850 [BitClient-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 26d2d481-d949-1de8-9497-c1492f086501:14:0: State change requested RUNNING --> FAILED 2017-05-30 08:21:32,851 [BitClient-1] WARN o.apache.drill.exec.rpc.RequestIdMap - Failure while attempting to fail rpc response. java.lang.IllegalArgumentException: Self-suppression not permitted at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[na:1.8.0_65] at org.apache.drill.common.DeferredException.addException(DeferredException.java:88) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:409) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.access$700(FragmentExecutor.java:55) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at
[jira] [Created] (DRILL-5543) Inner join query results in IndexOutOfBoundsException
Khurram Faraaz created DRILL-5543: - Summary: Inner join query results in IndexOutOfBoundsException Key: DRILL-5543 URL: https://issues.apache.org/jira/browse/DRILL-5543 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Reporter: Khurram Faraaz Inner join query results in IndexOutOfBoundsException Steps to reproduce the IOB. Copy data from drill test framework../framework/resources/Datasources/joins directory to /drill/testdata/joins directory Run the below SQL. Apache Drill 1.11.0 commit ID: d11aba2e MapR version : 5.2.1.42646.GA [test@centos-ab xyzb]# uname -r 2.6.32-642.el6.x86_64 Three node CentOS cluster {noformat} 0: jdbc:drill:schema=dfs.tmp> use dfs.joins; +---++ | ok |summary | +---++ | true | Default schema changed to [dfs.joins] | +---++ 1 row selected (0.279 seconds) 0: jdbc:drill:schema=dfs.tmp> select . . . . . . . . . . . . . . > count(*) . . . . . . . . . . . . . . > from . . . . . . . . . . . . . . > j1 INNER JOIN j4 ON . . . . . . . . . . . . . . > ( . . . . . . . . . . . . . . > j1.c_date = j4.c_date . . . . . . . . . . . . . . > and j1.c_time = j4.c_time . . . . . . . . . . . . . . > and j1.c_timestamp = j4.c_timestamp . . . . . . . . . . . . . . > and j1.c_boolean = j4.c_boolean . . . . . . . . . . . . . . > and j1.c_varchar = j4.c_varchar . . . . . . . . . . . . . . > and j1.c_float = j4.c_float . . . . . . . . . . . . . . > and j1.c_double = j4.c_double . . . . . . . . . . . . . . > and j1.d9 = j4.d9 . . . . . . . . . . . . . . > and j1.d18 = j4.d18 . . . . . . . . . . . . . . > . . . . . . . . . . . . . . > ) . . . . . . . . . . . . . . > where j1.c_date IN (select c_date from j2); Error: SYSTEM ERROR: IndexOutOfBoundsException: index: 9, length: 1 (expected: range(0, 0)) Fragment 0:0 [Error Id: f8323164-581e-4377-87e0-f44e0bc1d037 on centos-01.qa.lab:31010] (state=,code=0) {noformat} Stack trace from drillbit.log {noformat} 2017-05-26 08:47:05,151 [26d8147b-394f-3fea-f1e1-ece75d9b3f41:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IndexOutOfBoundsException: index: 9, length: 1 (expected: range(0, 0)) Fragment 0:0 [Error Id: f8323164-581e-4377-87e0-f44e0bc1d037 on centos-01.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IndexOutOfBoundsException: index: 9, length: 1 (expected: range(0, 0)) Fragment 0:0 [Error Id: f8323164-581e-4377-87e0-f44e0bc1d037 on centos-01.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] Caused by: java.lang.IndexOutOfBoundsException: index: 9, length: 1 (expected: range(0, 0)) at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:123) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final] at io.netty.buffer.DrillBuf.chk(DrillBuf.java:147) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final] at io.netty.buffer.DrillBuf.getByte(DrillBuf.java:775) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final] at org.apache.drill.exec.vector.UInt1Vector$Accessor.get(UInt1Vector.java:354) ~[vector-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.vector.NullableDateVector$Accessor.isSet(NullableDateVector.java:394) ~[vector-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.test.generated.StreamingAggregatorGen383.outputRecordKeysPrev(StreamingAggTemplate.java:186) ~[na:na] at org.apache.drill.exec.test.generated.StreamingAggregatorGen383.outputToBatchPrev(StreamingAggTemplate.java:322) ~[na:na] at
[jira] [Created] (DRILL-5499) TPC-DS Query 4 concurrent execution RESOURCE ERROR: One or more nodes ran out of memory while executing the query.
Khurram Faraaz created DRILL-5499: - Summary: TPC-DS Query 4 concurrent execution RESOURCE ERROR: One or more nodes ran out of memory while executing the query. Key: DRILL-5499 URL: https://issues.apache.org/jira/browse/DRILL-5499 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Environment: single node CentOS Reporter: Khurram Faraaz Running TPC-DS query 4 sequentially does not result in error/OOM. TPC-DS query 4 takes 7 GB memory on an average, on Drill 1.11.0. However, there is an OOM and RESOURCE ERROR when we run TPC-DS query 4 concurrently using 8 threads on a single node that has 24 processing units available. {noformat} 2017-05-09 13:42:30,547 [BitServer-5] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication. Connection: /10.10.100.201:31012 <--> /10.10.100.201:59746 (data server). Closing connection. io.netty.handler.codec.DecoderException: org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer of size 25165824 due to memory limit. Current allocation: 21037056 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:233) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer of size 25165824 due to memory limit. Current allocation: 21037056 at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:220) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:195) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at io.netty.buffer.ExpandableByteBuf.capacity(ExpandableByteBuf.java:43) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final] at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251) ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849) ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:841) ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:831) ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final] at io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:92) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:227) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] ... 13 common frames omitted ... 2017-05-09 13:42:30,547 [26ee38c2-75c8-05cd-0cc0-9263fefba955:frag:34:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 26ee38c2-75c8-05cd-0cc0-9263fefba955:34:0: State change requested AWAITING_ALLOCATION --> RUNNING 2017-05-09 13:42:30,547 [26ee38c2-75c8-05cd-0cc0-9263fefba955:frag:34:0] INFO
[jira] [Created] (DRILL-5458) REPEATED_COUNT over varchar data results in SchemaChangeException
Khurram Faraaz created DRILL-5458: - Summary: REPEATED_COUNT over varchar data results in SchemaChangeException Key: DRILL-5458 URL: https://issues.apache.org/jira/browse/DRILL-5458 Project: Apache Drill Issue Type: Bug Affects Versions: 1.11.0 Reporter: Khurram Faraaz REPEATED_COUNT over varchar type data results in SchemaChangeException Apache Drill 1.11.0 commit id: 3e8b01d5 {noformat} // CTAS over JSON file. 0: jdbc:drill:schema=dfs.tmp> create table tblwarr01 as select convert_to(t.arr,'JSON') arr, id from `rptd_count.json` t; +---++ | Fragment | Number of records written | +---++ | 0_0 | 10 | +---++ 1 row selected (0.221 seconds) 0: jdbc:drill:schema=dfs.tmp> select * from tblwarr01; +--+-+ | arr | id | +--+-+ | [B@67cf1438 | 1 | | [B@4c389dc6 | 2 | | [B@18fe5942 | 3 | | [B@629608df | 4 | | [B@68209b09 | 5 | | [B@34a2a147 | 6 | | [B@210a5750 | 7 | | [B@2dea5622 | 8 | | [B@73bce9ba | 9 | | [B@7794edb2 | 10 | +--+-+ 10 rows selected (0.209 seconds) {noformat} {noformat} 0: jdbc:drill:schema=dfs.tmp> select convert_from(t.arr,'UTF8') from tblwarr01 t; +---+ |EXPR$0 | +---+ | [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] | | [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] | | [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] | | [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] | | [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] | | [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] | | [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] | | [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] | | [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] | | [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] | +---+ 10 rows selected (0.195 seconds) // Performing a REPEATED_COUNT on the array results in SchemaChangeException 0: jdbc:drill:schema=dfs.tmp> select REPEATED_COUNT(convert_from(t.arr,'UTF8')) from tblwarr01 t; Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to materialize incoming schema. Errors: Error in expression at index -1. Error: Missing function implementation: [repeated_count(VARCHAR-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.. Fragment 0:0 [Error Id: 1a891918-79f6-49d1-a594-f9ec5f28a99a on centos-01.qa.lab:31010] (state=,code=0) {noformat} Data used in CTAS. {noformat} [root@centos-01 ~]# cat rptd_count.json {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":1} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":2} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":3} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":4} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":5} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":6} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":7} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":8} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":9} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":10} [root@centos-01 ~]# {noformat} Stack trace from drillbit.log {noformat} 2017-05-02 05:17:03,651 [26f7e9b0-5f11-e764-6fa8-92c27ca2c6cf:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: SchemaChangeException: Failure while trying to materialize incoming schema. Errors: Error in expression at index -1. Error: Missing function implementation: [repeated_count(VARCHAR-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.. Fragment 0:0 [Error Id: 1a891918-79f6-49d1-a594-f9ec5f28a99a on centos-01.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: SchemaChangeException: Failure while trying to materialize incoming schema. Errors: Error in expression at index -1. Error: Missing function implementation: [repeated_count(VARCHAR-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.. Fragment 0:0 [Error Id: 1a891918-79f6-49d1-a594-f9ec5f28a99a on centos-01.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
[jira] [Created] (DRILL-5456) StringIndexOutOfBoundsException when converting a JSON array to UTF-8
Khurram Faraaz created DRILL-5456: - Summary: StringIndexOutOfBoundsException when converting a JSON array to UTF-8 Key: DRILL-5456 URL: https://issues.apache.org/jira/browse/DRILL-5456 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Reporter: Khurram Faraaz Convert a JSON array to UTF-8 using CONVERT_TO function results in StringIndexOutOfBoundsException Apache Drill 1.11.0 commit ID: 3e8b01d Data used in test {noformat} [root@centos-01 ~]# cat rptd_count.json {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":1} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":2} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":3} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":4} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":5} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":6} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":7} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":8} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":9} {"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":10} [root@centos-01 ~]# {noformat} {noformat} 0: jdbc:drill:schema=dfs.tmp> select convert_to(t.arr,'UTF-8') c, id from `rptd_count.json` t; Error: SYSTEM ERROR: StringIndexOutOfBoundsException: String index out of range: -3 [Error Id: 056a13c0-6c9f-403e-877e-040e907d6581 on centos-01.qa.lab:31010] (state=,code=0) 0: jdbc:drill:schema=dfs.tmp> {noformat} Data from the JSON file {noformat} 0: jdbc:drill:schema=dfs.tmp> select * from `rptd_count.json`; +---+-+ |arr| id | +---+-+ | [0,1,2,3,4,5,6,7,8,9,10] | 1 | | [0,1,2,3,4,5,6,7,8,9,10] | 2 | | [0,1,2,3,4,5,6,7,8,9,10] | 3 | | [0,1,2,3,4,5,6,7,8,9,10] | 4 | | [0,1,2,3,4,5,6,7,8,9,10] | 5 | | [0,1,2,3,4,5,6,7,8,9,10] | 6 | | [0,1,2,3,4,5,6,7,8,9,10] | 7 | | [0,1,2,3,4,5,6,7,8,9,10] | 8 | | [0,1,2,3,4,5,6,7,8,9,10] | 9 | | [0,1,2,3,4,5,6,7,8,9,10] | 10 | +---+-+ 10 rows selected (0.224 seconds) {noformat} Stack trace from drillbit.log {noformat} 2017-05-01 19:32:34,209 [26f872ad-62a3-d7a7-aec1-9c7d937a2416:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 26f872ad-62a3-d7a7-aec1-9c7d937a2416: select convert_to(t.arr,'UTF-8') c, id from `rptd_count.json` t ... 2017-05-01 19:32:34,328 [26f872ad-62a3-d7a7-aec1-9c7d937a2416:foreman] ERROR o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: StringIndexOutOfBoundsException: String index out of range: -3 [Error Id: 056a13c0-6c9f-403e-877e-040e907d6581 on centos-01.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: StringIndexOutOfBoundsException: String index out of range: -3 [Error Id: 056a13c0-6c9f-403e-877e-040e907d6581 on centos-01.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:847) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:977) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:297) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception during fragment initialization: String index out of range: -3 ... 4 common frames omitted Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -3 at java.lang.String.substring(String.java:1931) ~[na:1.8.0_91] at org.apache.drill.exec.planner.logical.PreProcessLogicalRel.getConvertFunctionException(PreProcessLogicalRel.java:244) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.planner.logical.PreProcessLogicalRel.visit(PreProcessLogicalRel.java:148) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.calcite.rel.logical.LogicalProject.accept(LogicalProject.java:129) ~[calcite-core-1.4.0-drill-r21.jar:1.4.0-drill-r21] at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.preprocessNode(DefaultSqlHandler.java:641) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:196) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:164)
[jira] [Created] (DRILL-5434) IllegalStateException: Memory was leaked by query.
Khurram Faraaz created DRILL-5434: - Summary: IllegalStateException: Memory was leaked by query. Key: DRILL-5434 URL: https://issues.apache.org/jira/browse/DRILL-5434 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Reporter: Khurram Faraaz Issue a long running COUNT query. While the query is being executed, stop the foreman drillbit, ./drillbit.sh stop A memory leak is reported in the drillbit.log Apache Drill 1.11.0 git.commit.id.abbrev=06e1522 Stack trace from drillbit.log {noformat} 2017-04-13 06:14:36,828 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 2710e8b2-d4dc-1bee-016e-b69fd4966916: SELECT COUNT(*) FROM `twoKeyJsn.json` 2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-04-13 06:14:36,930 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-04-13 06:14:36,930 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-04-13 06:14:36,930 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-04-13 06:14:36,932 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-04-13 06:14:36,934 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Time: 2ms total, 2.102992ms avg, 2ms max. 2017-04-13 06:14:36,934 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Earliest start: 0.555000 μs, Latest start: 0.555000 μs, Average start: 0.555000 μs . 2017-04-13 06:14:36,949 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0: State change requested AWAITING_ALLOCATION --> RUNNING 2017-04-13 06:14:36,949 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0: State to report: RUNNING Thu Apr 13 06:14:40 UTC 2017 Terminating drillbit pid 5107 2017-04-13 06:14:40,756 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Received shutdown request. 2017-04-13 06:14:47,819 [pool-169-thread-2] INFO o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup io.netty.channel.nio.NioEventLoopGroup@4f3ee67c in 1024 ms 2017-04-13 06:14:47,819 [pool-169-thread-2] INFO o.a.drill.exec.service.ServiceEngine - closed dataPool in 1024 ms 2017-04-13 06:14:49,806 [Drillbit-ShutdownHook#0] WARN o.apache.drill.exec.work.WorkManager - Closing WorkManager but there are 1 running fragments. 2017-04-13 06:14:49,807 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0: State change requested RUNNING --> FAILED 2017-04-13 06:14:49,807 [Drillbit-ShutdownHook#0] INFO o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 6964, cache miss count: 335, hit rate: 95% 2017-04-13 06:14:49,807 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0: State change requested FAILED --> FINISHED 2017-04-13 06:14:49,809 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException Fragment 0:0 [Error Id: 94817261-98a9-4153-8b3a-2d9c95d80cc1 on centos-01.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: NullPointerException Fragment 0:0 [Error Id: 94817261-98a9-4153-8b3a-2d9c95d80cc1 on centos-01.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[jira] [Created] (DRILL-5430) select on fastparquet created parquet file results in IOB Exception
Khurram Faraaz created DRILL-5430: - Summary: select on fastparquet created parquet file results in IOB Exception Key: DRILL-5430 URL: https://issues.apache.org/jira/browse/DRILL-5430 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Affects Versions: 1.11.0 Reporter: Khurram Faraaz Select on a parquet file created using python's fastparquet results in IOB Exception. Parquet file was created using fastparquet (python implementation of parquet format) Parquet schema information for parquet file used in test. {noformat} [root@centos-12q parquet-tools]# ./parquet-schema frmPandas_9.parquet message schema { required int32 c_int8_signed (INT_8); required int32 c_uint8 (UINT_8); } {noformat} Apache Drill 1.11.0 git.commit.id.abbrev=06e1522 Select on the parquet results in DATA_READ error, stack trace shows an IndexOutOfBoundsException {noformat} 0: jdbc:drill:schema=dfs.tmp> select * from `frmPandas_9.parquet`; Error: DATA_READ ERROR: Error reading from Parquet file File: /tmp/frmPandas_9.parquet Column: c_int8_signed Row Group Start: 4 Fragment 0:0 [Error Id: 32b4d0ec-4198-4b38-bdd7-a27c881072e6 on centos-01.qa.lab:31010] (state=,code=0) {noformat} Stack trace from drillbit.log {noformat} 2017-04-12 04:23:55,214 [27125424-657c-7f8f-c27b-4d1516e4bb97:frag:0:0] INFO o.a.d.e.s.p.c.ColumnReader - User Error Occurred: Error reading from Parquet file (srcIndex: 0) org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: Error reading from Parquet file File: /tmp/frmPandas_9.parquet Column: c_int8_signed Row Group Start: 4 [Error Id: 32b4d0ec-4198-4b38-bdd7-a27c881072e6 ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readValues(ColumnReader.java:151) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPageData(ColumnReader.java:199) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:179) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPages(ColumnReader.java:129) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFieldsSerial(ParquetRecordReader.java:512) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFields(ParquetRecordReader.java:505) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:590) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:179) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:232) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_91] at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_91] at
[jira] [Created] (DRILL-5410) Unicode characters in JSON data result in DATA_READ error.
Khurram Faraaz created DRILL-5410: - Summary: Unicode characters in JSON data result in DATA_READ error. Key: DRILL-5410 URL: https://issues.apache.org/jira/browse/DRILL-5410 Project: Apache Drill Issue Type: Bug Components: Storage - JSON Affects Versions: 1.11.0 Reporter: Khurram Faraaz Unicode characters in JSON data result in DATA_READ error. Apache Drill 1.11.0 commit ID : adbf363 {noformat} Data used in test. [root@centos-01 ~]# cat test.json "arr":[{u'\ue4ea\ue4ba': 5.167460680922489e+18}] apache drill 1.11.0-SNAPSHOT "a drill in the hand is better than two in the bush" 0: jdbc:drill:schema=dfs.tmp> alter session set `store.json.all_text_mode`=true; +---++ | ok | summary | +---++ | true | store.json.all_text_mode updated. | +---++ 1 row selected (0.156 seconds) 0: jdbc:drill:schema=dfs.tmp> select * from `test.json`; Error: DATA_READ ERROR: Failure while parsing JSON. Found token of [VALUE_STRING]. Drill currently only supports parsing json strings that contain either lists or maps. The root object cannot be a scalar. Line 1 Column 3 Field Fragment 0:0 [Error Id: 535e08fb-89b0-443b-86dc-032e40bbb4e1 on centos-01.qa.lab:31010] (state=,code=0) {noformat} {noformat} 2017-04-03 09:42:45,845 [271de6e9-e9f3-51bc-7091-1a2df15f12a7:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 271de6e9-e9f3-51bc-7091-1a2df15f12a7: select * from `test.json` ... 2017-04-03 09:42:45,935 [271de6e9-e9f3-51bc-7091-1a2df15f12a7:frag:0:0] INFO o.a.d.e.vector.complex.fn.JsonReader - User Error Occurred: Failure while parsing JSON. Found token of [VALUE_STRING]. Drill currently only supports parsing json strings that contain either lists or maps. The root object cannot be a scalar. org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: Failure while parsing JSON. Found token of [VALUE_STRING]. Drill currently only supports parsing json strings that contain either lists or maps. The root object cannot be a scalar. Line 1 Column 3 Field [Error Id: 535e08fb-89b0-443b-86dc-032e40bbb4e1 ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.vector.complex.fn.JsonReader.writeToVector(JsonReader.java:305) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.vector.complex.fn.JsonReader.write(JsonReader.java:217) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.store.easy.json.JSONRecordReader.next(JSONRecordReader.java:206) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:179) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:232) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_91] at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_91] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) [hadoop-common-2.7.0-mapr-1607.jar:na] at
[jira] [Created] (DRILL-5401) wrong results - IS NULL / IS NOT NULL over a list in JSON
Khurram Faraaz created DRILL-5401: - Summary: wrong results - IS NULL / IS NOT NULL over a list in JSON Key: DRILL-5401 URL: https://issues.apache.org/jira/browse/DRILL-5401 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Reporter: Khurram Faraaz Checking if a list is null or if it is not null, results in incorrect results. Drill 1.11.0 commit id: adbf363d Data used in test {noformat} [root@centos-01 ~]# cat empty_array.json { "a": { "b": { "c": [ { "d": { "e": "f" } } ] } } } { "a": { "b": { "c": [] } } } {noformat} {noformat} 0: jdbc:drill:schema=dfs.tmp> alter session set `store.json.all_text_mode`=true; +---++ | ok | summary | +---++ | true | store.json.all_text_mode updated. | +---++ 1 row selected (0.189 seconds) 0: jdbc:drill:schema=dfs.tmp> select * from `empty_array.json`; ++ | a| ++ | {"b":{"c":[{"d":{"e":"f"}}]}} | | {"b":{"c":[]}} | ++ 2 rows selected (0.138 seconds) /* wrong results */ 0: jdbc:drill:schema=dfs.tmp> select * from `empty_array.json` t where t.b.c IS NULL; ++ | a| ++ | {"b":{"c":[{"d":{"e":"f"}}]}} | | {"b":{"c":[]}} | ++ 2 rows selected (0.152 seconds) /* wrong results */ 0: jdbc:drill:schema=dfs.tmp> select * from `empty_array.json` t where t.b.c IS NOT NULL; ++ | a | ++ ++ No rows selected (0.154 seconds) {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5392) implement castINT(BIT-REQUIRED) and castINT(BIT-OPTIONAL)
Khurram Faraaz created DRILL-5392: - Summary: implement castINT(BIT-REQUIRED) and castINT(BIT-OPTIONAL) Key: DRILL-5392 URL: https://issues.apache.org/jira/browse/DRILL-5392 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Reporter: Khurram Faraaz We need to implement castINT(BIT-REQUIRED) and castINT(BIT-OPTIONAL) Below SQL fails on Drill 1.11.0 {noformat} 0: jdbc:drill:schema=dfs.tmp> SELECT res, CASE res WHEN true THEN CAST('1990-10-10 22:40:50' AS TIMESTAMP) ELSE null END . . . . . . . . . . . . . . > FROM . . . . . . . . . . . . . . > ( . . . . . . . . . . . . . . > SELECT . . . . . . . . . . . . . . > (CASE WHEN (true) THEN 1 ELSE null END) res . . . . . . . . . . . . . . > FROM (values(1)) foo . . . . . . . . . . . . . . > ) foobar; Error: PLAN ERROR: Failure while materializing expression in constant expression evaluator [CASE(=(CASE(true, 1, null), true), CAST('1990-10-10 22:40:50'):TIMESTAMP(0) NOT NULL, null)]. Errors: Error in expression at index -1. Error: Missing function implementation: [castINT(BIT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--. [Error Id: 1c0c850f-99b7-4d64-a0e3-d6ee70b11259 on centos-01.qa.lab:31010] (state=,code=0) {noformat} Stack trace from drillbit.log {noformat} 2017-03-28 09:50:54,500 [2725ce01-529d-3479-7467-ed4aa20d3ca5:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 2725ce01-529d-3479-7467-ed4aa20d3ca5: SELECT res, CASE res WHEN true THEN CAST('1990-10-10 22:40:50' AS TIMESTAMP) ELSE null END FROM ( SELECT (CASE WHEN (true) THEN 1 ELSE null END) res FROM (values(1)) foo ) foobar 2017-03-28 09:50:54,585 [2725ce01-529d-3479-7467-ed4aa20d3ca5:foreman] INFO o.a.d.e.p.logical.DrillConstExecutor - User Error Occurred: Failure while materializing expression in constant expression evaluator [CASE(=(CASE(true, 1, null), true), CAST('1990-10-10 22:40:50'):TIMESTAMP(0) NOT NULL, null)]. Errors: Error in expression at index -1. Error: Missing function implementation: [castINT(BIT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--. org.apache.drill.common.exceptions.UserException: PLAN ERROR: Failure while materializing expression in constant expression evaluator [CASE(=(CASE(true, 1, null), true), CAST('1990-10-10 22:40:50'):TIMESTAMP(0) NOT NULL, null)]. Errors: Error in expression at index -1. Error: Missing function implementation: [castINT(BIT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--. [Error Id: 1c0c850f-99b7-4d64-a0e3-d6ee70b11259 ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.planner.logical.DrillConstExecutor.reduce(DrillConstExecutor.java:131) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressions(ReduceExpressionsRule.java:499) [calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] at org.apache.calcite.rel.rules.ReduceExpressionsRule$1.onMatch(ReduceExpressionsRule.java:241) [calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228) [calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808) [calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) [calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:404) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:343) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:240) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:290) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:168) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:131) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:79) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1050)
[jira] [Created] (DRILL-5382) Error: Missing function implementation: [isnotnull(MAP-REQUIRED)]
Khurram Faraaz created DRILL-5382: - Summary: Error: Missing function implementation: [isnotnull(MAP-REQUIRED)] Key: DRILL-5382 URL: https://issues.apache.org/jira/browse/DRILL-5382 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.11.0 Reporter: Khurram Faraaz Projecting a map from JSON data and filter non null values results in SchemaChangeException, Error: Missing function implementation: [isnotnull(MAP-REQUIRED)]. Data used in test is available here - wget http://data.githubarchive.org/2015-01-01-15.json.gz Drill 1.11.0 commit id: adbf363 {noformat} 0: jdbc:drill:schema=dfs.tmp> select org from `2015-01-01-15.json` where org is not null; Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to materialize incoming schema. Errors: Error in expression at index -1. Error: Missing function implementation: [isnotnull(MAP-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.. Fragment 0:0 [Error Id: 3a776c68-6476-4bd8-a9e6-928bfc2ef5bd on centos-01.qa.lab:31010] (state=,code=0) 0: jdbc:drill:schema=dfs.tmp> {noformat} Select without IS NOT NULL filter, returns correct results. {noformat} apache drill 1.11.0-SNAPSHOT "just drill it" 0: jdbc:drill:schema=dfs.tmp> select org from `2015-01-01-15.json`; +-+ | org | +-+ | {} | | {} | | {} | | {"id":9285252,"login":"visionmedia","gravatar_id":"","url":"https://api.github.com/orgs/visionmedia","avatar_url":"https://avatars.githubusercontent.com/u/9285252?"} | | {} | | {} | | {} | ... ... | {} | | {} | | {} | | {"id":9216151,"login":"osp","gravatar_id":"","url":"https://api.github.com/orgs/osp","avatar_url":"https://avatars.githubusercontent.com/u/9216151?"} | | {} | | {"id":296074,"login":"zendframework","gravatar_id":"","url":"https://api.github.com/orgs/zendframework","avatar_url":"https://avatars.githubusercontent.com/u/296074?"} | | {} | | {} | | {} | | {} | | {"id":9216151,"login":"osp","gravatar_id":"","url":"https://api.github.com/orgs/osp","avatar_url":"https://avatars.githubusercontent.com/u/9216151?"} | | {} | | {"id":5822862,"login":"SuaveIO","gravatar_id":"","url":"https://api.github.com/orgs/SuaveIO","avatar_url":"https://avatars.githubusercontent.com/u/5822862?"} | | {"id":5822862,"login":"SuaveIO","gravatar_id":"","url":"https://api.github.com/orgs/SuaveIO","avatar_url":"https://avatars.githubusercontent.com/u/5822862?"} | | {} | | {"id":5822862,"login":"SuaveIO","gravatar_id":"","url":"https://api.github.com/orgs/SuaveIO","avatar_url":"https://avatars.githubusercontent.com/u/5822862?"} | | {"id":2918581,"login":"twbs","gravatar_id":"","url":"https://api.github.com/orgs/twbs","avatar_url":"https://avatars.githubusercontent.com/u/2918581?"} | | {"id":1104713,"login":"s9y","gravatar_id":"","url":"https://api.github.com/orgs/s9y","avatar_url":"https://avatars.githubusercontent.com/u/1104713?"} | | {} | | {} | | {} | | {} | +-+ 11,351 rows selected (0.865 seconds) {noformat} Stack trace from drillbit.log {noformat} Caused by: org.apache.drill.exec.exception.SchemaChangeException: Failure while trying to materialize incoming schema. Errors: Error in expression at index -1. Error: Missing function implementation: [isnotnull(MAP-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.. at org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.generateSV2Filterer(FilterRecordBatch.java:186) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.setupNewSchema(FilterRecordBatch.java:111) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:78) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:93) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[jira] [Created] (DRILL-5367) Join query returns wrong results
Khurram Faraaz created DRILL-5367: - Summary: Join query returns wrong results Key: DRILL-5367 URL: https://issues.apache.org/jira/browse/DRILL-5367 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.10.0 Environment: 3 node cluster Reporter: Khurram Faraaz Join query returns wrong results Drill 1.10.0 does not return any results. {noformat} 0: jdbc:drill:schema=dfs.tmp> SELECT * FROM using_f1 JOIN (SELECT * FROM using_f2) foo USING(col_prime); +-+++-+-+---+--+-+-+--+--++ | col_dt | col_state | col_prime | col_varstr | col_id | col_name | col_dt0 | col_state0 | col_prime0 | col_varstr0 | col_id0 | col_name0 | +-+++-+-+---+--+-+-+--+--++ +-+++-+-+---+--+-+-+--+--++ No rows selected (0.314 seconds) {noformat} {noformat} Explain plan for above failing query 0: jdbc:drill:schema=dfs.tmp> explain plan for SELECT * FROM using_f1 JOIN (SELECT * FROM using_f2) foo USING(col_prime); +--+--+ | text | json | +--+--+ | 00-00Screen 00-01 ProjectAllowDup(*=[$0], *0=[$1]) 00-02Project(T49¦¦*=[$0], T48¦¦*=[$2]) 00-03 Project(T49¦¦*=[$1], col_prime=[$2], T48¦¦*=[$0]) 00-04HashJoin(condition=[=($2, $0)], joinType=[inner]) 00-06 Project(T48¦¦*=[$0]) 00-08Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:///tmp/using_f2]], selectionRoot=maprfs:/tmp/using_f2, numFiles=1, usedMetadataFile=false, columns=[`*`]]]) 00-05 Project(T49¦¦*=[$0], col_prime=[$1]) 00-07Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:///tmp/using_f1]], selectionRoot=maprfs:/tmp/using_f1, numFiles=1, usedMetadataFile=false, columns=[`*`]]]) {noformat} Whereas Postgres 9.3 returns expected results for the same data. {noformat} postgres=# SELECT * FROM using_f1 JOIN (SELECT * FROM using_f2) foo USING(col_prime); col_prime | col_dt | col_state | col_varstr | col_id | col_name | col_dt | col_state | col_varstr| col_id | col_name ---++---+--- ++---++---+- ++-- 103 | 2014-12-24 | TX| LUW2QzWGdJfnxHrqm3vwyndzRBFwH8l5xVDaM3hTiZAanp j | 19462 | Julie Lennox | 1990-01-11 | WV| KKzEOgle6E5h NANduNAAIp9DQnGLGxO | 54217 | Derek Wilson 103 | 1985-07-18 | CA| aYQ2uLpPxebGGRvcX0fahrAOO4yhkDRvMPES6PuYsIfwkU Mrcq6NSdt0j | 48987 | Lillian Lupo | 1990-01-11 | WV| KKzEOgle6E5h NANduNAAIp9DQnGLGxO | 54217 | Derek Wilson 103 | 1988-02-27 | SC| OcVKheHMyeKLgcvamrJHUxKyCGGJGci3Y9ht2LI9T5Ek1n wckB| 52840 | Martha Rose | 1990-01-11 | WV| KKzEOgle6E5h NANduNAAIp9DQnGLGxO | 54217 | Derek Wilson 211 | 1989-12-06 | SD| HHlmvV4 | 1131 | Kenneth Hayes | 1989-05-31 | MT| yhHfCGaCqnAr XUCD4jRoZQ4fj6IQIKZHUGLlIsSr1L7voCE3lEmj3DOSFqJ0Kq | 49191 | Joan Stein 43 | 2006-01-24 | NV| EJAN2JjRqoQWgp7rHLT1yPMBR50g1Kil3klu1vPritFKB2 5EjmL1tLXleagAP | 30179 | William Strassel | 2006-03-02 | MI| W9G0nWo8QNtH r9YxOscigPbtXEtNPZ | 44849 | Catherine Turner 193 | 1990-01-14 | NV| 9nd3po1bnyasqINVA | 47775 | James Walters ... 1990-01-14 | NV| 9nd3po1bnyasqINVA | 47775 | James Walters | 1980-04-22 | ID| jR8jr1lqDprU FPhAX4xZnulndYNd3 | 5876 | Rosie Johnson 5 | 2004-01-27 | KS| 0A8Gwqm66k6wQ1KzcUdSQKZU3AZtPImxb8 | 57787 | Dean Salazar | 1997-09-13 | SC| uq35Sqf1GfPt IV1mE2CzwxKaX | 17041 | Dorothy Paulsen 5 | 1999-07-12 | UT| hQk9DBx1egLNIpi9btvN7GPewgvPROWaNArsxAbRILW3dN fwi526 | 38130 | Beverly Flores| 1997-09-13 | SC| uq35Sqf1GfPt IV1mE2CzwxKaX | 17041 | Dorothy Paulsen (239 rows) {noformat} -- This message was sent by Atlassian JIRA
[jira] [Created] (DRILL-5357) Partition pruning information not available in query plan for COUNT aggregate query
Khurram Faraaz created DRILL-5357: - Summary: Partition pruning information not available in query plan for COUNT aggregate query Key: DRILL-5357 URL: https://issues.apache.org/jira/browse/DRILL-5357 Project: Apache Drill Issue Type: Bug Components: Query Planning & Optimization Affects Versions: 1.10.0 Environment: 3 node CentOS cluster Reporter: Khurram Faraaz Priority: Critical We are not seeing partition pruning information in the query plan for the below, COUNT(*) and COUNT() query ? Drill 1.10.0-SNAPSHOT git commit id: b657d44f parquet table has 6 columns total number of rows = 1638640 {noformat} 0: jdbc:drill:schema=dfs.tmp> CREATE TABLE tbl_prtn_prune_01 PARTITION BY (col_state) AS SELECT CAST(columns[0] AS DATE) col_date, CAST(columns[1] AS CHAR(3)) col_state, CAST(columns[2] AS INTEGER) col_prime, CAST(columns[3] AS VARCHAR(256)) col_varstr, CAST(columns[4] AS INTEGER) col_id, CAST(columns[5] AS VARCHAR(50)) col_name from `partition_prune_data.csv`; +---++ | Fragment | Number of records written | +---++ | 0_0 | 1638640| +---++ 1 row selected (17.675 seconds) 0: jdbc:drill:schema=dfs.tmp> select COUNT(*) from tbl_prtn_prune_01 where col_state = 'CA'; +-+ | EXPR$0 | +-+ | 35653 | +-+ 1 row selected (0.471 seconds) 0: jdbc:drill:schema=dfs.tmp> explain plan for select COUNT(*) from tbl_prtn_prune_01 where col_state = 'CA'; +--+--+ | text | json | +--+--+ | 00-00Screen 00-01 Project(EXPR$0=[$0]) 00-02Project(EXPR$0=[$0]) 00-03 Scan(groupscan=[org.apache.drill.exec.store.pojo.PojoRecordReader@1d4bb67d[columns = null, isStarQuery = false, isSkipQuery = false]]) {noformat} And then I did a REFRESH TABLE METADATA on the parquet table {noformat} 0: jdbc:drill:schema=dfs.tmp> refresh table metadata tbl_prtn_prune_01; +---+-+ | ok | summary | +---+-+ | true | Successfully updated metadata for table tbl_prtn_prune_01. | +---+-+ 1 row selected (0.321 seconds) 0: jdbc:drill:schema=dfs.tmp> explain plan for select COUNT(col_state) from tbl_prtn_prune_01 where col_state = 'CA'; +--+--+ | text | json | +--+--+ | 00-00Screen 00-01 Project(EXPR$0=[$0]) 00-02Project(EXPR$0=[$0]) 00-03 Scan(groupscan=[org.apache.drill.exec.store.pojo.PojoRecordReader@2e0f4be9[columns = null, isStarQuery = false, isSkipQuery = false]]) 0: jdbc:drill:schema=dfs.tmp> explain plan for select COUNT(*) from tbl_prtn_prune_01 where col_state = 'CA'; +--+--+ | text | json | +--+--+ | 00-00Screen 00-01 Project(EXPR$0=[$0]) 00-02Project(EXPR$0=[$0]) 00-03 Scan(groupscan=[org.apache.drill.exec.store.pojo.PojoRecordReader@3fc1f8e7[columns = null, isStarQuery = false, isSkipQuery = false]]) 0: jdbc:drill:schema=dfs.tmp> explain plan for select COUNT(col_date) from tbl_prtn_prune_01 where col_state = 'CA'; +--+--+ | text | json | +--+--+ | 00-00Screen 00-01 Project(EXPR$0=[$0]) 00-02Project(EXPR$0=[$0]) 00-03 Scan(groupscan=[org.apache.drill.exec.store.pojo.PojoRecordReader@7afc851e[columns = null, isStarQuery = false, isSkipQuery = false]]) {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5347) Join query returns incorrect results, returns an extra column.
Khurram Faraaz created DRILL-5347: - Summary: Join query returns incorrect results, returns an extra column. Key: DRILL-5347 URL: https://issues.apache.org/jira/browse/DRILL-5347 Project: Apache Drill Issue Type: Bug Components: Query Planning & Optimization Affects Versions: 1.10.0 Reporter: Khurram Faraaz Priority: Critical Join query returns incorrect results, returns an extra column. Note that the columns in both the parquet files are in different order. Postgres returns correct results. {noformat} postgres=# create table using_l1(c1 int, c2 varchar(25), c3 date); CREATE TABLE postgres=# create table using_r1(c2 varchar(25), c3 date, c1 int); CREATE TABLE postgres=# copy using_l1 from '/root/using_clause/using_l1.txt' DELIMITER ',' CSV; COPY 17 postgres=# copy using_r1 from '/root/using_clause/using_r1.txt' DELIMITER ',' CSV; COPY 18 postgres=# select * from using_l1 JOIN using_r1 USING(c1); c1 |c2 | c3 | c2 | c3 ---+---+++ 1 | test | 1982-09-11 | teststring | 2009-10-10 1 | test | 1982-09-11 | yes| 2001-09-09 2 | string| 1989-04-12 | never | 1989-11-11 3 | foo | 2000-10-12 | none | 1967-03-05 3 | foo | 2000-10-12 | here | 2011-09-09 4 | foobar| 2015-01-07 | champ | 2008-06-07 5 | bar | 2004-05-08 | bar| 2010-02-04 17 | barfoobar | 1992-12-06 | hello | 2014-07-02 19 | hello | 1980-10-10 | hi | 2000-01-01 0 | now | 2007-07-02 | foobar | 2011-01-01 10 | here | 2009-11-11 | done | 1947-08-14 9 | there | 1993-09-03 | there | 1993-09-03 22 | hi| 2000-01-04 | there | 2016-03-22 13 | no| 2016-10-01 | foo| 1982-10-11 13 | no| 2016-10-01 | no | 1992-12-06 (15 rows) {noformat} Results from Drill 1.10.0 (3dfb4972), for the same input data we see an extra column c10 in the results of join query. {noformat} ## using_l1 => (c1 INTEGER, c2 VARCHAR(25), c3 DATE) 0: jdbc:drill:schema=dfs.tmp> CREATE TABLE using_l1 AS SELECT CAST(columns[0] AS INTEGER) c1, CAST(columns[1] AS VARCHAR(25)) c2, CAST(columns[2] AS DATE) c3 FROM `using_l1.txt`; +---++ | Fragment | Number of records written | +---++ | 0_0 | 17 | +---++ 1 row selected (0.258 seconds) ## using_r1 => (c2 VARCHAR, c3 DATE, c1 INTEGER) 0: jdbc:drill:schema=dfs.tmp> CREATE TABLE using_r1 AS SELECT CAST(columns[0] AS VARCHAR(25)) c2, CAST(columns[1] AS DATE) c3, CAST(columns[2] AS INTEGER) c1 FROM `using_r1.txt`; +---++ | Fragment | Number of records written | +---++ | 0_0 | 18 | +---++ 1 row selected (0.249 seconds) 0: jdbc:drill:schema=dfs.tmp> ## Last column c10 is un-expected in the results of the JOIN 0: jdbc:drill:schema=dfs.tmp> select * from using_l1 JOIN using_r1 USING(c1); +++-+-+-++ | c1 | c2 | c3 | c20 | c30 | c10 | +++-+-+-++ | 1 | test | 1982-09-11 | yes | 2001-09-09 | 1 | | 1 | test | 1982-09-11 | teststring | 2009-10-10 | 1 | | 2 | string | 1989-04-12 | never | 1989-11-11 | 2 | | 3 | foo| 2000-10-12 | here| 2011-09-09 | 3 | | 3 | foo| 2000-10-12 | none| 1967-03-05 | 3 | | 4 | foobar | 2015-01-07 | champ | 2008-06-07 | 4 | | 5 | bar| 2004-05-08 | bar | 2010-02-04 | 5 | | 17 | barfoobar | 1992-12-06 | hello | 2014-07-02 | 17 | | 19 | hello | 1980-10-10 | hi | 2000-01-01 | 19 | | 0 | now| 2007-07-02 | foobar | 2011-01-01 | 0 | | 10 | here | 2009-11-11 | done| 1947-08-14 | 10 | | 9 | there | 1993-09-03 | there | 1993-09-03 | 9 | | 22 | hi | 2000-01-04 | there | 2016-03-22 | 22 | | 13 | no | 2016-10-01 | no | 1992-12-06 | 13 | | 13 | no | 2016-10-01 | foo | 1982-10-11 | 13 | +++-+-+-++ 15 rows selected (0.358 seconds) {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5345) USING() should return error, when column in not present in the right table
Khurram Faraaz created DRILL-5345: - Summary: USING() should return error, when column in not present in the right table Key: DRILL-5345 URL: https://issues.apache.org/jira/browse/DRILL-5345 Project: Apache Drill Issue Type: Bug Components: Query Planning & Optimization Affects Versions: 1.10.0 Reporter: Khurram Faraaz When the column used in, USING(column) is not present in the right table of the JOIN, we show report an error to user instead of returning zero rows. {noformat} ::= USING ::= ::= [ { }... ] ::= {noformat} Postgres 9.3 reports an error for such a scenario, because column c1 is not present in the other table in the join. {noformat} postgres=# select * from tbl_l JOIN tbl_r USING (c1); ERROR: column "c1" specified in USING clause does not exist in right table {noformat} Drill 1.10.0 returns zero rows in one case and RuntimeException in the other case. In both cases Drill should return a meaning error message, like the one reported by Postgres. {noformat} 0: jdbc:drill:schema=dfs.tmp> create table tbl_l as select cast(columns[0] as integer) c1, cast(columns[1] as varchar(25)) c2 from `tbl_l.txt`; +---++ | Fragment | Number of records written | +---++ | 0_0 | 10 | +---++ 1 row selected (0.197 seconds) 0: jdbc:drill:schema=dfs.tmp> create table tbl_r as select cast(columns[0] as integer) c3, cast(columns[1] as varchar(25)) c4 from `tbl_r.txt`; +---++ | Fragment | Number of records written | +---++ | 0_0 | 10 | +---++ 1 row selected (0.221 seconds) ## this is not right, we should report error for below query. 0: jdbc:drill:schema=dfs.tmp> select * from tbl_l JOIN tbl_r USING (c1); +-+-+-+-+ | c1 | c2 | c3 | c4 | +-+-+-+-+ +-+-+-+-+ No rows selected (0.222 seconds) ## we should return a meaningful error message 0: jdbc:drill:schema=dfs.tmp> select * from tbl_l JOIN tbl_r USING (c2); Error: SYSTEM ERROR: DrillRuntimeException: Join only supports implicit casts between 1. Numeric data 2. Varchar, Varbinary data 3. Date, Timestamp data Left type: INT, Right type: VARCHAR. Add explicit casts to avoid this error Fragment 0:0 [Error Id: 56107e6e-54b5-4905-9aa4-b0af2aea4dd9 on centos-01.qa.lab:31010] (state=,code=0) {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5336) Columns returned by select over CTAS created parquet are not in correct order.
Khurram Faraaz created DRILL-5336: - Summary: Columns returned by select over CTAS created parquet are not in correct order. Key: DRILL-5336 URL: https://issues.apache.org/jira/browse/DRILL-5336 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Affects Versions: 1.10.0 Reporter: Khurram Faraaz The ordering of the columns in the result of the SELECT over CTAS created parquet file is not right. Column col_int should be present before col_chr, however col_chr appears before col_int, in the result of select. Note that there is a UNION ALL in the CTAS's SELECT statement. And each of the select statements in the UNION ALL has an ORDER BY The problem seems to be with the use of LIMIT clause in the SELECT on dfs.tmp.temp_tbl_unall. Here is the parquet schema for the CTAS created parquet file. Note that col_int appears before col_chr in the parquet schema too. {noformat} [root@centos-01 parquet-tools]# hadoop fs -get /tmp/temp_tbl_unall/0_0_0.parquet . [root@centos-01 parquet-tools]# ./parquet-schema 0_0_0.parquet message root { optional int32 col_int; optional binary col_chr (UTF8); optional binary col_vrchr1 (UTF8); optional binary col_vrchr2 (UTF8); } {noformat} Drill 1.10.0 git commit id : 3dfb4972 {noformat} 0: jdbc:drill:schema=dfs.tmp> CREATE TABLE dfs.tmp.temp_tbl_unall as ( SELECT col_int, col_chr, col_vrchr1, col_vrchr2 FROM typeall_l order by col_int ) UNION ALL ( SELECT col_int, col_chr, col_vrchr1, col_vrchr2 FROM typeall_r order by col_int ); +---++ | Fragment | Number of records written | +---++ | 0_0 | 1107 | +---++ 1 row selected (0.381 seconds) 0: jdbc:drill:schema=dfs.tmp> SELECT * FROM dfs.tmp.temp_tbl_unall ORDER BY col_int LIMIT 100; +-+-+++ | col_chr | col_int | col_vrchr1 | col_vrchr2 | +-+-+++ | MI | 0 | Felecia Gourd | NLBQMg9 | | DE | 1 | Alvina Jenkins | f9MqJlnNettlCVGcShifgMgnzL5FrZmHysoMBe6kDtA | | HI | 1 | Fredrick Vanderburg | eN3CNLW8FE5voAksuJCSYnMdJrVown7my6DiAlI8KhrG69kQoAxKFJmOHPVca1FjGyHWd5Ag53vvODvKB8YwqXcbDihjR0DDbed1cgs7L1tndiPRvU1OreN5ByB8pF0QisgwSBWRKRvS8RVOzA3CyxOpjyxVujRLLlctww0jWwn09m3iINTi6Delw | | CA | 19 | John Doe | test string | | CA | 19 | John Doe | test string | ... | LA | 6854 | William Burk | 5krBT7wj8BkoiRUWV9HjkyIT1DRpPj6bNixK15g4gs9IEsKc5myCyzMKQk5k1 | ++--+---+-+ | col_chr | col_int | col_vrchr1 | col_vrchr2 | ++--+---+-+ | IN | 6870 | Caroline Bell | M2811poVmVJLuxqsHz0jzRSGrAJDXfl3UuE0Iz8ldqvRURURvq2dO4Q1358eiureI20NCGBl9lBpoKPc78TWS0gsWhIt280E8JZPQpj7lOJXnHUmvydDiBPgAzNoGn7SSP6xYlnMyBhvWRxB5NF3I9vszosjmpW1Yx7et56QvwLfWBb3unJPnrxVYXX5tAfeyednJ4A90aOE2dhMXy1wLwewMJ91SWBEUM8TU3aGikQ5Ax6dDhDBQLaP | ++--+---+-+ 100 rows selected (0.173 seconds) {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (DRILL-5278) CTTAS store.format=csv, returns one extra record
[ https://issues.apache.org/jira/browse/DRILL-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz resolved DRILL-5278. --- Resolution: Not A Bug > CTTAS store.format=csv, returns one extra record > > > Key: DRILL-5278 > URL: https://issues.apache.org/jira/browse/DRILL-5278 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Text & CSV >Affects Versions: 1.10.0 >Reporter: Khurram Faraaz > > When store.format = csv, we see incorrect results returned, an extra record > is returned, as compared to when store.format = parquet. The difference is > seen when doing a count over a temporary table created in Drill. > Drill 1.10.0 git commit ID : 300e9349 > Steps to reproduce the problem. > {noformat} > 0: jdbc:drill:schema=dfs.tmp> reset all; > +---+---+ > | ok |summary| > +---+---+ > | true | ALL updated. | > +---+---+ > 1 row selected (0.279 seconds) > 0: jdbc:drill:schema=dfs.tmp> ALTER SESSION SET `store.format`='csv'; > +---++ > | ok |summary | > +---++ > | true | store.format updated. | > +---++ > 1 row selected (0.21 seconds) > 0: jdbc:drill:schema=dfs.tmp> CREATE TEMPORARY TABLE dfs.tmp.temp_tbl > . . . . . . . . . . . . . . > AS > . . . . . . . . . . . . . . > SELECT * FROM typeall_l; > +---++ > | Fragment | Number of records written | > +---++ > | 0_0 | 105| > +---++ > 1 row selected (0.233 seconds) > 0: jdbc:drill:schema=dfs.tmp> SELECT COUNT(*) FROM dfs.tmp.temp_tbl; > +-+ > | EXPR$0 | > +-+ > | 106 | > +-+ > 1 row selected (0.189 seconds) > 0: jdbc:drill:schema=dfs.tmp> drop table dfs.tmp.temp_tbl; > +---+-+ > | ok | summary | > +---+-+ > | true | Temporary table [temp_tbl] dropped | > +---+-+ > 1 row selected (0.186 seconds) > 0: jdbc:drill:schema=dfs.tmp> ALTER SESSION SET `store.format`='parquet'; > +---++ > | ok |summary | > +---++ > | true | store.format updated. | > +---++ > 1 row selected (0.196 seconds) > 0: jdbc:drill:schema=dfs.tmp> CREATE TEMPORARY TABLE dfs.tmp.temp_tbl > . . . . . . . . . . . . . . > AS > . . . . . . . . . . . . . . > SELECT * FROM typeall_l; > +---++ > | Fragment | Number of records written | > +---++ > | 0_0 | 105| > +---++ > 1 row selected (0.263 seconds) > 0: jdbc:drill:schema=dfs.tmp> SELECT COUNT(*) FROM dfs.tmp.temp_tbl; > +-+ > | EXPR$0 | > +-+ > | 105 | > +-+ > 1 row selected (0.169 seconds) > 0: jdbc:drill:schema=dfs.tmp> drop table dfs.tmp.temp_tbl; > +---+-+ > | ok | summary | > +---+-+ > | true | Temporary table [temp_tbl] dropped | > +---+-+ > 1 row selected (0.165 seconds) > 0: jdbc:drill:schema=dfs.tmp> > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (DRILL-5298) CTAS with 0 records from a SELECT query should create the table with metadata
[ https://issues.apache.org/jira/browse/DRILL-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz resolved DRILL-5298. --- Resolution: Duplicate > CTAS with 0 records from a SELECT query should create the table with metadata > - > > Key: DRILL-5298 > URL: https://issues.apache.org/jira/browse/DRILL-5298 > Project: Apache Drill > Issue Type: Bug > Components: Metadata, Query Planning & Optimization >Affects Versions: 1.9.0 > Environment: MapR 5.2 >Reporter: Senthilkumar > Fix For: 1.9.0 > > > Hello team, > I create a table in Drill using CTAS as > CREATE TABLE CTAS_TEST AS SELECT * FROM `hive.default`.`test` WHERE 1 = 0 > It runs successfully. > But the table is not getting created as there are 0 records getting returned > from the SELECT query. > CTAS should still go ahead and create the table with the column metadata. > When BI tools fire up multi-pass queries, with CTAS in the first query, the > subsequent queries fail because of a missing table. > In databases like SQL Server, Postgres, CTAS will create the table, even if > the SELECT doesnt return any rows. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5288) JSON data not read as string when store.json.all_text_mode=true
Khurram Faraaz created DRILL-5288: - Summary: JSON data not read as string when store.json.all_text_mode=true Key: DRILL-5288 URL: https://issues.apache.org/jira/browse/DRILL-5288 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Reporter: Khurram Faraaz Setting all text mode to true (at system/session level) does not help in this case. Data from the JSON input file is not returned as string. Drill 1.10.0 git commit id: 300e9349 Data used in test {noformat} [root@centos-01 ~]# cat f2.json {"key":"string", "key":123, "key":[1,2,3], "key":true, "key":false, "key":null, "key":{"key2":"b"}, "key":"2011-08-21"} [root@centos-01 ~]# {noformat} {noformat} 0: jdbc:drill:schema=dfs.tmp> alter session set `store.json.all_text_mode`=true; +---++ | ok | summary | +---++ | true | store.json.all_text_mode updated. | +---++ 1 row selected (0.176 seconds) 0: jdbc:drill:schema=dfs.tmp> select key from `f2.json`; Error: SYSTEM ERROR: IllegalStateException: You tried to start when you are using a ValueWriter of type NullableVarCharWriterImpl. Fragment 0:0 [Error Id: 3cb5b806-53b6-49bf-ab5e-59c666259463 on centos-01.qa.lab:31010] (state=,code=0) {noformat} stack trace from drillbit.log {noformat} [Error Id: 3cb5b806-53b6-49bf-ab5e-59c666259463 on centos-01.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: You tried to start when you are using a ValueWriter of type NullableVarCharWriterImpl. Fragment 0:0 [Error Id: 3cb5b806-53b6-49bf-ab5e-59c666259463 on centos-01.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) ~[drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293) [drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262) [drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] Caused by: java.lang.IllegalStateException: You tried to start when you are using a ValueWriter of type NullableVarCharWriterImpl. at org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.startList(AbstractFieldWriter.java:108) ~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.vector.complex.impl.NullableVarCharWriterImpl.startList(NullableVarCharWriterImpl.java:88) ~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.vector.complex.fn.JsonReader.writeDataAllText(JsonReader.java:621) ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.vector.complex.fn.JsonReader.writeDataAllText(JsonReader.java:466) ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.vector.complex.fn.JsonReader.writeDataSwitch(JsonReader.java:319) ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.vector.complex.fn.JsonReader.writeToVector(JsonReader.java:262) ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.vector.complex.fn.JsonReader.write(JsonReader.java:217) ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.store.easy.json.JSONRecordReader.next(JSONRecordReader.java:206) ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:179) ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at
[jira] [Created] (DRILL-5278) CTTAS store.format=csv, returns one extra record
Khurram Faraaz created DRILL-5278: - Summary: CTTAS store.format=csv, returns one extra record Key: DRILL-5278 URL: https://issues.apache.org/jira/browse/DRILL-5278 Project: Apache Drill Issue Type: Bug Components: Storage - Text & CSV Affects Versions: 1.10.0 Reporter: Khurram Faraaz When store.format = csv, we see incorrect results returned, an extra record is returned, as compared to when store.format = parquet. The difference is seen when doing a count over a temporary table created in Drill. Drill 1.10.0 git commit ID : 300e9349 Steps to reproduce the problem. {noformat} 0: jdbc:drill:schema=dfs.tmp> reset all; +---+---+ | ok |summary| +---+---+ | true | ALL updated. | +---+---+ 1 row selected (0.279 seconds) 0: jdbc:drill:schema=dfs.tmp> ALTER SESSION SET `store.format`='csv'; +---++ | ok |summary | +---++ | true | store.format updated. | +---++ 1 row selected (0.21 seconds) 0: jdbc:drill:schema=dfs.tmp> CREATE TEMPORARY TABLE dfs.tmp.temp_tbl . . . . . . . . . . . . . . > AS . . . . . . . . . . . . . . > SELECT * FROM typeall_l; +---++ | Fragment | Number of records written | +---++ | 0_0 | 105| +---++ 1 row selected (0.233 seconds) 0: jdbc:drill:schema=dfs.tmp> SELECT COUNT(*) FROM dfs.tmp.temp_tbl; +-+ | EXPR$0 | +-+ | 106 | +-+ 1 row selected (0.189 seconds) 0: jdbc:drill:schema=dfs.tmp> drop table dfs.tmp.temp_tbl; +---+-+ | ok | summary | +---+-+ | true | Temporary table [temp_tbl] dropped | +---+-+ 1 row selected (0.186 seconds) 0: jdbc:drill:schema=dfs.tmp> ALTER SESSION SET `store.format`='parquet'; +---++ | ok |summary | +---++ | true | store.format updated. | +---++ 1 row selected (0.196 seconds) 0: jdbc:drill:schema=dfs.tmp> CREATE TEMPORARY TABLE dfs.tmp.temp_tbl . . . . . . . . . . . . . . > AS . . . . . . . . . . . . . . > SELECT * FROM typeall_l; +---++ | Fragment | Number of records written | +---++ | 0_0 | 105| +---++ 1 row selected (0.263 seconds) 0: jdbc:drill:schema=dfs.tmp> SELECT COUNT(*) FROM dfs.tmp.temp_tbl; +-+ | EXPR$0 | +-+ | 105 | +-+ 1 row selected (0.169 seconds) 0: jdbc:drill:schema=dfs.tmp> drop table dfs.tmp.temp_tbl; +---+-+ | ok | summary | +---+-+ | true | Temporary table [temp_tbl] dropped | +---+-+ 1 row selected (0.165 seconds) 0: jdbc:drill:schema=dfs.tmp> {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5191) OutOfMemoryException - TPCDS query4
Khurram Faraaz created DRILL-5191: - Summary: OutOfMemoryException - TPCDS query4 Key: DRILL-5191 URL: https://issues.apache.org/jira/browse/DRILL-5191 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.10.0 Environment: 4 node cluster CentOS Reporter: Khurram Faraaz Priority: Critical TPC-DS Query4 against SF100 on Drill 1.10.0 (ee399317), on a 4 node CentOS cluster Query4 => https://raw.githubusercontent.com/Agirish/tpcds/master/query4.sql total number of fragments : 1,125 Stack trace from drillbit.log {noformat} 2017-01-11 11:17:57,007 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO o.a.d.e.w.fragment.FragmentExecutor - 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State change requested AWAITING_ALLOCATION --> RUNNING 2017-01-11 11:17:57,008 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO o.a.d.e.w.f.FragmentStatusReporter - 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State to report: RUNNING 2017-01-11 11:17:57,009 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO o.a.d.e.w.fragment.FragmentExecutor - 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State change requested RUNNING --> FAILED 2017-01-11 11:17:57,009 [BitServer-6] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication. Connection: /10.10.100.202:31012 <--> /10.10.100.201:44712 (data server). Closing connection. org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer of size 16384 due to memory limit. Current allocation: 16777216 at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216) ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:191) ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.memory.DrillByteBufAllocator.buffer(DrillByteBufAllocator.java:49) ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.memory.DrillByteBufAllocator.ioBuffer(DrillByteBufAllocator.java:64) ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] 2017-01-11 11:17:57,009 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:24:5] INFO o.a.d.e.w.fragment.FragmentExecutor - 2789eba3-60f0-0b2f-eba8-82331735d5c4:24:5: State change requested AWAITING_ALLOCATION --> FAILED 2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:63:5] INFO o.a.d.e.w.fragment.FragmentExecutor - 2789eba3-60f0-0b2f-eba8-82331735d5c4:63:5: State change requested AWAITING_ALLOCATION --> RUNNING 2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:63:5] INFO o.a.d.e.w.f.FragmentStatusReporter - 2789eba3-60f0-0b2f-eba8-82331735d5c4:63:5: State to report: RUNNING 2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:24:5] INFO o.a.d.e.w.fragment.FragmentExecutor - 2789eba3-60f0-0b2f-eba8-82331735d5c4:24:5: State change requested FAILED --> FINISHED 2017-01-11 11:17:57,010 [BitServer-6] INFO o.a.d.exec.rpc.ProtobufLengthDecoder - Channel is closed, discarding remaining 3240924 byte(s) in buffer. 2017-01-11 11:17:57,011 [BitServer-10] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication. Connection: /10.10.100.202:31012 <--> /10.10.100.202:52127 (data server). Closing connection. org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer of size 4096 due to memory limit. Current allocation: 16777216 at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216) ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:191) ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
[jira] [Created] (DRILL-5188) TPC-DS query16 fails - IllegalArgumentException: Target must be less than target count
Khurram Faraaz created DRILL-5188: - Summary: TPC-DS query16 fails - IllegalArgumentException: Target must be less than target count Key: DRILL-5188 URL: https://issues.apache.org/jira/browse/DRILL-5188 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.10.0 Environment: 4 node cluster Reporter: Khurram Faraaz TPC-DS query 16 fails with below Exception. Query 32, Query 92, query 95 also fail due to same Exception, on Drill 1.10.0 (commit ID: bbcf4b76) against SF1 data. Details from drillbit.log and stack trace. {noformat} 2017-01-10 11:42:54,769 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 278b3740-fa7b-0f27-5c42-92f2f63f7f3d: SELECT Count(DISTINCT cs_order_number) AS `order count` , Sum(cs_ext_ship_cost) AS `total shipping cost` , Sum(cs_net_profit) AS `total net profit` FROM catalog_sales cs1 , date_dim , customer_address , call_center WHEREd_date BETWEEN '2002-3-01' AND ( Cast('2002-3-01' AS DATE) + INTERVAL '60' day) AND cs1.cs_ship_date_sk = d_date_sk AND cs1.cs_ship_addr_sk = ca_address_sk AND ca_state = 'IA' AND cs1.cs_call_center_sk = cc_call_center_sk AND cc_county IN ('Williamson County', 'Williamson County', 'Williamson County', 'Williamson County', 'Williamson County' ) AND EXISTS ( SELECT * FROM catalog_sales cs2 WHERE cs1.cs_order_number = cs2.cs_order_number ANDcs1.cs_warehouse_sk <> cs2.cs_warehouse_sk) AND NOT EXISTS ( SELECT * FROM catalog_returns cr1 WHERE cs1.cs_order_number = cr1.cr_order_number) ORDER BY count(DISTINCT cs_order_number) LIMIT 100 2017-01-10 11:42:54,887 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-01-10 11:42:54,888 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-01-10 11:42:54,895 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-01-10 11:42:54,895 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-01-10 11:42:54,902 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-01-10 11:42:54,903 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-01-10 11:42:54,908 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-01-10 11:42:54,909 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-01-10 11:42:54,919 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-01-10 11:42:54,920 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-01-10 11:42:54,946 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] ERROR o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: IllegalArgumentException: Target must be less than target count, 3 [Error Id: 61befda9-3479-48a4-a6ff-181703aea1b7 on centos-01.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalArgumentException: Target must be less than target count, 3 [Error Id: 61befda9-3479-48a4-a6ff-181703aea1b7 on centos-01.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) ~[drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:824) [drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:944) [drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281) [drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception during fragment initialization: Target must be less than target count, 3
[jira] [Created] (DRILL-5187) TPC-DS query 72 takes 32 minutes to complete
Khurram Faraaz created DRILL-5187: - Summary: TPC-DS query 72 takes 32 minutes to complete Key: DRILL-5187 URL: https://issues.apache.org/jira/browse/DRILL-5187 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.10.0 Environment: 4 node cluster Reporter: Khurram Faraaz Priority: Critical TPC-DS query takes close to 32 minutes to complete execution on a 4 node CentOS cluster, on Drill 1.10.0 against SF1 data. Attached here are, - drillbit.log (has only query 72 information) - JSON profile for query 72 - screen shot of fragment information from profiles tab on UI HASH_JOIN operator takes 21minutes (fragment : 04-xx-53). HASH_JOIN operator seems to be taking longer, it can be verified by looking at the profile information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5174) when different Drillbit versions are up on a cluster - we need to report an error
Khurram Faraaz created DRILL-5174: - Summary: when different Drillbit versions are up on a cluster - we need to report an error Key: DRILL-5174 URL: https://issues.apache.org/jira/browse/DRILL-5174 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.10.0 Reporter: Khurram Faraaz Drill 1.10.0 git commit : bbcf4b76 Note that one of the nodes in the 4 node cluster had a Drillbit from 1.9.0 and the other three Drillbits are of 1.10.0 version. We report an error on the Web UI (see DRILL-4604) for such a scenario, we need a similar message when querying sys.drillbits table from sqlline. {noformat} 0: jdbc:drill:schema=dfs.tmp> select * from sys.drillbits; +---++---++--+--+ | hostname | user_port | control_port | data_port | current | version | +---++---++--+--+ | centos-03.qa.lab | 31010 | 31011 | 31012 | false| 1.10.0-SNAPSHOT | | centos-02.qa.lab | 31010 | 31011 | 31012 | false| | | centos-04.qa.lab | 31010 | 31011 | 31012 | false| 1.10.0-SNAPSHOT | | centos-01.qa.lab | 31010 | 31011 | 31012 | true | 1.10.0-SNAPSHOT | +---++---++--+--+ 4 rows selected (0.285 seconds) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5165) wrong results - LIMIT ALL and OFFSET clause in same query
Khurram Faraaz created DRILL-5165: - Summary: wrong results - LIMIT ALL and OFFSET clause in same query Key: DRILL-5165 URL: https://issues.apache.org/jira/browse/DRILL-5165 Project: Apache Drill Issue Type: Bug Components: Query Planning & Optimization Affects Versions: 1.10.0 Reporter: Khurram Faraaz Priority: Critical This issue was reported by a user on Drill's user list. Drill 1.10.0 commit ID : bbcf4b76 I tried a similar query on apache Drill 1.10.0 and Drill returns wrong results when compared to Postgres, for a query that uses LIMIT ALL and OFFSET clause in the same query. We need to file a JIRA to track this issue. {noformat} 0: jdbc:drill:schema=dfs.tmp> select col_int from typeall_l order by 1 limit all offset 10; +--+ | col_int | +--+ +--+ No rows selected (0.211 seconds) 0: jdbc:drill:schema=dfs.tmp> select col_int from typeall_l order by col_int limit all offset 10; +--+ | col_int | +--+ +--+ No rows selected (0.24 seconds) {noformat} Query => select col_int from typeall_l limit all offset 10; Drill 1.10.0 returns 85 rows whereas for same query, postgres=# select col_int from typeall_l limit all offset 10; Postgres 9.3 returns 95 rows, which is the correct expected result. Query plan for above query that returns wrong results {noformat} 0: jdbc:drill:schema=dfs.tmp> explain plan for select col_int from typeall_l limit all offset 10; +--+--+ | text | json | +--+--+ | 00-00Screen 00-01 Project(col_int=[$0]) 00-02SelectionVectorRemover 00-03 Limit(offset=[10]) 00-04Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:///tmp/typeall_l]], selectionRoot=maprfs:/tmp/typeall_l, numFiles=1, usedMetadataFile=false, columns=[`col_int`]]]) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5164) equi-join query results in CompileException
Khurram Faraaz created DRILL-5164: - Summary: equi-join query results in CompileException Key: DRILL-5164 URL: https://issues.apache.org/jira/browse/DRILL-5164 Project: Apache Drill Issue Type: Bug Components: Execution - Codegen Affects Versions: 1.9.0 Reporter: Khurram Faraaz Priority: Critical Drill 1.9.0 git commit ID : 4c1b420b 4 node CentOS cluster JSON file has 4095 keys (columns) {noformat} 0: jdbc:drill:schema=dfs.tmp> select * from `manyColsInJson.json` t1, `manyColsInJson.json` t2 where t1.key2000 = t2.key2000; Error: SYSTEM ERROR: CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject[HashJoinProbeGen294.java]', Line 16397, Column 17: HashJoinProbeGen294.java:16397: error: code too large public void doSetup(FragmentContext context, VectorContainer buildBatch, RecordBatch probeBatch, RecordBatch outgoing) ^ (compiler.err.limit.code) Fragment 0:0 [Error Id: 7d0efa7e-e183-4c40-939a-4908699f94bf on centos-01.qa.lab:31010] (state=,code=0) {noformat} Stack trace from drillbit.log {noformat} 2016-12-26 09:52:11,321 [279f17fd-c8f0-5d18-1124-76099f0a5cc8:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject[HashJoinProbeGen294.java]', Line 16397, Column 17: HashJoinProbeGen294.java:16397: error: code too large public void doSetup(FragmentContext context, VectorContainer buildBatch, RecordBatch probeBatch, RecordBatch outgoing) ^ (compiler.err.limit.code) Fragment 0:0 [Error Id: 7d0efa7e-e183-4c40-939a-4908699f94bf on centos-01.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject[HashJoinProbeGen294.java]', Line 16397, Column 17: HashJoinProbeGen294.java:16397: error: code too large public void doSetup(FragmentContext context, VectorContainer buildBatch, RecordBatch probeBatch, RecordBatch outgoing) ^ (compiler.err.limit.code) Fragment 0:0 [Error Id: 7d0efa7e-e183-4c40-939a-4908699f94bf on centos-01.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) ~[drill-common-1.9.0.jar:1.9.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293) [drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262) [drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.9.0.jar:1.9.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: org.apache.drill.exec.exception.SchemaChangeException: org.apache.drill.exec.exception.ClassTransformationException: java.util.concurrent.ExecutionException: org.apache.drill.exec.exception.ClassTransformationException: Failure generating transformation classes for value: package org.apache.drill.exec.test.generated; ... public class HashJoinProbeGen294 { NullableVarCharVector[] vv0; NullableVarCharVector vv3; NullableVarCharVector[] vv6; ... vv49137 .copyFromSafe((probeIndex), (outIndex), vv49134); vv49143 .copyFromSafe((probeIndex), (outIndex), vv49140); vv49149 .copyFromSafe((probeIndex), (outIndex), vv49146); } } public void __DRILL_INIT__() throws SchemaChangeException { } } at org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:302) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:78) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:74) ~[drill-java-exec-1.9.0.jar:1.9.0] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na] ... 34 common frames omitted Caused by: org.codehaus.commons.compiler.CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject[HashJoinProbeGen294.java]', Line 16397, Column 17: HashJoinProbeGen294.java:16397: error: code too large public void
[jira] [Created] (DRILL-5141) Difference in output date format on sqlline vs Drill's Web UI
Khurram Faraaz created DRILL-5141: - Summary: Difference in output date format on sqlline vs Drill's Web UI Key: DRILL-5141 URL: https://issues.apache.org/jira/browse/DRILL-5141 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.9.0 Reporter: Khurram Faraaz Priority: Minor Difference in output date format on sqlline vs Drill's Web UI Drill 1.9.0 git commit ID : 4c1b420b >From sqlline prompt {noformat} 0: jdbc:drill:schema=dfs.tmp> SELECT cast('2016/09/22' as date) FROM (VALUES(1)); +-+ | EXPR$0| +-+ | 2016-09-22 | +-+ 1 row selected (0.474 seconds) {noformat} The same query returns a date in a different output format on the web UI {noformat} 2016-09-22T00:00:00.000Z {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5130) UNION ALL difference in results
Khurram Faraaz created DRILL-5130: - Summary: UNION ALL difference in results Key: DRILL-5130 URL: https://issues.apache.org/jira/browse/DRILL-5130 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.9.0 Reporter: Khurram Faraaz Drill 1.9.0 git commit ID: 51246693 {noformat} 0: jdbc:drill:schema=dfs.tmp> values(1,2,3,4,5,6) union all values(7,8,9,10,11,12); +-+-+-+-+-+-+ | EXPR$0 | EXPR$1 | EXPR$2 | EXPR$3 | EXPR$4 | EXPR$5 | +-+-+-+-+-+-+ | 7 | 8 | 9 | 10 | 11 | 12 | | 7 | 8 | 9 | 10 | 11 | 12 | +-+-+-+-+-+-+ 2 rows selected (0.209 seconds) {noformat} Postgres 9.3 {noformat} postgres=# values(1,2,3,4,5,6) union all values(7,8,9,10,11,12); column1 | column2 | column3 | column4 | column5 | column6 -+-+-+-+-+- 1 | 2 | 3 | 4 | 5 | 6 7 | 8 | 9 | 10 | 11 | 12 (2 rows) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5099) support OFFSET values greater than one, for LEAD & LAG window functions
Khurram Faraaz created DRILL-5099: - Summary: support OFFSET values greater than one, for LEAD & LAG window functions Key: DRILL-5099 URL: https://issues.apache.org/jira/browse/DRILL-5099 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.9.0 Reporter: Khurram Faraaz Priority: Minor Provide support for OFFSET values greater than one, for LEAD & LAG window functions. Adding [~adeneche] comments from the dev list here {noformat} Things that need to be done to make Lag (or Lead) support offsets other than 1: - WindowFunction.Lead should extract the offset value from its FunctionCall argument, you can look at WindowFunctionNtile.numTilesFromExpression() for and example on how to do that. - make sure calls to copyNext() and copyPrev() in NoFrameSupportTemplate use the offset and not the hard coded value (you already figured that out) - finally make sure you update UnsupportedOperatorsVisitor to no longer throw an exception when we pass an offset value other than 1 to Lead or Lag. Just search for DRILL-3596 in that class and you will find the if block that need to be removed I think this should be enough to get it to work in the general case. {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5077) Memory Leak -
Khurram Faraaz created DRILL-5077: - Summary: Memory Leak - Key: DRILL-5077 URL: https://issues.apache.org/jira/browse/DRILL-5077 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.9.0 Environment: 4 node cluster CentOS Reporter: Khurram Faraaz Priority: Blocker terminating foreman drillbit while a query is running results in memory leak Drill 1.9.0 git commit id: 4312d65b Stack trace from drillbit.log {noformat} 2016-11-28 06:12:45,338 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 27c43522-79ca-989f-3659-d7ccbc77e2e7: select count(*) from `twoKeyJsn.json` 2016-11-28 06:12:45,602 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2016-11-28 06:12:45,602 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2016-11-28 06:12:45,603 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2016-11-28 06:12:45,603 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2016-11-28 06:12:45,603 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2016-11-28 06:12:45,633 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2016-11-28 06:12:45,669 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Time: 33ms total, 33.494123ms avg, 33ms max. 2016-11-28 06:12:45,669 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Earliest start: 9.54 μs, Latest start: 9.54 μs, Average start: 9.54 μs . 2016-11-28 06:12:45,913 [27c43522-79ca-989f-3659-d7ccbc77e2e7:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 27c43522-79ca-989f-3659-d7ccbc77e2e7:0:0: State change requested AWAITING_ALLOCATION --> RUNNING 2016-11-28 06:12:45,913 [27c43522-79ca-989f-3659-d7ccbc77e2e7:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 27c43522-79ca-989f-3659-d7ccbc77e2e7:0:0: State to report: RUNNING Mon Nov 28 06:12:48 UTC 2016 Terminating drillbit pid 28004 2016-11-28 06:12:48,697 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Received shutdown request. 2016-11-28 06:12:55,749 [pool-6-thread-2] INFO o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup io.netty.channel.nio.NioEventLoopGroup@15bcfacd in 1017 ms 2016-11-28 06:12:55,750 [pool-6-thread-2] INFO o.a.drill.exec.service.ServiceEngine - closed dataPool in 1018 ms 2016-11-28 06:12:57,749 [Drillbit-ShutdownHook#0] WARN o.apache.drill.exec.work.WorkManager - Closing WorkManager but there are 1 running fragments. 2016-11-28 06:12:57,751 [27c43522-79ca-989f-3659-d7ccbc77e2e7:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 27c43522-79ca-989f-3659-d7ccbc77e2e7:0:0: State change requested RUNNING --> FAILED 2016-11-28 06:12:57,751 [27c43522-79ca-989f-3659-d7ccbc77e2e7:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 27c43522-79ca-989f-3659-d7ccbc77e2e7:0:0: State change requested FAILED --> FINISHED 2016-11-28 06:12:57,756 [27c43522-79ca-989f-3659-d7ccbc77e2e7:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException Fragment 0:0 [Error Id: 2df2f9a1-a7bf-4454-a31b-717ab4ebd815 on centos-01.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: NullPointerException Fragment 0:0 [Error Id: 2df2f9a1-a7bf-4454-a31b-717ab4ebd815 on centos-01.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) ~[drill-common-1.9.0.jar:1.9.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293) [drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262) [drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.9.0.jar:1.9.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_101] at
[jira] [Created] (DRILL-5049) difference in results - correlated subquery interacting with null equality join
Khurram Faraaz created DRILL-5049: - Summary: difference in results - correlated subquery interacting with null equality join Key: DRILL-5049 URL: https://issues.apache.org/jira/browse/DRILL-5049 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.9.0 Reporter: Khurram Faraaz Here is a query that uses null equality join. Drill 1.9.0 returns 124 records, whereas Postgres 9.3 returns 145 records. I am on Drill 1.9.0 git commit id: db308549 I have attached the results from Drill 1.9.0 and Postgres, please review. {noformat} 0: jdbc:drill:schema=dfs.tmp> explain plan for . . . . . . . . . . . . . . > SELECT * . . . . . . . . . . . . . . > FROM `t_alltype.parquet` t1 . . . . . . . . . . . . . . > WHERE EXISTS . . . . . . . . . . . . . . > ( . . . . . . . . . . . . . . > SELECT * . . . . . . . . . . . . . . > FROM `t_alltype.parquet` t2 . . . . . . . . . . . . . . > WHERE t1.c4 = t2.c4 OR (t1.c4 IS NULL AND t2.c4 IS NULL) . . . . . . . . . . . . . . > ); +--+--+ | text | json | +--+--+ | 00-00Screen 00-01 Project(*=[$0]) 00-02Project(T30¦¦*=[$0]) 00-03 HashJoin(condition=[AND(=($1, $2), =($1, $3))], joinType=[inner]) 00-05Project(T30¦¦*=[$0], c4=[$1]) 00-07 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:///tmp/t_alltype.parquet]], selectionRoot=maprfs:/tmp/t_alltype.parquet, numFiles=1, usedMetadataFile=false, columns=[`*`]]]) 00-04HashAgg(group=[{0, 1}], agg#0=[MIN($2)]) 00-06 Project(c40=[$1], c400=[$1], $f0=[true]) 00-08HashJoin(condition=[IS NOT DISTINCT FROM($0, $1)], joinType=[inner]) 00-10 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:///tmp/t_alltype.parquet]], selectionRoot=maprfs:/tmp/t_alltype.parquet, numFiles=1, usedMetadataFile=false, columns=[`c4`]]]) 00-09 Project(c40=[$0]) 00-11HashAgg(group=[{0}]) 00-12 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:///tmp/t_alltype.parquet]], selectionRoot=maprfs:/tmp/t_alltype.parquet, numFiles=1, usedMetadataFile=false, columns=[`c4`]]]) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5040) Interrupted CTAS should not succeed & should not create physical file on disk
Khurram Faraaz created DRILL-5040: - Summary: Interrupted CTAS should not succeed & should not create physical file on disk Key: DRILL-5040 URL: https://issues.apache.org/jira/browse/DRILL-5040 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.9.0 Reporter: Khurram Faraaz We should not allow CTAS to succeed (i.e create physical file on disk ) in the case where it was interrupted. (vis Ctrl-C) Drill 1.9.0 git commit ID : db30854 Consider the below CTAS that was interrupted using Ctrl-C {noformat} 0: jdbc:drill:schema=dfs.tmp> create table temp_t1 as select * from `twoKeyJsn.json`; [ issue Ctrl-C while the above CTAS is running ] No rows affected (7.694 seconds) {noformat} I verified that physical file was created on disk, even though the above CTAS was Canceled {noformat} [root@centos-01 ~]# hadoop fs -ls /tmp/temp_t1* -rwxr-xr-x 3 root root 36713198 2016-11-14 10:51 /tmp/temp_t1/0_0_0.parquet {noformat} We are able to do a select on the CTAS table (above) that was Canceled. {noformat} 0: jdbc:drill:schema=dfs.tmp> select count(*) from temp_t1; +--+ | EXPR$0 | +--+ | 3747840 | +--+ 1 row selected (0.183 seconds) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5039) NPE - CTAS PARTITION BY ()
Khurram Faraaz created DRILL-5039: - Summary: NPE - CTAS PARTITION BY () Key: DRILL-5039 URL: https://issues.apache.org/jira/browse/DRILL-5039 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.9.0 Reporter: Khurram Faraaz We are seeing an NPE, when CTAS is used with PARTITION BY ( and all columns are projected in SELECT of CTAS. Drill 1.9.0 git commit ID : db30854 {noformat} 0: jdbc:drill:schema=dfs.tmp> CREATE TABLE dfs.tmp.temp_tbl . . . . . . . . . . . . . . > PARTITION BY (col_chr) . . . . . . . . . . . . . . > AS . . . . . . . . . . . . . . > SELECT col_int, col_chr, col_vrchr1, col_vrchr2 , col_dt, col_tim, col_tmstmp , col_flt, col_intrvl_yr , col_intrvl_day , col_bln . . . . . . . . . . . . . . > FROM typeall_l; Error: SYSTEM ERROR: NullPointerException Fragment 0:0 [Error Id: ab6c199e-cb61-42dd-ae22-0090eea22ec5 on centos-01.qa.lab:31010] (state=,code=0) {noformat} {noformat} 2016-11-12 19:54:14,901 [27d88c99-a64d-0317-ba3b-d78195cf85cc:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException Fragment 0:0 [Error Id: ab6c199e-cb61-42dd-ae22-0090eea22ec5 on centos-01.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: NullPointerException Fragment 0:0 [Error Id: ab6c199e-cb61-42dd-ae22-0090eea22ec5 on centos-01.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) ~[drill-common-1.9.0.jar:1.9.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293) [drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262) [drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.9.0.jar:1.9.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] Caused by: java.lang.NullPointerException: null at org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:100) ~[vector-1.9.0.jar:1.9.0] at org.apache.drill.exec.test.generated.ProjectorGen3.doEval(ProjectorTemplate.java:88) ~[na:na] at org.apache.drill.exec.test.generated.ProjectorGen3.projectRecords(ProjectorTemplate.java:62) ~[na:na] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:199) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:91) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) ~[drill-java-exec-1.9.0.jar:1.9.0] at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81) ~[drill-java-exec-1.9.0.jar:1.9.0] at
[jira] [Created] (DRILL-5033) Query on JSON that has null as value for each key
Khurram Faraaz created DRILL-5033: - Summary: Query on JSON that has null as value for each key Key: DRILL-5033 URL: https://issues.apache.org/jira/browse/DRILL-5033 Project: Apache Drill Issue Type: Bug Components: Storage - JSON Affects Versions: 1.9.0 Reporter: Khurram Faraaz Drill 1.9.0 git commit ID : 83513daf Drill returns same result with or without `store.json.all_text_mode`=true Note that each key in the JSON has null as its value. [root@cent01 null_eq_joins]# cat right_all_nulls.json { "intKey" : null, "bgintKey": null, "strKey": null, "boolKey": null, "fltKey": null, "dblKey": null, "timKey": null, "dtKey": null, "tmstmpKey": null, "intrvldyKey": null, "intrvlyrKey": null } [root@cent01 null_eq_joins]# Querying the above JSON file results in null as query result. - We should see each of the keys in the JSON as a column in query result. - And in each column the value should be a null value. Current behavior does not look right. {noformat} 0: jdbc:drill:schema=dfs.tmp> select * from `right_all_nulls.json`; +---+ | * | +---+ | null | +---+ 1 row selected (0.313 seconds) {noformat} Adding comment from [~julianhyde] {noformat} IMHO it is similar but not the same as DRILL-1256. Worth logging an issue and let [~jnadeau] (or someone) put on the record what should be the behavior of an empty record (empty JSON map) when it is top-level (as in this case) or in a collection. {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5029) need better error - cast interval day to int or bigint
Khurram Faraaz created DRILL-5029: - Summary: need better error - cast interval day to int or bigint Key: DRILL-5029 URL: https://issues.apache.org/jira/browse/DRILL-5029 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.8.0 Reporter: Khurram Faraaz Priority: Minor We need a better error message, today Drill returns an AssertionError {noformat} 0: jdbc:drill:schema=dfs.tmp> values(cast('P162M24D' as INTERVAL DAY)); +-+ | EXPR$0 | +-+ | P24D| +-+ 1 row selected (0.419 seconds) {noformat} A better error would be ERROR: cannot cast type interval to int {noformat} 0: jdbc:drill:schema=dfs.tmp> values(cast(cast('P162M24D' as INTERVAL DAY) as INT)); Error: SYSTEM ERROR: AssertionError: Internal error: Conversion to relational algebra failed to preserve datatypes: validated type: RecordType(INTEGER NOT NULL EXPR$0) NOT NULL converted type: RecordType(BIGINT NOT NULL EXPR$0) NOT NULL rel: LogicalProject(EXPR$0=[/INT(Reinterpret(CAST('P162M24D'):INTERVAL DAY NOT NULL), 8640)]) LogicalValues(tuples=[[{ 0 }]]) [Error Id: 662716fb-c2c3-4032-8d92-835f8b0ec7ae on centos-01.qa.lab:31010] (state=,code=0) {noformat} A better message would be ERROR: cannot cast type interval to bigint {noformat} 0: jdbc:drill:schema=dfs.tmp> values(cast(cast('P162M24D' as INTERVAL DAY) as BIGINT)); Error: SYSTEM ERROR: AssertionError: todo: implement syntax SPECIAL(Reinterpret(CAST('P162M24D'):INTERVAL DAY NOT NULL)) [Error Id: ef2c31cd-dee3-4f13-aca0-05c16185f789 on centos-01.qa.lab:31010] (state=,code=0) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5024) CTAS with LIMIT 0 query in SELECT stmt does not create parquet file
Khurram Faraaz created DRILL-5024: - Summary: CTAS with LIMIT 0 query in SELECT stmt does not create parquet file Key: DRILL-5024 URL: https://issues.apache.org/jira/browse/DRILL-5024 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.8.0 Reporter: Khurram Faraaz Note that CTAS was successful {noformat} 0: jdbc:drill:schema=dfs.tmp> create table regtbl_w0rows as select * from typeall_l LIMIT 0; +---++ | Fragment | Number of records written | +---++ | 0_0 | 0 | +---++ 1 row selected (0.51 seconds) {noformat} But a SELECT on CTAS created file fails. {noformat} 0: jdbc:drill:schema=dfs.tmp> select * from regtbl_w0rows; Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 27: Table 'regtbl_w0rows' not found SQL Query null [Error Id: 0569cf98-3800-43ee-b635-aa101b016d46 on centos-01.qa.lab:31010] (state=,code=0) {noformat} DROP on the CTAS created table also fails {noformat} 0: jdbc:drill:schema=dfs.tmp> drop table regtbl_w0rows; Error: VALIDATION ERROR: Table [regtbl_w0rows] not found [Error Id: fb0b1ea8-f76d-42e2-b69c-4beae2798bdf on centos-01.qa.lab:31010] (state=,code=0) 0: jdbc:drill:schema=dfs.tmp> {noformat} Verified that CTAS did not create a physical file in dfs.tmp schema {noformat} [test@cent01 bin]# hadoop fs -ls /tmp/regtbl_w0rows ls: `/tmp/regtbl_w0rows': No such file or directory {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5010) Equality join condition is treated as a MergeJoin and not as a HashJoin.
Khurram Faraaz created DRILL-5010: - Summary: Equality join condition is treated as a MergeJoin and not as a HashJoin. Key: DRILL-5010 URL: https://issues.apache.org/jira/browse/DRILL-5010 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.9.0 Reporter: Khurram Faraaz Equality join condition is treated as a MergeJoin and not as a HashJoin. Drill 1.9.0 git commit ID: 83513daf Projecting the join columns results in merge join, whereas it should be doing a HashJoin. {noformat} 0: jdbc:drill:schema=dfs.tmp> explain plan for . . . . . . . . . . . . . . > select t1.intKey, t2.intKey from `left.json` t1, `right_all_nulls.json` t2 WHERE t1.intKey = t2.intKey OR ( t1.intKey IS NULL AND t2.intKey IS NULL); +--+--+ | text | json | +--+--+ | 00-00Screen 00-01 Project(intKey=[$0], intKey0=[$1]) 00-02Project(intKey=[$0], intKey0=[$1]) 00-03 MergeJoin(condition=[IS NOT DISTINCT FROM($0, $1)], joinType=[inner]) 00-05SelectionVectorRemover 00-07 Sort(sort0=[$0], dir0=[ASC]) 00-09Scan(groupscan=[EasyGroupScan [selectionRoot=maprfs:/tmp/left.json, numFiles=1, columns=[`intKey`], files=[maprfs:///tmp/left.json]]]) 00-04Project(intKey0=[$0]) 00-06 SelectionVectorRemover 00-08Sort(sort0=[$0], dir0=[ASC]) 00-10 Scan(groupscan=[EasyGroupScan [selectionRoot=maprfs:/tmp/right_all_nulls.json, numFiles=1, columns=[`intKey`], files=[maprfs:///tmp/right_all_nulls.json]]]) {noformat} Note that HashAgg and HashJoin were enabled. {noformat} | planner.enable_hashagg | BOOLEAN | SYSTEM | DEFAULT | null| null| true | null | | planner.enable_hashjoin| BOOLEAN | SYSTEM | DEFAULT | null| null| true | null | {noformat} Doing a SELECT results in a HashJoin in the query plan {noformat} 0: jdbc:drill:schema=dfs.tmp> explain plan for . . . . . . . . . . . . . . > select * from `left.json` t1, `right_all_nulls.json` t2 WHERE t1.intKey = t2.intKey OR ( t1.intKey IS NULL AND t2.intKey IS NULL); +--+--+ | text | json | +--+--+ | 00-00Screen 00-01 ProjectAllowDup(*=[$0], *0=[$1]) 00-02Project(T46¦¦*=[$0], T47¦¦*=[$2]) 00-03 HashJoin(condition=[IS NOT DISTINCT FROM($1, $3)], joinType=[inner]) 00-04Project(T47¦¦*=[$0], intKey0=[$1]) 00-06 Project(T47¦¦*=[$0], intKey=[$1]) 00-08Scan(groupscan=[EasyGroupScan [selectionRoot=maprfs:/tmp/right_all_nulls.json, numFiles=1, columns=[`*`], files=[maprfs:///tmp/right_all_nulls.json]]]) 00-05Project(T46¦¦*=[$0], intKey=[$1]) 00-07 Scan(groupscan=[EasyGroupScan [selectionRoot=maprfs:/tmp/left.json, numFiles=1, columns=[`*`], files=[maprfs:///tmp/left.json]]]) {noformat} Data used in above queries {noformat} [root@centos-01 null_eq_joins]# cat left.json { "intKey" : 123, "bgintKey": 1234567, "strKey": "this is a test string", "boolKey": true, "fltKey": 123.786, "dblKey": 457.984, "timKey": "18:30:45", "dtKey": "1997-10-21", "tmstmpKey": "2007-04-30 13:10:02.047", "intrvldyKey": "P9DT38833S", "intrvlyrKey": "P255M" } [root@centos-01 null_equality_joins]# [root@centos-01 null_eq_joins]# cat right_all_nulls.json { "intKey" : null, "bgintKey": null, "strKey": null, "boolKey": null, "fltKey": null, "dblKey": null, "timKey": null, "dtKey": null, "tmstmpKey": null, "intrvldyKey": null, "intrvlyrKey": null } [root@centos-01 null_eq_joins]# {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5003) need a better error message
Khurram Faraaz created DRILL-5003: - Summary: need a better error message Key: DRILL-5003 URL: https://issues.apache.org/jira/browse/DRILL-5003 Project: Apache Drill Issue Type: Bug Components: Query Planning & Optimization Affects Versions: 1.9.0 Reporter: Khurram Faraaz Priority: Minor We need a better error message, Drill 1.9.0 returns an AssertionError. {noformat} 0: jdbc:drill:schema=dfs.tmp> SELECT * . . . . . . . . . . . . . . > FROM ( . . . . . . . . . . . . . . > select * from `t_alltype.parquet` t1, `t_alltype.parquet` t2 . . . . . . . . . . . . . . > WHERE EXISTS(t1.c4 = t2.c4) OR (t1.c4 IS NULL AND t2.c4 IS NULL) . . . . . . . . . . . . . . > ); Error: SYSTEM ERROR: AssertionError: Internal error: not a query: `t1`.`c4` = `t2`.`c4` [Error Id: 91e7af43-27a7-4577-802a-643705aeaf4f on centos-01.qa.lab:31010] (state=,code=0) {noformat} {noformat} 0: jdbc:drill:schema=dfs.tmp> SELECT * . . . . . . . . . . . . . . > FROM ( . . . . . . . . . . . . . . > select * from `t_alltype.parquet` t1, `t_alltype.parquet` t2 . . . . . . . . . . . . . . > WHERE EXISTS(t1.c4 = t2.c4 OR (t1.c4 IS NULL AND t2.c4 IS NULL)) . . . . . . . . . . . . . . > ); Error: SYSTEM ERROR: AssertionError: Internal error: not a query: `t1`.`c4` = `t2`.`c4` OR `t1`.`c4` IS NULL AND `t2`.`c4` IS NULL {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4985) CAST in join predicate in null equality join - "query cannot be planned error"
Khurram Faraaz created DRILL-4985: - Summary: CAST in join predicate in null equality join - "query cannot be planned error" Key: DRILL-4985 URL: https://issues.apache.org/jira/browse/DRILL-4985 Project: Apache Drill Issue Type: Bug Components: Query Planning & Optimization Affects Versions: 1.9.0 Reporter: Khurram Faraaz Use of CAST in join predicate results in, "query cannot be planned error", in a Null Equality Join. Drill 1.9.0, git commit ID: a29f1e29 {noformat} 0: jdbc:drill:schema=dfs.tmp> SELECT t1.col_int , t2.col_int FROM typeall_l t1, typeall_r t2 WHERE CAST(t1.col_int AS BIGINT)= CAST(t2.col_int AS BIGINT) OR ( t1.col_int IS NULL AND t2.col_int IS NULL ); Error: UNSUPPORTED_OPERATION ERROR: This query cannot be planned possibly due to either a cartesian join or an inequality join [Error Id: 10452a20-c0a7-45b1-8ee7-065b28484738 on centos-01.qa.lab:31010] (state=,code=0) {noformat} {noformat} 0: jdbc:drill:schema=dfs.tmp> SELECT t1.col_int , t2.col_int FROM typeall_l t1, typeall_r t2 WHERE CAST(t1.col_int AS BIGINT)= CAST(t2.col_int AS BIGINT) OR ( CAST(t1.col_int AS BIGINT) IS NULL AND CAST(t2.col_int AS BIGINT) IS NULL ); Error: UNSUPPORTED_OPERATION ERROR: This query cannot be planned possibly due to either a cartesian join or an inequality join [Error Id: ca8eba6c-b8c0-42d9-a0a0-16e94f887280 on centos-01.qa.lab:31010] (state=,code=0) {noformat} Whereas, without the CAST to BIGINT in the join predicate, query returns correct results. {noformat} 0: jdbc:drill:schema=dfs.tmp> SELECT t1.col_int , t2.col_int FROM typeall_l t1, typeall_r t2 WHERE t1.col_int = t2.col_int OR ( t1.col_int IS NULL AND t2.col_int IS NULL ); +--+---+ | col_int | col_int0 | +--+---+ | 30410| 30410 | | 37420| 37420 | | 58583| 58583 | | 1| 1 | | null | null | | null | null | | null | null | | 20326| 20326 | | null | null | | null | null | | null | null | | 19 | 19| | 60718| 60718 | | null | null | | null | null | | null | null | +--+---+ 16 rows selected (0.304 seconds) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)