[jira] [Created] (DRILL-5781) Fix unit test failures to use tests config even if default config is available
Volodymyr Vysotskyi created DRILL-5781: -- Summary: Fix unit test failures to use tests config even if default config is available Key: DRILL-5781 URL: https://issues.apache.org/jira/browse/DRILL-5781 Project: Apache Drill Issue Type: Bug Affects Versions: 1.11.0 Reporter: Volodymyr Vysotskyi Assignee: Volodymyr Vysotskyi Unit tests fail when they are run with the mapr profile. Tests failures, connected with the Zookeeper configuration that differs from expected: {noformat} DrillClientTest>TestWithZookeeper.setUp:32 » Runtime java.io.IOException: Coul... TestZookeeperClient.testPutWithMatchingVersion » IO Could not configure server... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testStartingClientEnablesCacheAndEnsuresRootNodeExists » IO TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testHasPathThrowsDrillRuntimeException » IO Could not conf... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testHasPathFalseWithVersion » IO Could not configure serve... TestZookeeperClient.tearDown:86 NullPointer TestEphemeralStore.testPutAndGetWorksAntagonistacally » IO Could not configure... TestEphemeralStore.tearDown:132 NullPointer TestZookeeperClient.testGetWithVersion » IO Could not configure server because... TestZookeeperClient.tearDown:86 NullPointer TestEphemeralStore.testStoreRegistersDispatcherAndStartsItsClient » IO Could n... TestEphemeralStore.tearDown:132 NullPointer TestZookeeperClient.testPutWithNonMatchingVersion » IO Could not configure ser... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testGetWithEventualConsistencyHitsCache » IO Could not con... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testPutIfAbsentWhenPresent » IO Could not configure server... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testHasPathTrueWithVersion » IO Could not configure server... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testPutAndGetWorks » IO Could not configure server because... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testPutIfAbsentWhenAbsent » IO Could not configure server ... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testHasPathWithEventualConsistencyHitsCache » IO Could not... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testCreate » IO Could not configure server because SASL co... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testDelete » IO Could not configure server because SASL co... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testEntriesReturnsRelativePaths » IO Could not configure s... TestZookeeperClient.tearDown:86 NullPointer TestPStoreProviders>TestWithZookeeper.setUp:32 » Runtime java.io.IOException: ... TestPauseInjection.pauseOnSpecificBit:151 » Runtime java.io.IOException: Could... TestExceptionInjection.injectionOnSpecificBit:217 » Runtime java.io.IOExceptio... HBaseTestsSuite.initCluster:110 » IO No JAAS configuration section named 'Serv... {noformat} Test failures, connected with Hadoop configuration that differs from expected: {noformat} TestInboundImpersonation.setup:58->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111 » ClassCast TestImpersonationMetadata.setup:58->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111 » ClassCast TestImpersonationDisabledWithMiniDFS.setup:37->BaseTestImpersonation.startMiniDfsCluster:106 » Runtime TestImpersonationQueries.setup:46->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111 » ClassCast TestHiveStorage>HiveTestBase.generateHive:34 » Runtime java.lang.RuntimeExcept... TestInfoSchemaOnHiveStorage>HiveTestBase.generateHive:34 » Runtime java.lang.R... TestInbuiltHiveUDFs>HiveTestBase.generateHive:35 » ExecutionSetup Failure sett... TestSampleHiveUDFs>HiveTestBase.generateHive:35 » ExecutionSetup Failure setti... TestStorageBasedHiveAuthorization.setup:109->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111 » ClassCast TestSqlStdBasedAuthorization.setup:72->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111 » ClassCast TestHivePartitionPruning>HiveTestBase.generateHive:35 » ExecutionSetup Failure... TestViewSupportOnHiveTables.generateHive:35 » ExecutionSetup Failure setting u... TestHiveProjectPushDown>HiveTestBase.generateHive:35 » ExecutionSetup Failure ... {noformat} Test specific configuration should be added to fix these failures. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5781) Fix unit test failures to use tests config even if default config is available
[ https://issues.apache.org/jira/browse/DRILL-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5781: Issue Type: Task (was: Bug) > Fix unit test failures to use tests config even if default config is available > -- > > Key: DRILL-5781 > URL: https://issues.apache.org/jira/browse/DRILL-5781 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.11.0 >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi > > Unit tests fail when they are run with the mapr profile. > Tests failures, connected with the Zookeeper configuration that differs from > expected: > {noformat} > DrillClientTest>TestWithZookeeper.setUp:32 » Runtime java.io.IOException: > Coul... > TestZookeeperClient.testPutWithMatchingVersion » IO Could not configure > server... > TestZookeeperClient.tearDown:86 NullPointer > TestZookeeperClient.testStartingClientEnablesCacheAndEnsuresRootNodeExists > » IO > TestZookeeperClient.tearDown:86 NullPointer > TestZookeeperClient.testHasPathThrowsDrillRuntimeException » IO Could not > conf... > TestZookeeperClient.tearDown:86 NullPointer > TestZookeeperClient.testHasPathFalseWithVersion » IO Could not configure > serve... > TestZookeeperClient.tearDown:86 NullPointer > TestEphemeralStore.testPutAndGetWorksAntagonistacally » IO Could not > configure... > TestEphemeralStore.tearDown:132 NullPointer > TestZookeeperClient.testGetWithVersion » IO Could not configure server > because... > TestZookeeperClient.tearDown:86 NullPointer > TestEphemeralStore.testStoreRegistersDispatcherAndStartsItsClient » IO > Could n... > TestEphemeralStore.tearDown:132 NullPointer > TestZookeeperClient.testPutWithNonMatchingVersion » IO Could not configure > ser... > TestZookeeperClient.tearDown:86 NullPointer > TestZookeeperClient.testGetWithEventualConsistencyHitsCache » IO Could not > con... > TestZookeeperClient.tearDown:86 NullPointer > TestZookeeperClient.testPutIfAbsentWhenPresent » IO Could not configure > server... > TestZookeeperClient.tearDown:86 NullPointer > TestZookeeperClient.testHasPathTrueWithVersion » IO Could not configure > server... > TestZookeeperClient.tearDown:86 NullPointer > TestZookeeperClient.testPutAndGetWorks » IO Could not configure server > because... > TestZookeeperClient.tearDown:86 NullPointer > TestZookeeperClient.testPutIfAbsentWhenAbsent » IO Could not configure > server ... > TestZookeeperClient.tearDown:86 NullPointer > TestZookeeperClient.testHasPathWithEventualConsistencyHitsCache » IO Could > not... > TestZookeeperClient.tearDown:86 NullPointer > TestZookeeperClient.testCreate » IO Could not configure server because SASL > co... > TestZookeeperClient.tearDown:86 NullPointer > TestZookeeperClient.testDelete » IO Could not configure server because SASL > co... > TestZookeeperClient.tearDown:86 NullPointer > TestZookeeperClient.testEntriesReturnsRelativePaths » IO Could not > configure s... > TestZookeeperClient.tearDown:86 NullPointer > TestPStoreProviders>TestWithZookeeper.setUp:32 » Runtime java.io.IOException: > ... > TestPauseInjection.pauseOnSpecificBit:151 » Runtime java.io.IOException: > Could... > TestExceptionInjection.injectionOnSpecificBit:217 » Runtime > java.io.IOExceptio... > HBaseTestsSuite.initCluster:110 » IO No JAAS configuration section named > 'Serv... > {noformat} > Test failures, connected with Hadoop configuration that differs from expected: > {noformat} > TestInboundImpersonation.setup:58->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111 > » ClassCast > > TestImpersonationMetadata.setup:58->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111 > » ClassCast > > TestImpersonationDisabledWithMiniDFS.setup:37->BaseTestImpersonation.startMiniDfsCluster:106 > » Runtime > > TestImpersonationQueries.setup:46->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111 > » ClassCast > TestHiveStorage>HiveTestBase.generateHive:34 » Runtime > java.lang.RuntimeExcept... > TestInfoSchemaOnHiveStorage>HiveTestBase.generateHive:34 » Runtime > java.lang.R... > TestInbuiltHiveUDFs>HiveTestBase.generateHive:35 » ExecutionSetup Failure > sett... > TestSampleHiveUDFs>HiveTestBase.generateHive:35 » ExecutionSetup Failure > setti... > > TestStorageBasedHiveAuthorization.setup:109->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111 > » ClassCast > > TestSqlStdBasedAuthorization.setup:72->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111 > » ClassCast > TestHivePartitionPruning>HiveTestBase.generateHi
[jira] [Updated] (DRILL-5377) Five-digit year dates are displayed incorrectly via jdbc
[ https://issues.apache.org/jira/browse/DRILL-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5377: Labels: (was: ready-to-commit) > Five-digit year dates are displayed incorrectly via jdbc > > > Key: DRILL-5377 > URL: https://issues.apache.org/jira/browse/DRILL-5377 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: 1.10.0 >Reporter: Rahul Challapalli >Assignee: Vitalii Diravka > Fix For: 1.12.0 > > > git.commit.id.abbrev=38ef562 > The issue is connected to displaying five-digit year dates via jdbc > Below is the output, I get from test framework when I disable auto correction > for date fields > {code} > select l_shipdate from table(cp.`tpch/lineitem.parquet` (type => 'parquet', > autoCorrectCorruptDates => false)) order by l_shipdate limit 10; > ^@356-03-19 > ^@356-03-21 > ^@356-03-21 > ^@356-03-23 > ^@356-03-24 > ^@356-03-24 > ^@356-03-26 > ^@356-03-26 > ^@356-03-26 > ^@356-03-26 > {code} > Or a simpler case: > {code} > 0: jdbc:drill:> select cast('11356-02-16' as date) as FUTURE_DATE from > (VALUES(1)); > +--+ > | FUTURE_DATE | > +--+ > | 356-02-16 | > +--+ > 1 row selected (0.293 seconds) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5377) Five-digit year dates are displayed incorrectly via jdbc
[ https://issues.apache.org/jira/browse/DRILL-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5377: Reviewer: Paul Rogers > Five-digit year dates are displayed incorrectly via jdbc > > > Key: DRILL-5377 > URL: https://issues.apache.org/jira/browse/DRILL-5377 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: 1.10.0 >Reporter: Rahul Challapalli >Assignee: Vitalii Diravka > Fix For: 1.12.0 > > > git.commit.id.abbrev=38ef562 > The issue is connected to displaying five-digit year dates via jdbc > Below is the output, I get from test framework when I disable auto correction > for date fields > {code} > select l_shipdate from table(cp.`tpch/lineitem.parquet` (type => 'parquet', > autoCorrectCorruptDates => false)) order by l_shipdate limit 10; > ^@356-03-19 > ^@356-03-21 > ^@356-03-21 > ^@356-03-23 > ^@356-03-24 > ^@356-03-24 > ^@356-03-26 > ^@356-03-26 > ^@356-03-26 > ^@356-03-26 > {code} > Or a simpler case: > {code} > 0: jdbc:drill:> select cast('11356-02-16' as date) as FUTURE_DATE from > (VALUES(1)); > +--+ > | FUTURE_DATE | > +--+ > | 356-02-16 | > +--+ > 1 row selected (0.293 seconds) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5346) Web UI: remove link on user in query profile list
[ https://issues.apache.org/jira/browse/DRILL-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5346: Fix Version/s: 1.12.0 > Web UI: remove link on user in query profile list > - > > Key: DRILL-5346 > URL: https://issues.apache.org/jira/browse/DRILL-5346 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva >Priority: Minor > Fix For: 1.12.0 > > Attachments: query_profile_link.JPG > > > Remove link on user column that leads to query profile. Since we don't have > user page information, it's better to leave user name without link to avoid > confusion. Plus we already have link that leads to query profile on query > itself. > Page - http://localhost:8047/profiles > Screenshot - query_profile_link.JPG -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5341) Web UI: remove duplicating link to documentation in Options page
[ https://issues.apache.org/jira/browse/DRILL-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5341: Fix Version/s: 1.12.0 > Web UI: remove duplicating link to documentation in Options page > > > Key: DRILL-5341 > URL: https://issues.apache.org/jira/browse/DRILL-5341 > Project: Apache Drill > Issue Type: Bug >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Fix For: 1.12.0 > > Attachments: doc_link.JPG > > > Remove link to Documentation on Options page (http://localhost:8047/options) > as we have the same link on page header. Screenshot - doc_link.JPG -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (DRILL-5341) Web UI: remove duplicating link to documentation in Options page
[ https://issues.apache.org/jira/browse/DRILL-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva reassigned DRILL-5341: --- Assignee: Arina Ielchiieva > Web UI: remove duplicating link to documentation in Options page > > > Key: DRILL-5341 > URL: https://issues.apache.org/jira/browse/DRILL-5341 > Project: Apache Drill > Issue Type: Bug >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Fix For: 1.12.0 > > Attachments: doc_link.JPG > > > Remove link to Documentation on Options page (http://localhost:8047/options) > as we have the same link on page header. Screenshot - doc_link.JPG -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (DRILL-5346) Web UI: remove link on user in query profile list
[ https://issues.apache.org/jira/browse/DRILL-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva reassigned DRILL-5346: --- Assignee: Arina Ielchiieva > Web UI: remove link on user in query profile list > - > > Key: DRILL-5346 > URL: https://issues.apache.org/jira/browse/DRILL-5346 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Fix For: 1.12.0 > > Attachments: query_profile_link.JPG > > > Remove link on user column that leads to query profile. Since we don't have > user page information, it's better to leave user name without link to avoid > confusion. Plus we already have link that leads to query profile on query > itself. > Page - http://localhost:8047/profiles > Screenshot - query_profile_link.JPG -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (DRILL-5346) Web UI: remove link on user in query profile list
[ https://issues.apache.org/jira/browse/DRILL-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-5346. - Resolution: Fixed Fixed in the scope of DRILL-5766 > Web UI: remove link on user in query profile list > - > > Key: DRILL-5346 > URL: https://issues.apache.org/jira/browse/DRILL-5346 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Fix For: 1.12.0 > > Attachments: query_profile_link.JPG > > > Remove link on user column that leads to query profile. Since we don't have > user page information, it's better to leave user name without link to avoid > confusion. Plus we already have link that leads to query profile on query > itself. > Page - http://localhost:8047/profiles > Screenshot - query_profile_link.JPG -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (DRILL-5341) Web UI: remove duplicating link to documentation in Options page
[ https://issues.apache.org/jira/browse/DRILL-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-5341. - Resolution: Fixed Fixed in the scope of DRILL-5766 > Web UI: remove duplicating link to documentation in Options page > > > Key: DRILL-5341 > URL: https://issues.apache.org/jira/browse/DRILL-5341 > Project: Apache Drill > Issue Type: Bug >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Fix For: 1.12.0 > > Attachments: doc_link.JPG > > > Remove link to Documentation on Options page (http://localhost:8047/options) > as we have the same link on page header. Screenshot - doc_link.JPG -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5338) Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile that they are part of total time duration
[ https://issues.apache.org/jira/browse/DRILL-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5338: Fix Version/s: 1.12.0 > Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query > Profile that they are part of total time duration > --- > > Key: DRILL-5338 > URL: https://issues.apache.org/jira/browse/DRILL-5338 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva > Fix For: 1.12.0 > > Attachments: QueryProfile.JPG > > > Add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile > that they are part of total time duration > Screenshot is attached. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (DRILL-5338) Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile that they are part of total time duration
[ https://issues.apache.org/jira/browse/DRILL-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva reassigned DRILL-5338: --- Assignee: Arina Ielchiieva > Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query > Profile that they are part of total time duration > --- > > Key: DRILL-5338 > URL: https://issues.apache.org/jira/browse/DRILL-5338 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: 1.12.0 > > Attachments: QueryProfile.JPG > > > Add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile > that they are part of total time duration > Screenshot is attached. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (DRILL-5339) Web UI: flaws in query profile display on error
[ https://issues.apache.org/jira/browse/DRILL-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva reassigned DRILL-5339: --- Assignee: Arina Ielchiieva > Web UI: flaws in query profile display on error > --- > > Key: DRILL-5339 > URL: https://issues.apache.org/jira/browse/DRILL-5339 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Fix For: 1.12.0 > > Attachments: error_shift.JPG, fragment_profiles.JPG, > verbose_error.JPG, visualized_plan.JPG > > > 1. Error is shifted to the right (screenshot - error_shift.jpg) > 2. When Fragment Profiles doesn't have any data table, there is offset in > overview tab (screenshot - fragment_profiles.jpg) > 3. Verbose error message tab can be designed like overview tab to be more > concise (screenshot - verbose_error.jpg) > 4. Remove three dots in the end of "Verbose error message..." . > 5. When visualized plan is empty, there is big offset (screenshot - > visualized_plan.jpg) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5339) Web UI: flaws in query profile display on error
[ https://issues.apache.org/jira/browse/DRILL-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5339: Fix Version/s: 1.12.0 > Web UI: flaws in query profile display on error > --- > > Key: DRILL-5339 > URL: https://issues.apache.org/jira/browse/DRILL-5339 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva >Priority: Minor > Fix For: 1.12.0 > > Attachments: error_shift.JPG, fragment_profiles.JPG, > verbose_error.JPG, visualized_plan.JPG > > > 1. Error is shifted to the right (screenshot - error_shift.jpg) > 2. When Fragment Profiles doesn't have any data table, there is offset in > overview tab (screenshot - fragment_profiles.jpg) > 3. Verbose error message tab can be designed like overview tab to be more > concise (screenshot - verbose_error.jpg) > 4. Remove three dots in the end of "Verbose error message..." . > 5. When visualized plan is empty, there is big offset (screenshot - > visualized_plan.jpg) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (DRILL-5339) Web UI: flaws in query profile display on error
[ https://issues.apache.org/jira/browse/DRILL-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-5339. - Resolution: Fixed Points 1-4 fixed in the scope of DRILL-5766. > Web UI: flaws in query profile display on error > --- > > Key: DRILL-5339 > URL: https://issues.apache.org/jira/browse/DRILL-5339 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Fix For: 1.12.0 > > Attachments: error_shift.JPG, fragment_profiles.JPG, > verbose_error.JPG, visualized_plan.JPG > > > 1. Error is shifted to the right (screenshot - error_shift.jpg) > 2. When Fragment Profiles doesn't have any data table, there is offset in > overview tab (screenshot - fragment_profiles.jpg) > 3. Verbose error message tab can be designed like overview tab to be more > concise (screenshot - verbose_error.jpg) > 4. Remove three dots in the end of "Verbose error message..." . > 5. When visualized plan is empty, there is big offset (screenshot - > visualized_plan.jpg) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (DRILL-5338) Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile that they are part of total time duration
[ https://issues.apache.org/jira/browse/DRILL-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-5338. - Resolution: Fixed Fixed in the scope of DRILL-5766 > Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query > Profile that they are part of total time duration > --- > > Key: DRILL-5338 > URL: https://issues.apache.org/jira/browse/DRILL-5338 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: 1.12.0 > > Attachments: QueryProfile.JPG > > > Add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile > that they are part of total time duration > Screenshot is attached. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5339) Web UI: flaws in query profile display on error
[ https://issues.apache.org/jira/browse/DRILL-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162797#comment-16162797 ] Arina Ielchiieva commented on DRILL-5339: - For the point 5 separate Jira was created - DRILL-5782. > Web UI: flaws in query profile display on error > --- > > Key: DRILL-5339 > URL: https://issues.apache.org/jira/browse/DRILL-5339 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Fix For: 1.12.0 > > Attachments: error_shift.JPG, fragment_profiles.JPG, > verbose_error.JPG, visualized_plan.JPG > > > 1. Error is shifted to the right (screenshot - error_shift.jpg) > 2. When Fragment Profiles doesn't have any data table, there is offset in > overview tab (screenshot - fragment_profiles.jpg) > 3. Verbose error message tab can be designed like overview tab to be more > concise (screenshot - verbose_error.jpg) > 4. Remove three dots in the end of "Verbose error message..." . > 5. When visualized plan is empty, there is big offset (screenshot - > visualized_plan.jpg) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5782) Web UI: do not attempt build visualized plan when plan is absent
[ https://issues.apache.org/jira/browse/DRILL-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5782: Attachment: error_when_creating_visualized_plan.JPG > Web UI: do not attempt build visualized plan when plan is absent > > > Key: DRILL-5782 > URL: https://issues.apache.org/jira/browse/DRILL-5782 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Arina Ielchiieva > Attachments: error_when_creating_visualized_plan.JPG > > > When trying to execute query with incorrect syntax, there is an error on > query profile page (screenshot attached). Error occurs when we try to build > visualized plan when plan is absent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5782) Web UI: do not attempt build visualized plan when plan is absent
Arina Ielchiieva created DRILL-5782: --- Summary: Web UI: do not attempt build visualized plan when plan is absent Key: DRILL-5782 URL: https://issues.apache.org/jira/browse/DRILL-5782 Project: Apache Drill Issue Type: Bug Affects Versions: 1.11.0 Reporter: Arina Ielchiieva Attachments: error_when_creating_visualized_plan.JPG When trying to execute query with incorrect syntax, there is an error on query profile page (screenshot attached). Error occurs when we try to build visualized plan when plan is absent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5772) Add unit tests to indicate how utf-8 support can be enabled / disabled in Drill
[ https://issues.apache.org/jira/browse/DRILL-5772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163093#comment-16163093 ] ASF GitHub Bot commented on DRILL-5772: --- Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/936 @paul-rogers, the unit tests just show which influence saffron property has on Drill (since people in the community where asking how to enable support UTF-8 in Drill), a long with this PR we'll also add description to Drill documentation. So far Drill relies on Calcite saffron property to determine which charset it supports. By default, it's ISO-8859-1. So currently if customer wants to query UTF-8 data in Drill, he will get an error (one of the unit tests shows it) and needs to override saffron property to proceed. I guess, we don't support UTF-8 by default since there are some issues and Drill did not fully tested UTF-8 support. > Add unit tests to indicate how utf-8 support can be enabled / disabled in > Drill > --- > > Key: DRILL-5772 > URL: https://issues.apache.org/jira/browse/DRILL-5772 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.11.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Labels: doc-impacting > Fix For: 1.12.0 > > > Add unit test to indicated how utf-8 support can be enabled in Drill. > To select utf-8 data user needs to update system property > {{saffron.default.charset}} to {{UTF-16LE}} before starting the drillbit. > Calcite uses this property to get default charset, if it is not set then > {{ISO-8859-1}} is used by default. Drill gets default charset from Calcite. > This information should be also documented, probably in > https://drill.apache.org/docs/data-type-conversion/. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5772) Add unit tests to indicate how utf-8 support can be enabled / disabled in Drill
[ https://issues.apache.org/jira/browse/DRILL-5772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5772: Reviewer: Paul Rogers > Add unit tests to indicate how utf-8 support can be enabled / disabled in > Drill > --- > > Key: DRILL-5772 > URL: https://issues.apache.org/jira/browse/DRILL-5772 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.11.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Labels: doc-impacting > Fix For: 1.12.0 > > > Add unit test to indicated how utf-8 support can be enabled in Drill. > To select utf-8 data user needs to update system property > {{saffron.default.charset}} to {{UTF-16LE}} before starting the drillbit. > Calcite uses this property to get default charset, if it is not set then > {{ISO-8859-1}} is used by default. Drill gets default charset from Calcite. > This information should be also documented, probably in > https://drill.apache.org/docs/data-type-conversion/. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5782) Web UI: do not tattempt to build visualized plan when plan is absent
[ https://issues.apache.org/jira/browse/DRILL-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5782: Summary: Web UI: do not tattempt to build visualized plan when plan is absent (was: Web UI: do not attempt build visualized plan when plan is absent) > Web UI: do not tattempt to build visualized plan when plan is absent > > > Key: DRILL-5782 > URL: https://issues.apache.org/jira/browse/DRILL-5782 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Arina Ielchiieva > Attachments: error_when_creating_visualized_plan.JPG > > > When trying to execute query with incorrect syntax, there is an error on > query profile page (screenshot attached). Error occurs when we try to build > visualized plan when plan is absent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5782) Web UI: do not attempt to build visualized plan when plan is absent
[ https://issues.apache.org/jira/browse/DRILL-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5782: Summary: Web UI: do not attempt to build visualized plan when plan is absent (was: Web UI: do not tattempt to build visualized plan when plan is absent) > Web UI: do not attempt to build visualized plan when plan is absent > --- > > Key: DRILL-5782 > URL: https://issues.apache.org/jira/browse/DRILL-5782 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Arina Ielchiieva > Attachments: error_when_creating_visualized_plan.JPG > > > When trying to execute query with incorrect syntax, there is an error on > query profile page (screenshot attached). Error occurs when we try to build > visualized plan when plan is absent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5761) Disable Lilith ClassicMultiplexSocketAppender by default
[ https://issues.apache.org/jira/browse/DRILL-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163284#comment-16163284 ] ASF GitHub Bot commented on DRILL-5761: --- Github user vrozov commented on a diff in the pull request: https://github.com/apache/drill/pull/930#discussion_r138407993 --- Diff: common/src/test/resources/logback-test.xml --- @@ -0,0 +1,111 @@ + + + + + + + +true +1 +true +${LILITH_HOSTNAME:-localhost} + + + + + + + + %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n + + + + + + + --- End diff -- Can this logger be defined between line 21 and 22 in a single if condition? > Disable Lilith ClassicMultiplexSocketAppender by default > > > Key: DRILL-5761 > URL: https://issues.apache.org/jira/browse/DRILL-5761 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi > Labels: ready-to-commit > Fix For: 1.12.0 > > > When running unit tests on the node where Hiveserver2 service is running, > tests run hangs in the middle. Jstack shows that some threads are waiting for > a condition. > {noformat} > Full thread dump > "main" prio=10 tid=0x7f0998009800 nid=0x17f7 waiting on condition > [0x7f09a0c6d000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00076004ebf0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:324) > at > de.huxhorn.lilith.sender.MultiplexSendBytesService.sendBytes(MultiplexSendBytesService.java:132) > at > de.huxhorn.lilith.logback.appender.MultiplexSocketAppenderBase.sendBytes(MultiplexSocketAppenderBase.java:336) > at > de.huxhorn.lilith.logback.appender.MultiplexSocketAppenderBase.append(MultiplexSocketAppenderBase.java:348) > at > ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:88) > at > ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48) > at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:272) > at ch.qos.logback.classic.Logger.callAppenders(Logger.java:259) > at > ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:441) > at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:395) > at ch.qos.logback.classic.Logger.error(Logger.java:558) > at > org.apache.drill.test.DrillTest$TestLogReporter.failed(DrillTest.java:153) > at org.junit.rules.TestWatcher.failedQuietly(TestWatcher.java:84) > at org.junit.rules.TestWatcher.access$300(TestWatcher.java:46) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:62) > at > org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.junit.runners.Suite.runChild(Suite.java:127) > at org.junit.runners.Suite.runChild(Suite.java:26) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runner
[jira] [Commented] (DRILL-5761) Disable Lilith ClassicMultiplexSocketAppender by default
[ https://issues.apache.org/jira/browse/DRILL-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163288#comment-16163288 ] ASF GitHub Bot commented on DRILL-5761: --- Github user vrozov commented on a diff in the pull request: https://github.com/apache/drill/pull/930#discussion_r138408349 --- Diff: common/src/test/resources/logback-test.xml --- @@ -0,0 +1,92 @@ + + + + + + + +true +1 +true +${LILITH_HOSTNAME:-localhost} + + + + + + + + %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n + + + + > Key: DRILL-5761 > URL: https://issues.apache.org/jira/browse/DRILL-5761 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi > Labels: ready-to-commit > Fix For: 1.12.0 > > > When running unit tests on the node where Hiveserver2 service is running, > tests run hangs in the middle. Jstack shows that some threads are waiting for > a condition. > {noformat} > Full thread dump > "main" prio=10 tid=0x7f0998009800 nid=0x17f7 waiting on condition > [0x7f09a0c6d000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00076004ebf0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:324) > at > de.huxhorn.lilith.sender.MultiplexSendBytesService.sendBytes(MultiplexSendBytesService.java:132) > at > de.huxhorn.lilith.logback.appender.MultiplexSocketAppenderBase.sendBytes(MultiplexSocketAppenderBase.java:336) > at > de.huxhorn.lilith.logback.appender.MultiplexSocketAppenderBase.append(MultiplexSocketAppenderBase.java:348) > at > ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:88) > at > ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48) > at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:272) > at ch.qos.logback.classic.Logger.callAppenders(Logger.java:259) > at > ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:441) > at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:395) > at ch.qos.logback.classic.Logger.error(Logger.java:558) > at > org.apache.drill.test.DrillTest$TestLogReporter.failed(DrillTest.java:153) > at org.junit.rules.TestWatcher.failedQuietly(TestWatcher.java:84) > at org.junit.rules.TestWatcher.access$300(TestWatcher.java:46) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:62) > at > org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.junit.runners.Suite.runChild(Suite.java:127) > at org.junit.runners.Suite.runChild(Suite.java:26) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.junit.runner.JUnitCore.run(JUnitCore.java:16
[jira] [Commented] (DRILL-1162) 25 way join ended up with OOM
[ https://issues.apache.org/jira/browse/DRILL-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163295#comment-16163295 ] ASF GitHub Bot commented on DRILL-1162: --- Github user vvysotskyi commented on the issue: https://github.com/apache/drill/pull/905 @jinfengni thanks for looking into this. Completely agree with you that it would be better to consider both row and column count. Unfortunately, it does not help to fix this issue, since the ratio of column count of tables very often much smaller than the ratio of row count (actual row count). (`c1/c2 << r2/r1`) So I guess it makes sense to implement your proposal to consider the column count together with the row count in another pull request. > 25 way join ended up with OOM > - > > Key: DRILL-1162 > URL: https://issues.apache.org/jira/browse/DRILL-1162 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow, Query Planning & Optimization >Reporter: Rahul Challapalli >Assignee: Volodymyr Vysotskyi >Priority: Critical > Fix For: Future > > Attachments: error.log, oom_error.log > > > git.commit.id.abbrev=e5c2da0 > The below query results in 0 results being returned > {code:sql} > select count(*) from `lineitem1.parquet` a > inner join `part.parquet` j on a.l_partkey = j.p_partkey > inner join `orders.parquet` k on a.l_orderkey = k.o_orderkey > inner join `supplier.parquet` l on a.l_suppkey = l.s_suppkey > inner join `partsupp.parquet` m on j.p_partkey = m.ps_partkey and l.s_suppkey > = m.ps_suppkey > inner join `customer.parquet` n on k.o_custkey = n.c_custkey > inner join `lineitem2.parquet` b on a.l_orderkey = b.l_orderkey > inner join `lineitem2.parquet` c on a.l_partkey = c.l_partkey > inner join `lineitem2.parquet` d on a.l_suppkey = d.l_suppkey > inner join `lineitem2.parquet` e on a.l_extendedprice = e.l_extendedprice > inner join `lineitem2.parquet` f on a.l_comment = f.l_comment > inner join `lineitem2.parquet` g on a.l_shipdate = g.l_shipdate > inner join `lineitem2.parquet` h on a.l_commitdate = h.l_commitdate > inner join `lineitem2.parquet` i on a.l_receiptdate = i.l_receiptdate > inner join `lineitem2.parquet` o on a.l_receiptdate = o.l_receiptdate > inner join `lineitem2.parquet` p on a.l_receiptdate = p.l_receiptdate > inner join `lineitem2.parquet` q on a.l_receiptdate = q.l_receiptdate > inner join `lineitem2.parquet` r on a.l_receiptdate = r.l_receiptdate > inner join `lineitem2.parquet` s on a.l_receiptdate = s.l_receiptdate > inner join `lineitem2.parquet` t on a.l_receiptdate = t.l_receiptdate > inner join `lineitem2.parquet` u on a.l_receiptdate = u.l_receiptdate > inner join `lineitem2.parquet` v on a.l_receiptdate = v.l_receiptdate > inner join `lineitem2.parquet` w on a.l_receiptdate = w.l_receiptdate > inner join `lineitem2.parquet` x on a.l_receiptdate = x.l_receiptdate; > {code} > However when we remove the last 'inner join' and run the query it returns > '716372534'. Since the last inner join is similar to the one's before it, it > should match some records and return the data appropriately. > The logs indicated that it actually returned 0 results. Attached the log file. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5670) Varchar vector throws an assertion error when allocating a new vector
[ https://issues.apache.org/jira/browse/DRILL-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163297#comment-16163297 ] Paul Rogers commented on DRILL-5670: One more thought on this one. Rahul sometimes ran these tests after disabling exchanges. Perhaps try that here. If the sort passes, then we can file a bug for the exchange issue and start focusing on that specific problem. > Varchar vector throws an assertion error when allocating a new vector > - > > Key: DRILL-5670 > URL: https://issues.apache.org/jira/browse/DRILL-5670 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.11.0 >Reporter: Rahul Challapalli >Assignee: Paul Rogers > Fix For: 1.12.0 > > Attachments: 26498995-bbad-83bc-618f-914c37a84e1f.sys.drill, > 26555749-4d36-10d2-6faf-e403db40c370.sys.drill, > 266290f3-5fdc-5873-7372-e9ee053bf867.sys.drill, > 269969ca-8d4d-073a-d916-9031e3d3fbf0.sys.drill, drillbit.log, drillbit.log, > drillbit.log, drillbit.log, drillbit.log, drillbit.log.sort, drillbit.out, > drill-override.conf > > > I am running this test on a private branch of [paul's > repository|https://github.com/paul-rogers/drill]. Below is the commit info > {code} > git.commit.id.abbrev=d86e16c > git.commit.user.email=prog...@maprtech.com > git.commit.message.full=DRILL-5601\: Rollup of external sort fixes an > improvements\n\n- DRILL-5513\: Managed External Sort \: OOM error during the > merge phase\n- DRILL-5519\: Sort fails to spill and results in an OOM\n- > DRILL-5522\: OOM during the merge and spill process of the managed external > sort\n- DRILL-5594\: Excessive buffer reallocations during merge phase of > external sort\n- DRILL-5597\: Incorrect "bits" vector allocation in nullable > vectors allocateNew()\n- DRILL-5602\: Repeated List Vector fails to > initialize the offset vector\n\nAll of the bugs have to do with handling > low-memory conditions, and with\ncorrectly estimating the sizes of vectors, > even when those vectors come\nfrom the spill file or from an exchange. Hence, > the changes for all of\nthe above issues are interrelated.\n > git.commit.id=d86e16c551e7d3553f2cde748a739b1c5a7a7659 > git.commit.message.short=DRILL-5601\: Rollup of external sort fixes an > improvements > git.commit.user.name=Paul Rogers > git.build.user.name=Rahul Challapalli > git.commit.id.describe=0.9.0-1078-gd86e16c > git.build.user.email=challapallira...@gmail.com > git.branch=d86e16c551e7d3553f2cde748a739b1c5a7a7659 > git.commit.time=05.07.2017 @ 20\:34\:39 PDT > git.build.time=12.07.2017 @ 14\:27\:03 PDT > git.remote.origin.url=g...@github.com\:paul-rogers/drill.git > {code} > Below query fails with an Assertion Error > {code} > 0: jdbc:drill:zk=10.10.100.190:5181> ALTER SESSION SET > `exec.sort.disable_managed` = false; > +---+-+ > | ok | summary | > +---+-+ > | true | exec.sort.disable_managed updated. | > +---+-+ > 1 row selected (1.044 seconds) > 0: jdbc:drill:zk=10.10.100.190:5181> alter session set > `planner.memory.max_query_memory_per_node` = 482344960; > +---++ > | ok | summary | > +---++ > | true | planner.memory.max_query_memory_per_node updated. | > +---++ > 1 row selected (0.372 seconds) > 0: jdbc:drill:zk=10.10.100.190:5181> alter session set > `planner.width.max_per_node` = 1; > +---+--+ > | ok | summary| > +---+--+ > | true | planner.width.max_per_node updated. | > +---+--+ > 1 row selected (0.292 seconds) > 0: jdbc:drill:zk=10.10.100.190:5181> alter session set > `planner.width.max_per_query` = 1; > +---+---+ > | ok |summary| > +---+---+ > | true | planner.width.max_per_query updated. | > +---+---+ > 1 row selected (0.25 seconds) > 0: jdbc:drill:zk=10.10.100.190:5181> select count(*) from (select * from > dfs.`/drill/testdata/resource-manager/3500cols.tbl` order by > columns[450],columns[330],columns[230],columns[220],columns[110],columns[90],columns[80],columns[70],columns[40],columns[10],columns[20],columns[30],columns[40],columns[50], > > columns[454],columns[413],columns[940],columns[834],columns[73],columns[
[jira] [Commented] (DRILL-5338) Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile that they are part of total time duration
[ https://issues.apache.org/jira/browse/DRILL-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163418#comment-16163418 ] Paul Rogers commented on DRILL-5338: UI was reformatted and made clearer in DRILL-5716. > Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query > Profile that they are part of total time duration > --- > > Key: DRILL-5338 > URL: https://issues.apache.org/jira/browse/DRILL-5338 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: 1.12.0 > > Attachments: QueryProfile.JPG > > > Add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile > that they are part of total time duration > Screenshot is attached. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5783) Make code generation in the TopN operator more modular and test it
Timothy Farkas created DRILL-5783: - Summary: Make code generation in the TopN operator more modular and test it Key: DRILL-5783 URL: https://issues.apache.org/jira/browse/DRILL-5783 Project: Apache Drill Issue Type: Improvement Reporter: Timothy Farkas Assignee: Timothy Farkas -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5783) Make code generation in the TopN operator more modular and test it
[ https://issues.apache.org/jira/browse/DRILL-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163524#comment-16163524 ] Paul Rogers commented on DRILL-5783: Very good idea. A crude attempt to do something similar was done in the external sort work. (See the "managed" version of the {{ExternalSortBatch}}.) In the sort, we created a "wrapper" that generates the code and performs the operation. The sort has multiple chunks of generated code, so this was a useful approach. Ideally, we'd leverage this work to implement better code caching. Today, we use the source code as the key to the code cache. Since the source can be large (100s of K) and we need two copies (to erase unique class names), the memory needs add up. Better would be to define a class that contains all parameters needed to generate the code. Objects of that class, which should be much smaller than the code itself, would be the key to check for an existing matching definition. Regardless of the refactoring, consider reviewing the "managed" external sort tests for some useful tools: a {{RowSet}} abstraction that lets you quickly and easily set up input batches and verify output batches, a "sub-operator" test framework, and examples for how to test code generation separately from the full Drill server (that is, examples for how to break the tight coupling that one normally has to deal with.) > Make code generation in the TopN operator more modular and test it > -- > > Key: DRILL-5783 > URL: https://issues.apache.org/jira/browse/DRILL-5783 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163555#comment-16163555 ] ASF GitHub Bot commented on DRILL-5752: --- GitHub user ilooner opened a pull request: https://github.com/apache/drill/pull/940 DRILL-5752 Speed Up Unit Tests add Test Categories This PR does the following. **Increased Concurrency:** Previously tests only executed in 2 simultaneous processes, now one process per core is used to execute tests. This PR also fixed a few issues that popped up when concurrency was increased. **Test Categories:** Tests can now be executed according to their categories. There are two main categories of tests **Fast Tests** and **Secondary Tests**. **Fast Tests** are fast and are executed by default with the following command: ``` mvn -T 1C clean install ``` **Note:** -T 1C increased the number of threads used to compile classes, and allows us to run tests for sub modules in parallel. This command takes about 15 minutes to run on my laptop. **Secondary Tests** are slower. To run both fast tests and **Secondary Tests** you can use the following command. ``` mvn -T 1C clean install -DexcludedGroups="" ``` In addition to **Fast** and **Secondary** tests there are more categories like: - **JdbcTest** - **ParquetTest** - **HiveStorageTest** - And many more All of these categories are in the **drill-common** sub modules in **org.apahce.drill.categories**. All the tests are marked with one or more of these categories, and we can use these categories to select the tests we run. To run all the **Fast** tests that belong to the **JdbcTest** category we can use the following command: ``` mvn -T 1C clean install -Dgroups="org.apache.drill.categories.JdbcTest" ``` In order to run all the **Fast** AND **Secondary** JdbcTests you can use this command ``` mvn -T 1C clean install -Dgroups="org.apache.drill.categories.JdbcTest" -DexcludedGroups="" ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/ilooner/drill DRILL-5752 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/940.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #940 commit ba3d1ed53d7cabc5f5ac860531f6e989f567bc65 Author: Timothy Farkas Date: 2017-08-30T19:53:49Z - Removed redundant exclusion from java-exec pom - Marked the slow tests in contrib as secondary tests commit 9a3b7d2e1de85ce4397311e9ff2433432982fb03 Author: Timothy Farkas Date: 2017-08-30T22:02:23Z - Made slow unit tests SecondaryTests - Changed the number of forked test processes from just 2 processes to 1.5 process per core commit 4159bf8f3ad4966751b9176843e240bb4885cec5 Author: Timothy Farkas Date: 2017-08-31T00:10:38Z - Added JdbcTest category commit 0ab8ba8474b68bae8175f4b3c4a9b372f20eef8a Author: Timothy Farkas Date: 2017-08-31T00:24:03Z - Removed unused imports from tests commit bf3748c6ddd96b303762d8fde9046215cb243c02 Author: Timothy Farkas Date: 2017-08-31T00:34:22Z - Added JdbcStorageTest Category commit 749fa9b1c4ae113a010de33bf359a38a1d56833e Author: Timothy Farkas Date: 2017-08-31T18:15:18Z - Created the HiveStorageTest category - Made excludedGroups overrideable commit 50ddbb63f05eb08ac06d46dcf3c078e61dd3d042 Author: Timothy Farkas Date: 2017-08-31T18:39:28Z - Added HbaseStorageTest category commit 10097f61cb2df8898c2e7d8ff20cfb63fddcb6c5 Author: Timothy Farkas Date: 2017-08-31T18:50:33Z - Added the KuduStorageTest category commit 4dbad7edd5f479abc7db698e3ee2a2dad62abf20 Author: Timothy Farkas Date: 2017-08-31T19:06:34Z - Added the MongoStorageTest Category commit 5077b7beeccddcb0c84c70645c879cd3168f1bc1 Author: Timothy Farkas Date: 2017-08-31T19:42:09Z - Added a MemoryTest category commit a0e3832511041aeb18590d65ebb7166e40be045f Author: Timothy Farkas Date: 2017-08-31T19:53:42Z - Add another secondary test. commit efc9b8bb5af3ce891b890217ed1595d0db26cbbb Author: Timothy Farkas Date: 2017-08-31T20:03:08Z - Added a SecurityTest category - Made more of the security tests secondary tests commit 8dfacb0715252e063eca7d7a51895b8da191b273 Author: Timothy Farkas Date: 2017-08-31T20:05:05Z - Removed unused import for security test commit 72c3db7f1774fb0e37dd61a25ef1215f0391da63 Author: Timothy Farkas Date: 2017-08-31T20:05:57Z - Added another memory test commit 3fd24f8062887253a27bbdb8f31f12a06aa97270 Author: Timothy Farkas Date: 2017-08-31T20:22:14Z - Added an OperatorTest category commit 112
[jira] [Commented] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163581#comment-16163581 ] ASF GitHub Bot commented on DRILL-5752: --- Github user ilooner commented on the issue: https://github.com/apache/drill/pull/940 @paul-rogers This is an implementation of the Test Categories suggestion you made on the dev list. It's ready for review. > Speed Up Unit Tests > --- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Farkas updated DRILL-5752: -- Reviewer: Paul Rogers > Speed Up Unit Tests > --- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163720#comment-16163720 ] ASF GitHub Bot commented on DRILL-5752: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138466695 --- Diff: common/src/test/java/org/apache/drill/categories/package-info.java --- @@ -0,0 +1,23 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * This package stores the JUnit test categories. + */ --- End diff -- Very cool. One thing that is not intuitively obvious here is the purpose and usage of the categories. On the one hand, we already have modules for memory, vectors, Hive, etc. One can easily run just those tests by cd'ing into that module and running Maven from there. So, one would imagine categories to provide another dimension, orthogonal to modules. For example, the classic "fast" vs. "slow." One could also imagine a core set of "smoke" tests that run quickly and do a quick validation. Then a set of standard tests that are more thorough. Then, very in-depth tests that either take a long time, or validate very specific bugs or regressions. Can you add a bit of commentary to advise how you envision the categories to be used? > Speed Up Unit Tests > --- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163724#comment-16163724 ] ASF GitHub Bot commented on DRILL-5752: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138472376 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestCorruptParquetDateCorrection.java --- @@ -59,6 +61,7 @@ * Use of this option is assumed to be extremely unlikely, but it is included * for completeness. */ +@Category(ParquetTest.class) --- End diff -- Secondary? Would not seem that this test need to be run on each build... > Speed Up Unit Tests > --- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163721#comment-16163721 ] ASF GitHub Bot commented on DRILL-5752: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138475152 --- Diff: exec/jdbc/src/test/java/org/apache/drill/jdbc/ConnectionTest.java --- @@ -57,7 +61,9 @@ public static void setUpConnection() throws SQLException { // Connection--and other JDBC objects--on test method failure, but this test // class uses some objects across methods.) Driver.load(); -connection = DriverManager.getConnection( "jdbc:drill:zk=local" ); +Properties properties = new Properties(); +properties.setProperty(OptionValidator.OPTION_DEFAULTS_ROOT + ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS, "3"); --- End diff -- Also, since properties don't have to be strings, why is this property stored as a string when the value is an int? > Speed Up Unit Tests > --- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163725#comment-16163725 ] ASF GitHub Bot commented on DRILL-5752: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138473108 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/xsort/TestSimpleExternalSort.java --- @@ -148,10 +149,10 @@ public void sortOneKeyDescendingExternalSortLegacy() throws Throwable { private void sortOneKeyDescendingExternalSort(boolean testLegacy) throws Throwable { FixtureBuilder builder = ClusterFixture.builder() -.configProperty(ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 4) -.configProperty(ExecConstants.EXTERNAL_SORT_SPILL_GROUP_SIZE, 4) -.configProperty(ExecConstants.EXTERNAL_SORT_BATCH_LIMIT, 4) -.configProperty(ExecConstants.EXTERNAL_SORT_DISABLE_MANAGED, false) +.configProperty(OptionValidator.OPTION_DEFAULTS_ROOT + ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 4) --- End diff -- These options are boot options, not runtime options, so no need for the prefix (unless someone changed something.) > Speed Up Unit Tests > --- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163723#comment-16163723 ] ASF GitHub Bot commented on DRILL-5752: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138475004 --- Diff: exec/jdbc/src/test/java/org/apache/drill/jdbc/ConnectionTest.java --- @@ -57,7 +61,9 @@ public static void setUpConnection() throws SQLException { // Connection--and other JDBC objects--on test method failure, but this test // class uses some objects across methods.) Driver.load(); -connection = DriverManager.getConnection( "jdbc:drill:zk=local" ); +Properties properties = new Properties(); +properties.setProperty(OptionValidator.OPTION_DEFAULTS_ROOT + ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS, "3"); --- End diff -- I wonder, should we define a function for this pattern? ``` ExecConstants.bootDefaultFor( ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS) ``` But, we use Java 7, so the static function must reside elsewhere. Actually, the function might exist; I seem to recall making the same comment in the PR that externalized the defaults. > Speed Up Unit Tests > --- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163722#comment-16163722 ] ASF GitHub Bot commented on DRILL-5752: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138472834 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java --- @@ -54,11 +56,13 @@ import org.junit.Ignore; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.TemporaryFolder; import org.junit.runner.RunWith; import org.junit.runners.Parameterized; @RunWith(Parameterized.class) +@Category({SecondaryTest.class, ParquetTest.class}) --- End diff -- On the other hand, the Parquet writer seems pretty important and we'd want to test it more often. This is why, in my earlier comment, I asked about the purpose of categorization: those that are worth running frequently ("smoke tests") vs. those that give a more thorough test, but will typically not find issues. > Speed Up Unit Tests > --- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5704) Improve error message on client side when queries fail with "Failed to create schema tree." when Impersonation is enabled and logins are anonymous
[ https://issues.apache.org/jira/browse/DRILL-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163749#comment-16163749 ] ASF GitHub Bot commented on DRILL-5704: --- Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/895 @sohami, please rebase on latest master and resolve conflicts. > Improve error message on client side when queries fail with "Failed to create > schema tree." when Impersonation is enabled and logins are anonymous > -- > > Key: DRILL-5704 > URL: https://issues.apache.org/jira/browse/DRILL-5704 > Project: Apache Drill > Issue Type: Improvement >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia > Labels: ready-to-commit > Fix For: 1.12.0 > > > Reported by [~agirish] > When username is not specified then Drill set's the session user as anonymous > if impersonation is enabled. During query execution Drill tries to build > schema tree and as part of that it validates if the user has access to the > workspace or not by using FileClient Api liststatus which verifies the user > from the OS user. Since impersonation is only enabled here without > authentication and we don't specify any user in connection string, Drill will > use default user which is "anonymous" and pass that to check workspace > permission which will fail as node doesn't have any valid user with that name. > {code:java} > Caused by: java.io.IOException: Error getting user info for current user, > anonymous >.. >.. > at > org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus(DrillFileSystem.java:523) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible(WorkspaceSchemaFactory.java:157) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.(FileSystemSchemaFactory.java:78) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas(FileSystemSchemaFactory.java:65) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas(FileSystemPlugin.java:150) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > ... 10 common frames omitted > {code} > # $DRILL_HOME/bin/sqlline -u "jdbc:drill:zk=localhost:5181" > sqlline> select * from sys.drillbits; > User Error Occurred > org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Failed to > create schema tree. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163823#comment-16163823 ] ASF GitHub Bot commented on DRILL-5752: --- Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138489633 --- Diff: common/src/test/java/org/apache/drill/categories/package-info.java --- @@ -0,0 +1,23 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * This package stores the JUnit test categories. + */ --- End diff -- Currently there are two broad categories: **Fast** and **Secondary**. The **Fast** tests are smoke tests. The **Secondary** tests are for slow tests. I can add a third **BugFix**category to represent tests for specific bugs. The rest of the categories represent broad groups of tests. Some of those categories have tests which span multiple class files and multiple packages. An example of such a category is **OperatorTest**, **PlannerTest**, or **SqlTest**. These categories are helpful when you are making a change in a particular area since they allow you to run all the tests for that area easily, and also effectively serve as a form of documentation. For example it would have been helpful to have an OptionsTest category when I was making the option system changes, since it took me a while to find all the relevant test classes and I had to construct a long maven command to run the tests. Also the structure of the tests as they are now is somewhat counter intuitive because tests can span different modules. For example consider the vector submodule. One would think all the relevant tests for vectors are in that submodule, but currently they are not. In fact there are no unit tests in the vector submodule and all relevant vector tests are in java-exec. Also the memory submodule has tests in it and a relevant test in the java-exec module as well. Some categories I agree are redundant like HiveStorageTest. Those tests have a clear one to one mapping to their submodule, however, I included a category for them as well for the sake of consistency. This way we can have a uniform mechanism for running subsets of tests instead of one mechanism for some groups of tests and another mechanism for other tests. To summarize, I can add a **BugFixTest** category to add another level of distinction between **Secondary** tests. But I am compelled to keep the other test categories intact for the reasons outlined above. Let me know what you think. > Speed Up Unit Tests > --- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5694) hash agg spill to disk, second phase OOM
[ https://issues.apache.org/jira/browse/DRILL-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163858#comment-16163858 ] ASF GitHub Bot commented on DRILL-5694: --- Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/938#discussion_r138492663 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -646,6 +687,46 @@ public AggOutcome doWork() { } /** + * Use reserved values memory (if available) to try and preemp an OOM + */ + private void useReservedValuesMemory() { +// try to preempt an OOM by using the reserved memory +long reservedMemory = reserveValueBatchMemory; +if ( reservedMemory > 0 ) { allocator.setLimit(allocator.getLimit() + reservedMemory); } + +reserveValueBatchMemory = 0; + } + /** + * Use reserved outgoing output memory (if available) to try and preemp an OOM + */ + private void useReservedOutgoingMemory() { +// try to preempt an OOM by using the reserved memory +long reservedMemory = reserveOutgoingMemory; +if ( reservedMemory > 0 ) { allocator.setLimit(allocator.getLimit() + reservedMemory); } + +reserveOutgoingMemory = 0; + } + /** + * Restore the reserve memory (both) + * + */ + private void restoreReservedMemory() { +if ( 0 == reserveOutgoingMemory ) { // always restore OutputValues first (needed for spilling) + long memAvail = allocator.getLimit() - allocator.getAllocatedMemory(); + if ( memAvail > estOutgoingAllocSize) { +allocator.setLimit(allocator.getLimit() - estOutgoingAllocSize); --- End diff -- Can never add twice, as the code only adds to an empty ( == 0 ) reserve. And these are not "relative deltas", but the actual expected batch size (so that the following allocation would not OOM). > hash agg spill to disk, second phase OOM > > > Key: DRILL-5694 > URL: https://issues.apache.org/jira/browse/DRILL-5694 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.11.0 >Reporter: Chun Chang >Assignee: Boaz Ben-Zvi > > | 1.11.0-SNAPSHOT | d622f76ee6336d97c9189fc589befa7b0f4189d6 | DRILL-5165: > For limit all case, no need to push down limit to scan | 21.07.2017 @ > 10:36:29 PDT > Second phase agg ran out of memory. Not suppose to. Test data currently only > accessible locally. > /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg15.q > Query: > select row_count, sum(row_count), avg(double_field), max(double_rand), > count(float_rand) from parquet_500m_v1 group by row_count order by row_count > limit 30 > Failed with exception > java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory > while executing the query. > HT was: 534773760 OOM at Second Phase. Partitions: 32. Estimated batch size: > 4849664. Planned batches: 0. Rows spilled so far: 6459928 Memory limit: > 536870912 so far allocated: 534773760. > Fragment 1:6 > [Error Id: a193babd-f783-43da-a476-bb8dd4382420 on 10.10.30.168:31010] > (org.apache.drill.exec.exception.OutOfMemoryException) HT was: 534773760 > OOM at Second Phase. Partitions: 32. Estimated batch size: 4849664. Planned > batches: 0. Rows spilled so far: 6459928 Memory limit: 536870912 so far > allocated: 534773760. > > org.apache.drill.exec.test.generated.HashAggregatorGen1823.checkGroupAndAggrValues():1175 > org.apache.drill.exec.test.generated.HashAggregatorGen1823.doWork():539 > org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51 > > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93 > org.apache.drill.exec.record.AbstractRecordBatc
[jira] [Commented] (DRILL-5694) hash agg spill to disk, second phase OOM
[ https://issues.apache.org/jira/browse/DRILL-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163880#comment-16163880 ] ASF GitHub Bot commented on DRILL-5694: --- Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/938#discussion_r138494777 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -1178,20 +1273,38 @@ private void checkGroupAndAggrValues(int incomingRowIdx) { hashCode >>>= bitsInMask; HashTable.PutStatus putStatus = null; long allocatedBeforeHTput = allocator.getAllocatedMemory(); - // == // Insert the key columns into the hash table // == +boolean noReserveMem = reserveValueBatchMemory == 0; try { + if ( noReserveMem && canSpill ) { throw new RetryAfterSpillException();} // proactive spill, skip put() + putStatus = htables[currentPartition].put(incomingRowIdx, htIdxHolder, hashCode); + +} catch (RetryAfterSpillException re) { --- End diff -- Done. > hash agg spill to disk, second phase OOM > > > Key: DRILL-5694 > URL: https://issues.apache.org/jira/browse/DRILL-5694 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.11.0 >Reporter: Chun Chang >Assignee: Boaz Ben-Zvi > > | 1.11.0-SNAPSHOT | d622f76ee6336d97c9189fc589befa7b0f4189d6 | DRILL-5165: > For limit all case, no need to push down limit to scan | 21.07.2017 @ > 10:36:29 PDT > Second phase agg ran out of memory. Not suppose to. Test data currently only > accessible locally. > /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg15.q > Query: > select row_count, sum(row_count), avg(double_field), max(double_rand), > count(float_rand) from parquet_500m_v1 group by row_count order by row_count > limit 30 > Failed with exception > java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory > while executing the query. > HT was: 534773760 OOM at Second Phase. Partitions: 32. Estimated batch size: > 4849664. Planned batches: 0. Rows spilled so far: 6459928 Memory limit: > 536870912 so far allocated: 534773760. > Fragment 1:6 > [Error Id: a193babd-f783-43da-a476-bb8dd4382420 on 10.10.30.168:31010] > (org.apache.drill.exec.exception.OutOfMemoryException) HT was: 534773760 > OOM at Second Phase. Partitions: 32. Estimated batch size: 4849664. Planned > batches: 0. Rows spilled so far: 6459928 Memory limit: 536870912 so far > allocated: 534773760. > > org.apache.drill.exec.test.generated.HashAggregatorGen1823.checkGroupAndAggrValues():1175 > org.apache.drill.exec.test.generated.HashAggregatorGen1823.doWork():539 > org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51 > > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.physical.impl.BaseRootExec.next():105 > > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92 > org.apache.drill.exec.physical.impl.BaseRootExec.next():95 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227 > java.security.AccessController.doPrivileged():-2 > javax.security.auth.Subject.doAs():415 > org.apache.hadoop.security.UserGroupInformation.doAs():1595 > org.apache.drill.exec.work.fragment.FragmentExecutor.run():227 > org.apache.drill.common.SelfCleaningRunnable.run():38 > java.util.concurrent.ThreadPoolExecutor.runWorker():1145 > java.util.concurrent.ThreadPoolExecutor$Wo
[jira] [Commented] (DRILL-5694) hash agg spill to disk, second phase OOM
[ https://issues.apache.org/jira/browse/DRILL-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163883#comment-16163883 ] ASF GitHub Bot commented on DRILL-5694: --- Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/938#discussion_r138495164 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -1178,20 +1273,38 @@ private void checkGroupAndAggrValues(int incomingRowIdx) { hashCode >>>= bitsInMask; HashTable.PutStatus putStatus = null; long allocatedBeforeHTput = allocator.getAllocatedMemory(); - // == // Insert the key columns into the hash table // == +boolean noReserveMem = reserveValueBatchMemory == 0; try { + if ( noReserveMem && canSpill ) { throw new RetryAfterSpillException();} // proactive spill, skip put() + putStatus = htables[currentPartition].put(incomingRowIdx, htIdxHolder, hashCode); + +} catch (RetryAfterSpillException re) { + if ( ! canSpill ) { throw new OutOfMemoryException(getOOMErrorMsg("Can not spill")); } --- End diff -- The method getOOMErrorMsg() does all this explanation ... > hash agg spill to disk, second phase OOM > > > Key: DRILL-5694 > URL: https://issues.apache.org/jira/browse/DRILL-5694 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.11.0 >Reporter: Chun Chang >Assignee: Boaz Ben-Zvi > > | 1.11.0-SNAPSHOT | d622f76ee6336d97c9189fc589befa7b0f4189d6 | DRILL-5165: > For limit all case, no need to push down limit to scan | 21.07.2017 @ > 10:36:29 PDT > Second phase agg ran out of memory. Not suppose to. Test data currently only > accessible locally. > /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg15.q > Query: > select row_count, sum(row_count), avg(double_field), max(double_rand), > count(float_rand) from parquet_500m_v1 group by row_count order by row_count > limit 30 > Failed with exception > java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory > while executing the query. > HT was: 534773760 OOM at Second Phase. Partitions: 32. Estimated batch size: > 4849664. Planned batches: 0. Rows spilled so far: 6459928 Memory limit: > 536870912 so far allocated: 534773760. > Fragment 1:6 > [Error Id: a193babd-f783-43da-a476-bb8dd4382420 on 10.10.30.168:31010] > (org.apache.drill.exec.exception.OutOfMemoryException) HT was: 534773760 > OOM at Second Phase. Partitions: 32. Estimated batch size: 4849664. Planned > batches: 0. Rows spilled so far: 6459928 Memory limit: 536870912 so far > allocated: 534773760. > > org.apache.drill.exec.test.generated.HashAggregatorGen1823.checkGroupAndAggrValues():1175 > org.apache.drill.exec.test.generated.HashAggregatorGen1823.doWork():539 > org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51 > > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.physical.impl.BaseRootExec.next():105 > > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92 > org.apache.drill.exec.physical.impl.BaseRootExec.next():95 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227 > java.security.AccessController.doPrivileged():-2 > javax.security.auth.Subject.doAs():415 > org.apache.hadoop.security.UserGroupInformation.doAs():1595 > org.apache.drill.exec.work.fragment.FragmentExecutor.run():227 > org.apache.dril
[jira] [Commented] (DRILL-5694) hash agg spill to disk, second phase OOM
[ https://issues.apache.org/jira/browse/DRILL-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163887#comment-16163887 ] ASF GitHub Bot commented on DRILL-5694: --- Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/938#discussion_r138495296 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTable.java --- @@ -58,7 +59,7 @@ public int getHashCode(int incomingRowIdx) throws SchemaChangeException; - public PutStatus put(int incomingRowIdx, IndexPointer htIdxHolder, int hashCode) throws SchemaChangeException; + public PutStatus put(int incomingRowIdx, IndexPointer htIdxHolder, int hashCode) throws SchemaChangeException, RetryAfterSpillException; --- End diff -- Done. > hash agg spill to disk, second phase OOM > > > Key: DRILL-5694 > URL: https://issues.apache.org/jira/browse/DRILL-5694 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.11.0 >Reporter: Chun Chang >Assignee: Boaz Ben-Zvi > > | 1.11.0-SNAPSHOT | d622f76ee6336d97c9189fc589befa7b0f4189d6 | DRILL-5165: > For limit all case, no need to push down limit to scan | 21.07.2017 @ > 10:36:29 PDT > Second phase agg ran out of memory. Not suppose to. Test data currently only > accessible locally. > /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg15.q > Query: > select row_count, sum(row_count), avg(double_field), max(double_rand), > count(float_rand) from parquet_500m_v1 group by row_count order by row_count > limit 30 > Failed with exception > java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory > while executing the query. > HT was: 534773760 OOM at Second Phase. Partitions: 32. Estimated batch size: > 4849664. Planned batches: 0. Rows spilled so far: 6459928 Memory limit: > 536870912 so far allocated: 534773760. > Fragment 1:6 > [Error Id: a193babd-f783-43da-a476-bb8dd4382420 on 10.10.30.168:31010] > (org.apache.drill.exec.exception.OutOfMemoryException) HT was: 534773760 > OOM at Second Phase. Partitions: 32. Estimated batch size: 4849664. Planned > batches: 0. Rows spilled so far: 6459928 Memory limit: 536870912 so far > allocated: 534773760. > > org.apache.drill.exec.test.generated.HashAggregatorGen1823.checkGroupAndAggrValues():1175 > org.apache.drill.exec.test.generated.HashAggregatorGen1823.doWork():539 > org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51 > > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.physical.impl.BaseRootExec.next():105 > > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92 > org.apache.drill.exec.physical.impl.BaseRootExec.next():95 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227 > java.security.AccessController.doPrivileged():-2 > javax.security.auth.Subject.doAs():415 > org.apache.hadoop.security.UserGroupInformation.doAs():1595 > org.apache.drill.exec.work.fragment.FragmentExecutor.run():227 > org.apache.drill.common.SelfCleaningRunnable.run():38 > java.util.concurrent.ThreadPoolExecutor.runWorker():1145 > java.util.concurrent.ThreadPoolExecutor$Worker.run():615 > java.lang.Thread.run():745 > Caused By (org.apache.drill.exec.exception.OutOfMemoryException) Unable to > allocate buffer of size 4194304 due to memory limit. Current allocation: > 534773760 > org.apache.drill.exec.memory.BaseAllocator.buffer():238 > org.apache.drill.exec.memory.BaseAllocator.buffer():213 > org.apache.drill.exec.vector.IntVector.alloc
[jira] [Commented] (DRILL-5694) hash agg spill to disk, second phase OOM
[ https://issues.apache.org/jira/browse/DRILL-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163892#comment-16163892 ] ASF GitHub Bot commented on DRILL-5694: --- Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/938#discussion_r138495914 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java --- @@ -158,19 +158,17 @@ public BatchHolder(int idx) { } finally { if (!success) { htContainer.clear(); - if (links != null) { -links.clear(); - } + if (links != null) { links.clear();} } } } private void init(IntVector links, IntVector hashValues, int size) { for (int i = 0; i < size; i++) { -links.getMutator().setSafe(i, EMPTY_SLOT); +links.getMutator().set(i, EMPTY_SLOT); --- End diff -- This init() method is not used looks like leftover old code > hash agg spill to disk, second phase OOM > > > Key: DRILL-5694 > URL: https://issues.apache.org/jira/browse/DRILL-5694 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.11.0 >Reporter: Chun Chang >Assignee: Boaz Ben-Zvi > > | 1.11.0-SNAPSHOT | d622f76ee6336d97c9189fc589befa7b0f4189d6 | DRILL-5165: > For limit all case, no need to push down limit to scan | 21.07.2017 @ > 10:36:29 PDT > Second phase agg ran out of memory. Not suppose to. Test data currently only > accessible locally. > /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg15.q > Query: > select row_count, sum(row_count), avg(double_field), max(double_rand), > count(float_rand) from parquet_500m_v1 group by row_count order by row_count > limit 30 > Failed with exception > java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory > while executing the query. > HT was: 534773760 OOM at Second Phase. Partitions: 32. Estimated batch size: > 4849664. Planned batches: 0. Rows spilled so far: 6459928 Memory limit: > 536870912 so far allocated: 534773760. > Fragment 1:6 > [Error Id: a193babd-f783-43da-a476-bb8dd4382420 on 10.10.30.168:31010] > (org.apache.drill.exec.exception.OutOfMemoryException) HT was: 534773760 > OOM at Second Phase. Partitions: 32. Estimated batch size: 4849664. Planned > batches: 0. Rows spilled so far: 6459928 Memory limit: 536870912 so far > allocated: 534773760. > > org.apache.drill.exec.test.generated.HashAggregatorGen1823.checkGroupAndAggrValues():1175 > org.apache.drill.exec.test.generated.HashAggregatorGen1823.doWork():539 > org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51 > > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.physical.impl.BaseRootExec.next():105 > > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92 > org.apache.drill.exec.physical.impl.BaseRootExec.next():95 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227 > java.security.AccessController.doPrivileged():-2 > javax.security.auth.Subject.doAs():415 > org.apache.hadoop.security.UserGroupInformation.doAs():1595 > org.apache.drill.exec.work.fragment.FragmentExecutor.run():227 > org.apache.drill.common.SelfCleaningRunnable.run():38 > java.util.concurrent.ThreadPoolExecutor.runWorker():1145 > java.util.concurrent.ThreadPoolExecutor$Worker.run():615 > java.lang.Thread.run():745 > Caused By (org.apache.drill.exec.exception.OutOfMemoryException) Unable to > allocate buffe
[jira] [Created] (DRILL-5784) SYSTEM ERROR: IndexOutOfBoundsException: index: 512, length: 4 (expected: range(0, 512))
Vlad Rozov created DRILL-5784: - Summary: SYSTEM ERROR: IndexOutOfBoundsException: index: 512, length: 4 (expected: range(0, 512)) Key: DRILL-5784 URL: https://issues.apache.org/jira/browse/DRILL-5784 Project: Apache Drill Issue Type: Bug Environment: planner.slice_target > 10 planner.enable_nljoin_for_scalar_only = false Reporter: Vlad Rozov Assignee: Vlad Rozov The following query causes IndexOutOfBoundsException: {code} SELECT `t1`.`one` `one` FROM ( SELECT 1 `one` FROM dfs.`/drill/exec/java-exec/src/test/resources/join/j1` INNER JOIN ( SELECT 314 `c_integer` FROM dfs.`/drill/exec/java-exec/src/test/resources/join/j1` ) `t0` ON ( `/drill/exec/java-exec/src/test/resources/join/j1`.c_integer IS NOT DISTINCT FROM `t0`.`c_integer` ) GROUP BY `one` ) `t1` INNER JOIN ( SELECT count(1) `measure` FROM dfs.`/drill/exec/java-exec/src/test/resources/join/j1` ) `t5` ON TRUE {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163909#comment-16163909 ] ASF GitHub Bot commented on DRILL-5752: --- Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138497914 --- Diff: exec/jdbc/src/test/java/org/apache/drill/jdbc/ConnectionTest.java --- @@ -57,7 +61,9 @@ public static void setUpConnection() throws SQLException { // Connection--and other JDBC objects--on test method failure, but this test // class uses some objects across methods.) Driver.load(); -connection = DriverManager.getConnection( "jdbc:drill:zk=local" ); +Properties properties = new Properties(); +properties.setProperty(OptionValidator.OPTION_DEFAULTS_ROOT + ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS, "3"); --- End diff -- The Properties class only allows strings to be set as values. The conversion to an integer is done somewhere inside the DriverManager. > Speed Up Unit Tests > --- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163907#comment-16163907 ] ASF GitHub Bot commented on DRILL-5752: --- Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138497894 --- Diff: exec/jdbc/src/test/java/org/apache/drill/jdbc/ConnectionTest.java --- @@ -57,7 +61,9 @@ public static void setUpConnection() throws SQLException { // Connection--and other JDBC objects--on test method failure, but this test // class uses some objects across methods.) Driver.load(); -connection = DriverManager.getConnection( "jdbc:drill:zk=local" ); +Properties properties = new Properties(); +properties.setProperty(OptionValidator.OPTION_DEFAULTS_ROOT + ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS, "3"); --- End diff -- I made ExecConstants a file class with a private constructor and added the static method there. > Speed Up Unit Tests > --- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5550) SELECT non-existent column produces empty required VARCHAR
[ https://issues.apache.org/jira/browse/DRILL-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163919#comment-16163919 ] ASF GitHub Bot commented on DRILL-5550: --- Github user prasadns14 commented on the issue: https://github.com/apache/drill/pull/939 @paul-rogers Can you please review the PR? > SELECT non-existent column produces empty required VARCHAR > -- > > Key: DRILL-5550 > URL: https://issues.apache.org/jira/browse/DRILL-5550 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Text & CSV >Affects Versions: 1.10.0 >Reporter: Paul Rogers >Priority: Minor > > Drill's CSV column reader supports two forms of files: > * Files with column headers as the first line of the file. > * Files without column headers. > The CSV storage plugin specifies which format to use for files accessed via > that storage plugin config. > Suppose we have a CSV file with headers: > {code} > a,b,c > 10,foo,bar > {code} > Suppose we configure a storage plugin to use headers: > {code} > TextFormatConfig csvFormat = new TextFormatConfig(); > csvFormat.fieldDelimiter = ','; > csvFormat.skipFirstLine = false; > csvFormat.extractHeader = true; > {code} > (The above can also be done using JSON when running Drill as a server.) > Execute the following query: > {code} > SELECT a, c, d FROM `dfs.data.example.csv` > {code} > Results: > {code} > a,c,d > 10,bar, > {code} > The actual type of column {{d}} is non-nullable VARCHAR. > This is inconsistent with other parts of Drill in two ways, one may be a bug. > Most other parts of Drill use a nullable INT for "missing" columns. > 1. For CSV it makes sense for the data type to be VARCHAR, since all CSV > columns are of that type. > 2. It may *not* make sense for the column to be non-nullable and blank rather > than nullable and NULL. In SQL, NULL means that the data is unknown, which is > the case here. > In the future, we may want to use some other indication for a missing column. > Until then, the requested change is to make the type of a missing CSV column > a nullable VARCHAR set to value NULL. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5532) TPCH Query 16 failed when planner.width.max_per_node <10
[ https://issues.apache.org/jira/browse/DRILL-5532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163928#comment-16163928 ] Jinfeng Ni commented on DRILL-5532: --- This seems to be exposing a problem in execution time, after the commit 841ead40109ff8364bee640a77881a8fea94d152 got a new query plan for the query. > TPCH Query 16 failed when planner.width.max_per_node <10 > > > Key: DRILL-5532 > URL: https://issues.apache.org/jira/browse/DRILL-5532 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.11.0 > Environment: 10 node cluster, rhel6.4 >Reporter: Dechang Gu >Assignee: Jinfeng Ni > Attachments: > pfofile_tpch16_gitid_8412ad4.26dce567-5ec0-8bbf-c099-f2d6307bdeef.sys.drill, > profile_tpch16_gitid_adbf363.26dce6f3-3d75-e906-bfcb-2c6541ee2ff0.sys.drill > > > When set planner.width.max_per_node < 10 (tried 6, 8, and 9): TPCH query 16 > failed with the following error: > {code} > java.sql.SQLException: SYSTEM ERROR: NullPointerException > Fragment 1:1 > [Error Id: ba4661b4-2ff6-4eff-889c-fc4604a57418 on ucs-node8.perf.lab:31010] > (java.lang.NullPointerException) null > org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe():607 > org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe():560 > > org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputRecordKeysPrev():525 > > org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputToBatchPrev():322 > > org.apache.drill.exec.test.generated.StreamingAggregatorGen250.doWork():182 > > org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():170 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.physical.impl.BaseRootExec.next():105 > > org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():144 > org.apache.drill.exec.physical.impl.BaseRootExec.next():95 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227 > java.security.AccessController.doPrivileged():-2 > javax.security.auth.Subject.doAs():415 > org.apache.hadoop.security.UserGroupInformation.doAs():1595 > org.apache.drill.exec.work.fragment.FragmentExecutor.run():227 > org.apache.drill.common.SelfCleaningRunnable.run():38 > java.util.concurrent.ThreadPoolExecutor.runWorker():1145 > java.util.concurrent.ThreadPoolExecutor$Worker.run():615 > java.lang.Thread.run():745 > at > org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489) > at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:593) > at > org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:215) > at > org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:140) > at PipSQueak.fetchRows(PipSQueak.java:420) > at PipSQueak.runTest(PipSQueak.java:116) > at PipSQueak.main(PipSQueak.java:556) > Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM > ERROR: NullPointerException > Fragment 1:1 > [Error Id: ba4661b4-2ff6-4eff-889c-fc4604a57418 on ucs-node8.perf.lab:31010] > (java.lang.NullPointerException) null > org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe():607 > org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe():560 > > org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputRecordKeysPrev():525 > > org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputToBatchPrev():322 > > org.apache.drill.exec.test.generated.StreamingAggregatorGen250.doWork():182 > > org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():170 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.physical.impl.BaseRootExec.next():105 > > org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():144 > org.apache.drill.exec.physical.impl.BaseRootExec.next():95 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227 > java.security.AccessController.doPrivileged():-2 > javax.security.auth.Subject.doAs():415 > org.apache.hadoop.security.UserGroupInformation.doAs():1595 > org.apache.drill.exec.work.fragment.FragmentExecutor.run():227 > org.apache.drill.common.SelfCleaningRunnable.run():38 > java.util.concurrent.ThreadPoolExecutor.runWorker():1145 > java.util.concurrent.ThreadPoolExecutor$Worker.run():615 > java.lang.Thread.run():745 > at > org.apache.drill.exec.rpc.user.Qu
[jira] [Commented] (DRILL-5532) TPCH Query 16 failed when planner.width.max_per_node <10
[ https://issues.apache.org/jira/browse/DRILL-5532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163935#comment-16163935 ] Jinfeng Ni commented on DRILL-5532: --- DRILL-5468 is a similar issue, where the commit 841ead40109ff8364bee640a77881a8fea94d152 got a new plan for Q18, which degrades the performance. However, after analysis, the new code is doing the right thing, and previously we are just lucky to have a "better" plan. For this issue, it's possible that the new plan exposes a NPE bug in execution code. > TPCH Query 16 failed when planner.width.max_per_node <10 > > > Key: DRILL-5532 > URL: https://issues.apache.org/jira/browse/DRILL-5532 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.11.0 > Environment: 10 node cluster, rhel6.4 >Reporter: Dechang Gu >Assignee: Jinfeng Ni > Attachments: > pfofile_tpch16_gitid_8412ad4.26dce567-5ec0-8bbf-c099-f2d6307bdeef.sys.drill, > profile_tpch16_gitid_adbf363.26dce6f3-3d75-e906-bfcb-2c6541ee2ff0.sys.drill > > > When set planner.width.max_per_node < 10 (tried 6, 8, and 9): TPCH query 16 > failed with the following error: > {code} > java.sql.SQLException: SYSTEM ERROR: NullPointerException > Fragment 1:1 > [Error Id: ba4661b4-2ff6-4eff-889c-fc4604a57418 on ucs-node8.perf.lab:31010] > (java.lang.NullPointerException) null > org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe():607 > org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe():560 > > org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputRecordKeysPrev():525 > > org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputToBatchPrev():322 > > org.apache.drill.exec.test.generated.StreamingAggregatorGen250.doWork():182 > > org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():170 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.physical.impl.BaseRootExec.next():105 > > org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():144 > org.apache.drill.exec.physical.impl.BaseRootExec.next():95 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227 > java.security.AccessController.doPrivileged():-2 > javax.security.auth.Subject.doAs():415 > org.apache.hadoop.security.UserGroupInformation.doAs():1595 > org.apache.drill.exec.work.fragment.FragmentExecutor.run():227 > org.apache.drill.common.SelfCleaningRunnable.run():38 > java.util.concurrent.ThreadPoolExecutor.runWorker():1145 > java.util.concurrent.ThreadPoolExecutor$Worker.run():615 > java.lang.Thread.run():745 > at > org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489) > at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:593) > at > org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:215) > at > org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:140) > at PipSQueak.fetchRows(PipSQueak.java:420) > at PipSQueak.runTest(PipSQueak.java:116) > at PipSQueak.main(PipSQueak.java:556) > Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM > ERROR: NullPointerException > Fragment 1:1 > [Error Id: ba4661b4-2ff6-4eff-889c-fc4604a57418 on ucs-node8.perf.lab:31010] > (java.lang.NullPointerException) null > org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe():607 > org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe():560 > > org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputRecordKeysPrev():525 > > org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputToBatchPrev():322 > > org.apache.drill.exec.test.generated.StreamingAggregatorGen250.doWork():182 > > org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():170 > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > org.apache.drill.exec.physical.impl.BaseRootExec.next():105 > > org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():144 > org.apache.drill.exec.physical.impl.BaseRootExec.next():95 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227 > java.security.AccessController.doPrivileged():-2 > javax.security.auth.Subject.doAs():415 > org.apache.hadoop.security.UserGroupInformation.doAs():1595 > org.apache.drill.exec.work.fragment.FragmentExecutor.run():227 > org.apache.drill.common.SelfCleaningRunnable.run():3
[jira] [Commented] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163937#comment-16163937 ] ASF GitHub Bot commented on DRILL-5752: --- Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138499857 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java --- @@ -54,11 +56,13 @@ import org.junit.Ignore; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.TemporaryFolder; import org.junit.runner.RunWith; import org.junit.runners.Parameterized; @RunWith(Parameterized.class) +@Category({SecondaryTest.class, ParquetTest.class}) --- End diff -- To me the main distinction between smoke and secondary tests should be runtime. It's hard in my mind to distinguish based on importance, because the relevance of a test depends on the change being done. And you would end up running all the tests before finishing your change no matter what. The workflow I had in mind was the following: 1. Make a change 1. Run Fast tests for my category of interest (e.g. **OperatorTest**). Oops there's a failure 1. Fix the failure 1. Run Fast tests for category **OperatorTest** again. Oops another failure 1. Fix the failure 1. Run Fast tests for category **OperatorTest** again. Yay they pass 1. Run all the tests 1. Done Let me know what you think. > Speed Up Unit Tests > --- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5704) Improve error message on client side when queries fail with "Failed to create schema tree." when Impersonation is enabled and logins are anonymous
[ https://issues.apache.org/jira/browse/DRILL-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163938#comment-16163938 ] ASF GitHub Bot commented on DRILL-5704: --- Github user sohami closed the pull request at: https://github.com/apache/drill/pull/895 > Improve error message on client side when queries fail with "Failed to create > schema tree." when Impersonation is enabled and logins are anonymous > -- > > Key: DRILL-5704 > URL: https://issues.apache.org/jira/browse/DRILL-5704 > Project: Apache Drill > Issue Type: Improvement >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia > Labels: ready-to-commit > Fix For: 1.12.0 > > > Reported by [~agirish] > When username is not specified then Drill set's the session user as anonymous > if impersonation is enabled. During query execution Drill tries to build > schema tree and as part of that it validates if the user has access to the > workspace or not by using FileClient Api liststatus which verifies the user > from the OS user. Since impersonation is only enabled here without > authentication and we don't specify any user in connection string, Drill will > use default user which is "anonymous" and pass that to check workspace > permission which will fail as node doesn't have any valid user with that name. > {code:java} > Caused by: java.io.IOException: Error getting user info for current user, > anonymous >.. >.. > at > org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus(DrillFileSystem.java:523) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible(WorkspaceSchemaFactory.java:157) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.(FileSystemSchemaFactory.java:78) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas(FileSystemSchemaFactory.java:65) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas(FileSystemPlugin.java:150) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > ... 10 common frames omitted > {code} > # $DRILL_HOME/bin/sqlline -u "jdbc:drill:zk=localhost:5181" > sqlline> select * from sys.drillbits; > User Error Occurred > org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Failed to > create schema tree. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5704) Improve error message on client side when queries fail with "Failed to create schema tree." when Impersonation is enabled and logins are anonymous
[ https://issues.apache.org/jira/browse/DRILL-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163941#comment-16163941 ] ASF GitHub Bot commented on DRILL-5704: --- Github user sohami commented on the issue: https://github.com/apache/drill/pull/895 This PR was already merged by @arina-ielchiieva on 08/24 > Improve error message on client side when queries fail with "Failed to create > schema tree." when Impersonation is enabled and logins are anonymous > -- > > Key: DRILL-5704 > URL: https://issues.apache.org/jira/browse/DRILL-5704 > Project: Apache Drill > Issue Type: Improvement >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia > Labels: ready-to-commit > Fix For: 1.12.0 > > > Reported by [~agirish] > When username is not specified then Drill set's the session user as anonymous > if impersonation is enabled. During query execution Drill tries to build > schema tree and as part of that it validates if the user has access to the > workspace or not by using FileClient Api liststatus which verifies the user > from the OS user. Since impersonation is only enabled here without > authentication and we don't specify any user in connection string, Drill will > use default user which is "anonymous" and pass that to check workspace > permission which will fail as node doesn't have any valid user with that name. > {code:java} > Caused by: java.io.IOException: Error getting user info for current user, > anonymous >.. >.. > at > org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus(DrillFileSystem.java:523) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible(WorkspaceSchemaFactory.java:157) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.(FileSystemSchemaFactory.java:78) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas(FileSystemSchemaFactory.java:65) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas(FileSystemPlugin.java:150) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365) > ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > ... 10 common frames omitted > {code} > # $DRILL_HOME/bin/sqlline -u "jdbc:drill:zk=localhost:5181" > sqlline> select * from sys.drillbits; > User Error Occurred > org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Failed to > create schema tree. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5772) Add unit tests to indicate how utf-8 support can be enabled / disabled in Drill
[ https://issues.apache.org/jira/browse/DRILL-5772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163949#comment-16163949 ] ASF GitHub Bot commented on DRILL-5772: --- Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/936 @arina-ielchiieva, thanks for the explanation. Drill's runtime framework assumes that data is either: 1. ASCII (or, at least, single-byte character set based on ASCII), or 2. UTF-8 (data is converted to/from UTF-8 when converting VarChar to a Java String.) Since Drill's code seems to assume ASCII (when it cares about character format), then one could claim that Drill does not have an encoding: it just treats characters as bytes. But, things such as determining string length, doing pattern matching, and so on must be aware of the character set -- if only to know which bytes are continuations of a multi-byte character. (That is, a three-byte sequence in UTF-8 might be one, two or three characters, depending.) Now, if the planner assumes ISO-8859-1, but the Drill execution engine assumes UTF-8, then string constants passed from one to the other can become corrupted in the case where a particular byte sequence in ISO-8859-1 represents a different character than that same byte sequence in UTF-8. Where would this occur? Look at the [ISO-8859-1](https://en.wikipedia.org/wiki/ISO/IEC_8859-1) definition. ISO-8859 is a single-byte character set with meanings associated to the bytes in the range 0x40 to 0x7f. But, in UTF-8, the high bit indicates a prefix character. So, 0xF7 is a valid single-byte character in ISO-8859, but is a lead-in character in UTF-8. The point here is that setting the character set would seem to be a global setting. If the Saffron setting is purely for the parser (how to interpret incoming text), and the parser always produces Java strings in UTF-16 (which are then encoded into UTF-8 for execution), then we're fine. But, if the parser encoding is written as bytes sent to the execution engine, we're in trouble. Further, Drill has a web UI. The typical web character set is UTF-8, so queries coming from the web UI are encoded in UTF-8. All this suggests two things: 1. Drill should either always accept UTF-8 (the Saffron property should always be set.) or 2. The property is specified by the client and used to decode the bytes within a Protobuf message to produce a character stream given to the parser. It appears that UTF-8 is the default Protobuf String type encoding; sender and receiver would have to agree on another format. Does Drill have such an RPC property? I've not seen it, but I'm not an expert. In short, if this change ensures that the parser *always* uses UTF-8, then this is good. If the character encoding is an option, then we have to consider all the above issues to have a fully working, end-to-end solution. > Add unit tests to indicate how utf-8 support can be enabled / disabled in > Drill > --- > > Key: DRILL-5772 > URL: https://issues.apache.org/jira/browse/DRILL-5772 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.11.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Labels: doc-impacting > Fix For: 1.12.0 > > > Add unit test to indicated how utf-8 support can be enabled in Drill. > To select utf-8 data user needs to update system property > {{saffron.default.charset}} to {{UTF-16LE}} before starting the drillbit. > Calcite uses this property to get default charset, if it is not set then > {{ISO-8859-1}} is used by default. Drill gets default charset from Calcite. > This information should be also documented, probably in > https://drill.apache.org/docs/data-type-conversion/. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5784) SYSTEM ERROR: IndexOutOfBoundsException: index: 512, length: 4 (expected: range(0, 512))
[ https://issues.apache.org/jira/browse/DRILL-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163964#comment-16163964 ] Vlad Rozov commented on DRILL-5784: --- Stack trace: {code} Caused by: java.lang.IndexOutOfBoundsException: index: 512, length: 4 (expected: range(0, 512)) at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:122) at io.netty.buffer.DrillBuf.chk(DrillBuf.java:146) at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:523) at org.apache.drill.exec.vector.IntVector$Accessor.get(IntVector.java:405) at org.apache.drill.exec.test.generated.NestedLoopJoinGen2.doEval(NestedLoopJoinGen2.java:102) at org.apache.drill.exec.physical.impl.join.NestedLoopJoinTemplate.populateOutgoingBatch(NestedLoopJoinTemplate.java:122) at org.apache.drill.exec.physical.impl.join.NestedLoopJoinTemplate.outputRecords(NestedLoopJoinTemplate.java:86) at org.apache.drill.exec.physical.impl.join.NestedLoopJoinBatch.innerNext(NestedLoopJoinBatch.java:181) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:141) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) at org.apache.drill.exec.test.generated.HashAggregatorGen4.doWork(HashAggTemplate.java:581) at org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:168) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:141) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) at org.apache.drill.exec.physical.impl.join.NestedLoopJoinBatch.buildSchema(NestedLoopJoinBatch.java:380) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:144) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:141) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:141) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105) at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:227) ... 4 common frames omitted {code} > SYSTEM ERROR: IndexOutOfBoundsException: index: 512, length: 4 (expected: > range(0, 512)) > > >
[jira] [Assigned] (DRILL-5775) Select * query on a maprdb binary table fails
[ https://issues.apache.org/jira/browse/DRILL-5775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinfeng Ni reassigned DRILL-5775: - Assignee: Jinfeng Ni > Select * query on a maprdb binary table fails > - > > Key: DRILL-5775 > URL: https://issues.apache.org/jira/browse/DRILL-5775 > Project: Apache Drill > Issue Type: Bug > Components: Storage - MapRDB >Affects Versions: 1.11.0 >Reporter: Prasad Nagaraj Subramanya >Assignee: Jinfeng Ni > > Select * query on a maprdb binary table fails with the below exception > Failed with exception > java.sql.SQLException: SYSTEM ERROR: IllegalArgumentException: > HBaseRecordReader does not allow column *. Column * should have been > converted to list of family_n > Commit ID - fde0a1df1734e0742b49aabdd28b02202ee2b044 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5783) Make code generation in the TopN operator more modular and test it
[ https://issues.apache.org/jira/browse/DRILL-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163975#comment-16163975 ] Timothy Farkas commented on DRILL-5783: --- Thanks [~Paul.Rogers]. I'll look through the code and follow the same patterns. > Make code generation in the TopN operator more modular and test it > -- > > Key: DRILL-5783 > URL: https://issues.apache.org/jira/browse/DRILL-5783 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5785) Query Error on a large PCAP file
Takeo Ogawara created DRILL-5785: Summary: Query Error on a large PCAP file Key: DRILL-5785 URL: https://issues.apache.org/jira/browse/DRILL-5785 Project: Apache Drill Issue Type: Bug Components: Storage - Other Affects Versions: 1.11.0 Reporter: Takeo Ogawara Priority: Minor Query on a very large PCAP file (larger than 100GB) failed with following error message. > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a > > Fragment 1:169 > > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] > (state=,code=0) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5785) Query Error on a large PCAP file
[ https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163984#comment-16163984 ] Takeo Ogawara commented on DRILL-5785: -- Here are the messages in Drill log files. drillbit.log --- 2017-09-11 15:06:52,390 [BitServer-2] WARN o.a.d.exec.rpc.control.WorkEventBus - A fragment message arrived but there was no registered listener for that message: profile { state: FAILED error { error_id: "bbf284b6-9da4-4869-ac20-fa100eed11b9" endpoint { address: "node22" user_port: 31010 control_port: 31011 data_port: 31012 version: "1.11.0" } error_type: SYSTEM message: "SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a\n\nFragment 1:200\n\n[Error Id: bbf284b6-9da4-4869-ac20-fa100eed11b9 on node22:31010]" exception { exception_class: "java.lang.IllegalStateException" message: "Bad magic number = 0a0d0d0a" stack_trace { class_name: "com.google.common.base.Preconditions" file_name: "Preconditions.java" line_number: 173 method_name: "checkState" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.store.pcap.decoder.PacketDecoder" file_name: "PacketDecoder.java" line_number: 84 method_name: "" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.store.pcap.PcapRecordReader" file_name: "PcapRecordReader.java" line_number: 104 method_name: "setup" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.ScanBatch" file_name: "ScanBatch.java" line_number: 104 method_name: "" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin" file_name: "EasyFormatPlugin.java" line_number: 166 method_name: "getReaderBatch" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator" file_name: "EasyReaderBatchCreator.java" line_number: 35 method_name: "getBatch" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator" file_name: "EasyReaderBatchCreator.java" line_number: 28 method_name: "getBatch" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.ImplCreator" file_name: "ImplCreator.java" line_number: 156 method_name: "getRecordBatch" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.ImplCreator" file_name: "ImplCreator.java" line_number: 179 method_name: "getChildren" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.ImplCreator" file_name: "ImplCreator.java" line_number: 136 method_name: "getRecordBatch" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.ImplCreator" file_name: "ImplCreator.java" line_number: 179 method_name: "getChildren" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.ImplCreator" file_name: "ImplCreator.java" line_number: 136 method_name: "getRecordBatch" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.ImplCreator" file_name: "ImplCreator.java" line_number: 179 method_name: "getChildren" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.ImplCreator" file_name: "ImplCreator.java" line_number: 109 method_name: "getRootExec" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.physical.impl.ImplCreator" file_name: "ImplCreator.java" line_number: 87 method_name: "getExec" is_native_method: false } stack_trace { class_name: "org.apache.drill.exec.work.fragment.FragmentExecutor" file_name: "FragmentExecutor.java" line_number: 207 method_name: "run" is_native_method: false } stack_trace { class_name: "org.apache.drill.common.SelfCleaningRunnable" file_name: "SelfCleaningRunnable.java" line_number: 38 method_name: "run" is_native_method: false } stack_trace { class_name: "..." line_number: 0
[jira] [Commented] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163990#comment-16163990 ] ASF GitHub Bot commented on DRILL-5752: --- Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138505403 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/xsort/TestSimpleExternalSort.java --- @@ -148,10 +149,10 @@ public void sortOneKeyDescendingExternalSortLegacy() throws Throwable { private void sortOneKeyDescendingExternalSort(boolean testLegacy) throws Throwable { FixtureBuilder builder = ClusterFixture.builder() -.configProperty(ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 4) -.configProperty(ExecConstants.EXTERNAL_SORT_SPILL_GROUP_SIZE, 4) -.configProperty(ExecConstants.EXTERNAL_SORT_BATCH_LIMIT, 4) -.configProperty(ExecConstants.EXTERNAL_SORT_DISABLE_MANAGED, false) +.configProperty(OptionValidator.OPTION_DEFAULTS_ROOT + ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 4) --- End diff -- Done, thanks for catching. > Speed Up Unit Tests > --- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164011#comment-16164011 ] ASF GitHub Bot commented on DRILL-5752: --- Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138506701 --- Diff: common/src/test/java/org/apache/drill/categories/package-info.java --- @@ -0,0 +1,23 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * This package stores the JUnit test categories. + */ --- End diff -- Terminology update. I'm renaming **SecondaryTest** to **SlowTest** and instead of **BugFixTest** I will create a category called **UnlikelyTest**. **UnlikelyTest** will be a category for tests that excercise specific bug fixes, or are for tests that you're unlikely to break. > Speed Up Unit Tests > --- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5752) Speed Up Unit Tests
[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164012#comment-16164012 ] ASF GitHub Bot commented on DRILL-5752: --- Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138506769 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java --- @@ -54,11 +56,13 @@ import org.junit.Ignore; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.TemporaryFolder; import org.junit.runner.RunWith; import org.junit.runners.Parameterized; @RunWith(Parameterized.class) +@Category({SecondaryTest.class, ParquetTest.class}) --- End diff -- Please see Terminology update in comment above. > Speed Up Unit Tests > --- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5785) Query Error on a large PCAP file
[ https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeo Ogawara updated DRILL-5785: - Attachment: Apache Drill_files.zip Apache Drill.html > Query Error on a large PCAP file > > > Key: DRILL-5785 > URL: https://issues.apache.org/jira/browse/DRILL-5785 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Other >Affects Versions: 1.11.0 >Reporter: Takeo Ogawara >Priority: Minor > Attachments: Apache Drill_files.zip, Apache Drill.html > > > Query on a very large PCAP file (larger than 100GB) failed with following > error message. > > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a > > > > Fragment 1:169 > > > > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] > > (state=,code=0) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5785) Query Error on a large PCAP file
[ https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeo Ogawara updated DRILL-5785: - Attachment: (was: Apache Drill_files.zip) > Query Error on a large PCAP file > > > Key: DRILL-5785 > URL: https://issues.apache.org/jira/browse/DRILL-5785 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Other >Affects Versions: 1.11.0 >Reporter: Takeo Ogawara >Priority: Minor > Attachments: Apache Drill_files.zip, Apache Drill.html > > > Query on a very large PCAP file (larger than 100GB) failed with following > error message. > > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a > > > > Fragment 1:169 > > > > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] > > (state=,code=0) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5785) Query Error on a large PCAP file
[ https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeo Ogawara updated DRILL-5785: - Attachment: (was: Apache Drill.html) > Query Error on a large PCAP file > > > Key: DRILL-5785 > URL: https://issues.apache.org/jira/browse/DRILL-5785 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Other >Affects Versions: 1.11.0 >Reporter: Takeo Ogawara >Priority: Minor > Attachments: Apache Drill_files.zip, Apache Drill.html > > > Query on a very large PCAP file (larger than 100GB) failed with following > error message. > > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a > > > > Fragment 1:169 > > > > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] > > (state=,code=0) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5785) Query Error on a large PCAP file
[ https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeo Ogawara updated DRILL-5785: - Attachment: Apache Drill_files.zip Apache Drill.html Query profiles. > Query Error on a large PCAP file > > > Key: DRILL-5785 > URL: https://issues.apache.org/jira/browse/DRILL-5785 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Other >Affects Versions: 1.11.0 >Reporter: Takeo Ogawara >Priority: Minor > Attachments: Apache Drill_files.zip, Apache Drill.html > > > Query on a very large PCAP file (larger than 100GB) failed with following > error message. > > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a > > > > Fragment 1:169 > > > > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] > > (state=,code=0) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5785) Query Error on a large PCAP file
[ https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164029#comment-16164029 ] Robert Hou commented on DRILL-5785: --- Can you attach a sample of your PCAP file? Can you attach the query profile? Go to the URL http://:8047. Click on Profiles in the upper left hand corner. You will see a list of queries that have run on your system. Click on the link for the query. You will see something like this in the URL: https://:8047/profiles/264e5071-f248-1001-d72a-5a4e850d6ea6 The long string is your queryID. There should be a file 264e5071-f248-1001-d72a-5a4e850d6ea6.sys.drill on your system in $DRILL_HOME/logs. This is your query file stored on disk. It would be helpful if you can attach it to this Jira. > Query Error on a large PCAP file > > > Key: DRILL-5785 > URL: https://issues.apache.org/jira/browse/DRILL-5785 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Other >Affects Versions: 1.11.0 >Reporter: Takeo Ogawara >Priority: Minor > Attachments: Apache Drill_files.zip, Apache Drill.html > > > Query on a very large PCAP file (larger than 100GB) failed with following > error message. > > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a > > > > Fragment 1:169 > > > > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] > > (state=,code=0) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5785) Query Error on a large PCAP file
[ https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeo Ogawara updated DRILL-5785: - Attachment: 2647724c-d367-c8cc-0b6d-579228ffa31e.sys.drill sample2.pcap Sorry, I attached html file. Profile file is here. You can create dummy large PCAP file with mergecap command using sample pcap file. > Query Error on a large PCAP file > > > Key: DRILL-5785 > URL: https://issues.apache.org/jira/browse/DRILL-5785 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Other >Affects Versions: 1.11.0 >Reporter: Takeo Ogawara >Priority: Minor > Attachments: 2647724c-d367-c8cc-0b6d-579228ffa31e.sys.drill, Apache > Drill_files.zip, Apache Drill.html, sample2.pcap > > > Query on a very large PCAP file (larger than 100GB) failed with following > error message. > > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a > > > > Fragment 1:169 > > > > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] > > (state=,code=0) -- This message was sent by Atlassian JIRA (v6.4.14#64029)