[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[ https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979633#comment-15979633 ] ASF GitHub Bot commented on DRILL-5270: --- Github user kkhatua commented on the issue: https://github.com/apache/drill/pull/755 @sudheeshkatkam Can you please review the PR? > Improve loading of profiles listing in the WebUI > > > Key: DRILL-5270 > URL: https://issues.apache.org/jira/browse/DRILL-5270 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.9.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua > Fix For: 1.11.0 > > > Currently, as the number of profiles increase, we reload the same list of > profiles from the FS. > An ideal improvement would be to detect if there are any new profiles and > only reload from the disk then. Otherwise, a cached list is sufficient. > For a directory of 280K profiles, the load time is close to 6 seconds on a 32 > core server. With the caching, we can get it down to as much as a few > milliseconds. > To render the cache as invalid, we inspect the last modified time of the > directory to confirm whether a reload is needed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (DRILL-5322) Provide an OperatorFixture for sub-operator unit testing setup
[
https://issues.apache.org/jira/browse/DRILL-5322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudheesh Katkam resolved DRILL-5322.
Resolution: Fixed
Fixed as part of DRILL-5318.
> Provide an OperatorFixture for sub-operator unit testing setup
> --
>
> Key: DRILL-5322
> URL: https://issues.apache.org/jira/browse/DRILL-5322
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> We recently created various "fixture" classes to assist with system-level
> testing: {{LogFixture}}, {{ClusterFixture}} and {{ClientFixture}}. Each
> handles the tedious work of setting up the conditions to run certain kinds of
> tests.
> In the same way, we need an {{OperatorFixture}} to set up the low-level bits
> and pieces needed for operator-level, and sub-operator-level unit testing.
> The {{DrillConfig}} is used by both the system-level and operator-level
> fixtures. So, pull the config-setup tasks our of (cluster) {{FixtureBuilder}}
> (should rename) and into a new {{ConfigBuilder}}. Leave the existing methods
> in {{FixtureBuilder}}, but modify them to be wrappers around the new config
> builder.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (DRILL-5320) Refactor OptionManager for unit testing
[
https://issues.apache.org/jira/browse/DRILL-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudheesh Katkam resolved DRILL-5320.
Resolution: Fixed
See DRILL-5319.
> Refactor OptionManager for unit testing
> ---
>
> Key: DRILL-5320
> URL: https://issues.apache.org/jira/browse/DRILL-5320
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> The {{OptionManager}} interface serves two purposes:
> * Create and modify options
> * Access option values
> The implementations of this class are integrated with the rest of Drill,
> making it difficult to use the classes in isolation in unit testing. Further,
> since operators are given the full interface, the operator has the ability to
> modify options, and so each unit test should either verify that no
> modification is, in fact, done, or must track down modifications and test
> them.
> For operator and sub-operator unit tests we need a simpler interface. As it
> turns out, most low-level uses of {{OptionManager}} are all read-only. This
> allows a simple refactoring to enhance unit testability: create a new
> super-interface {{OptionSet}}, which provides only the read-only methods.
> Then, refactor low-level classes (code generation, compilers, and so on) to
> use the restricted {{OptionSet}} interface.
> Finally, for unit tests, create a trivial, map-based implementation that can
> be populated as needed for each specific test.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (DRILL-5321) Refactor FragmentContext for unit testing
[
https://issues.apache.org/jira/browse/DRILL-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudheesh Katkam resolved DRILL-5321.
Resolution: Fixed
See DRILL-5319.
> Refactor FragmentContext for unit testing
> -
>
> Key: DRILL-5321
> URL: https://issues.apache.org/jira/browse/DRILL-5321
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> Each operator has visibility to the {{FragmentContext}} class.
> {{FragmentContext}} provides access to all of Drill internals: the Drillbit
> context, the network interfaces, RPC messages and so on.
> Further, all the code generation mechanisms require a {{FragmentContext}}
> object.
> This structure creates a large barrier to unit testing. To test, say, a
> particular bit of generated code, we must have the entire Drillbit running so
> we can obtain a {{FragmentContext}}. Clearly, this is less than ideal.
> Upon inspection, it turns out that the {{FragmentContext}} is mostly needed,
> by many operators, to generate code. Of the many methods in
> {{FragmentContext}}, code generation uses only six.
> The solution is to create a new super-interface, {{CodeGenContext}}, which
> holds those six methods. The {{CodeGenContext}} can be easily re-implemented
> for unit testing.
> Then, modify all the code-generation classes that currently take
> {{FragmentContext}} to take {{CodeGenContext}} instead.
> Since {{FragmentContext}} derives from {{CodeGenContext}}, existing operator
> code "just works."
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (DRILL-5318) Create a sub-operator test framework
[
https://issues.apache.org/jira/browse/DRILL-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979579#comment-15979579
]
ASF GitHub Bot commented on DRILL-5318:
---
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/788
> Create a sub-operator test framework
>
>
> Key: DRILL-5318
> URL: https://issues.apache.org/jira/browse/DRILL-5318
> Project: Apache Drill
> Issue Type: Improvement
> Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Labels: ready-to-commit
> Fix For: 1.11.0
>
> Attachments: Sub-OperatorTestFramework.pdf
>
>
> Drill provides two unit test frameworks for whole-server, SQL-based testing:
> the original {{BaseTestQuery}} and the newer {{ClusterFixture}}. Both use the
> {{TestBuilder}} mechanism to build system-level functional tests that run
> queries and check results.
> Jason provided an operator-level test framework based, in part on mocks:
> As Drill operators become more complex, we have a crying need for true
> unit-level tests at a level below the whole system and below operators. That
> is, we need to test the individual pieces that, together, form the operator.
> This umbrella ticket includes a number of tasks needed to create the
> sub-operator framework. Our intention is that, over time, as we find the need
> to revisit existing operators, or create new ones, we can employ the
> sub-operator test framework to exercise code at a finer granularity than is
> possible prior to this framework.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (DRILL-5323) Provide test tools to create, populate and compare row sets
[
https://issues.apache.org/jira/browse/DRILL-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979578#comment-15979578
]
ASF GitHub Bot commented on DRILL-5323:
---
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/785
> Provide test tools to create, populate and compare row sets
> ---
>
> Key: DRILL-5323
> URL: https://issues.apache.org/jira/browse/DRILL-5323
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Labels: ready-to-commit
> Fix For: 1.11.0
>
>
> Operators work with individual row sets. A row set is a collection of records
> stored as column vectors. (Drill uses various terms for this concept. A
> record batch is a row set with an operator implementation wrapped around it.
> A vector container is a row set, but with much functionality left as an
> exercise for the developer. And so on.)
> To simplify tests, we need a {{TestRowSet}} concept that wraps a
> {{VectorContainer}} and provides easy ways to:
> * Define a schema for the row set.
> * Create a set of vectors that implement the schema.
> * Populate the row set with test data via code.
> * Add an SV2 to the row set.
> * Pass the row set to operator components (such as generated code blocks.)
> * Compare the results of the operation with an expected result set.
> * Dispose of the underling direct memory when work is done.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (DRILL-5433) Authentication failed: Server requires authentication using [kerberos, plain]
[
https://issues.apache.org/jira/browse/DRILL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chun Chang closed DRILL-5433.
-
> Authentication failed: Server requires authentication using [kerberos, plain]
> -
>
> Key: DRILL-5433
> URL: https://issues.apache.org/jira/browse/DRILL-5433
> Project: Apache Drill
> Issue Type: Task
> Components: Functions - Drill
>Affects Versions: 1.10.0
> Environment: OS: Redhat Linux 6.7, HDP 2.5.3, Kerberos enabled,
> Hardware: VmWare
>Reporter: Parag Darji
>Priority: Minor
> Labels: newbie, security
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I've setup Apace drill 1.10.0 on RHEL 6.7, HDP 2.5.3, kerberos enabled
> I'm getting below error while running "drill-conf" or sqlline as user "drill"
> which is configured in the "drill-override.conf" file.
> {code}
> drill@host:/opt/drill/bin> drill-conf
> Error: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?] (state=,code=0)
> java.sql.SQLException: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?]
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:166)
> at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
> at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
> at
> sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
> at sqlline.Commands.connect(Commands.java:1083)
> at sqlline.Commands.connect(Commands.java:1015)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:742)
> at sqlline.SqlLine.initArgs(SqlLine.java:528)
> at sqlline.SqlLine.begin(SqlLine.java:596)
> at sqlline.SqlLine.start(SqlLine.java:375)
> at sqlline.SqlLine.main(SqlLine.java:268)
> Caused by: org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?]
> at
> org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:157)
> at
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
> at
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:157)
> ... 18 more
> Caused by: javax.security.sasl.SaslException: Authentication failed: Server
> requires authentication using [kerberos, plain]. Insufficient credentials?
> [Caused by javax.security.sasl.SaslException: Server requires authentication
> using [kerberos, plain]. Insufficient credentials?]
> at
> org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:204)
> at
> org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:197)
> at
> com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
> at
> org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:155)
> ... 21 more
> Caused by: javax.security.sasl.SaslException: Server requires authentication
> using [kerberos, plain]. Insufficient credentials?
> at
> org.apache.drill.exec.rpc.user.UserClient.getAuthenticatorFactory(UserClient.java:285)
>
[jira] [Commented] (DRILL-5323) Provide test tools to create, populate and compare row sets
[
https://issues.apache.org/jira/browse/DRILL-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979441#comment-15979441
]
ASF GitHub Bot commented on DRILL-5323:
---
Github user sudheeshkatkam commented on the issue:
https://github.com/apache/drill/pull/785
+1
> Provide test tools to create, populate and compare row sets
> ---
>
> Key: DRILL-5323
> URL: https://issues.apache.org/jira/browse/DRILL-5323
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Labels: ready-to-commit
> Fix For: 1.11.0
>
>
> Operators work with individual row sets. A row set is a collection of records
> stored as column vectors. (Drill uses various terms for this concept. A
> record batch is a row set with an operator implementation wrapped around it.
> A vector container is a row set, but with much functionality left as an
> exercise for the developer. And so on.)
> To simplify tests, we need a {{TestRowSet}} concept that wraps a
> {{VectorContainer}} and provides easy ways to:
> * Define a schema for the row set.
> * Create a set of vectors that implement the schema.
> * Populate the row set with test data via code.
> * Add an SV2 to the row set.
> * Pass the row set to operator components (such as generated code blocks.)
> * Compare the results of the operation with an expected result set.
> * Dispose of the underling direct memory when work is done.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (DRILL-5323) Provide test tools to create, populate and compare row sets
[
https://issues.apache.org/jira/browse/DRILL-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudheesh Katkam updated DRILL-5323:
---
Labels: ready-to-commit (was: )
> Provide test tools to create, populate and compare row sets
> ---
>
> Key: DRILL-5323
> URL: https://issues.apache.org/jira/browse/DRILL-5323
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Labels: ready-to-commit
> Fix For: 1.11.0
>
>
> Operators work with individual row sets. A row set is a collection of records
> stored as column vectors. (Drill uses various terms for this concept. A
> record batch is a row set with an operator implementation wrapped around it.
> A vector container is a row set, but with much functionality left as an
> exercise for the developer. And so on.)
> To simplify tests, we need a {{TestRowSet}} concept that wraps a
> {{VectorContainer}} and provides easy ways to:
> * Define a schema for the row set.
> * Create a set of vectors that implement the schema.
> * Populate the row set with test data via code.
> * Add an SV2 to the row set.
> * Pass the row set to operator components (such as generated code blocks.)
> * Compare the results of the operation with an expected result set.
> * Dispose of the underling direct memory when work is done.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (DRILL-5405) Add missing operator types
[ https://issues.apache.org/jira/browse/DRILL-5405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam updated DRILL-5405: --- Labels: (was: ready-to-commit) > Add missing operator types > -- > > Key: DRILL-5405 > URL: https://issues.apache.org/jira/browse/DRILL-5405 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Fix For: 1.11.0 > > Attachments: maprdb_sub_scan.JPG, unknown_operator.JPG > > > Add missing operator types: FLATTEN, MONGO_SUB_SCAN, MAPRDB_SUB_SCAN so they > won't be displayed on Web UI as UNKNOWN_OPERATOR. > Example: > before the fix -> unknown_operator.JPG > after the fix -> maprdb_sub_scan.JPG -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5405) Add missing operator types
[ https://issues.apache.org/jira/browse/DRILL-5405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979436#comment-15979436 ] ASF GitHub Bot commented on DRILL-5405: --- Github user sudheeshkatkam commented on the issue: https://github.com/apache/drill/pull/804 From the [comments on 214](https://github.com/apache/drill/pull/214#discussion_r42687575), looks like this change in not required. I could not find the resolution/ discussion on the dev list, as mentioned in the comments. But since a new operator type was not added in the PR, this change is not necessary. -0 for this change. > Add missing operator types > -- > > Key: DRILL-5405 > URL: https://issues.apache.org/jira/browse/DRILL-5405 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Fix For: 1.11.0 > > Attachments: maprdb_sub_scan.JPG, unknown_operator.JPG > > > Add missing operator types: FLATTEN, MONGO_SUB_SCAN, MAPRDB_SUB_SCAN so they > won't be displayed on Web UI as UNKNOWN_OPERATOR. > Example: > before the fix -> unknown_operator.JPG > after the fix -> maprdb_sub_scan.JPG -- This message was sent by Atlassian JIRA (v6.3.15#6346)
