[jira] [Commented] (DRILL-5421) Allow Complex type as @Workspace variables for aggregate functions
[
https://issues.apache.org/jira/browse/DRILL-5421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966075#comment-15966075
]
Manuel Aristaran commented on DRILL-5421:
-
Hi [~cgivre],
I posted about this on the users mailing list. Someone pointed out that an
{{ObjectHolder}} can be used as a {{@Workspace}} variable. That should help you
implementing statistical functions, as they have a numeric return type.
Link:
http://mail-archives.apache.org/mod_mbox/drill-user/201704.mbox/%3CCAJrw0OSO1p8k%2Bkescd2ismjVngBidhgf8dv2KMm8Wopfk5Bw4A%40mail.gmail.com%3E
> Allow Complex type as @Workspace variables for aggregate functions
> --
>
> Key: DRILL-5421
> URL: https://issues.apache.org/jira/browse/DRILL-5421
> Project: Apache Drill
> Issue Type: Wish
> Components: Functions - Drill
>Affects Versions: 1.10.0
>Reporter: Manuel Aristaran
>
> I'm trying to implement the equivalent of Postgres' {{array_agg()}} as a UDF
> in Drill. Unless I'm missing something, this doesn't seem to be possible. As
> per [the
> documentation|https://drill.apache.org/docs/developing-an-aggregate-function/]:
> {quote}
> Do not use complex @Workspace variables.
> You cannot allocate a Repeated* value or have a ComplexWriter in the
> @Workspace.
> {quote}
> With that restriction, it's not possible to keep the intermediate value of
> the resulting array during computation of the aggregation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (DRILL-5424) Fix IOBE for reverse function
[
https://issues.apache.org/jira/browse/DRILL-5424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966127#comment-15966127
]
ASF GitHub Bot commented on DRILL-5424:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/815#discussion_r92811
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/expr/fn/impl/TestStringFunctions.java
---
@@ -273,4 +273,13 @@ public void testSplit() throws Exception {
.run();
}
+ @Test
+ public void testReverse() throws Exception {
+testBuilder()
+ .sqlQuery("select reverse(reverse(n_comment)) words from
cp.`tpch/nation.parquet`")
+ .unOrdered()
+ .sqlBaselineQuery("select n_comment words from
cp.`tpch/nation.parquet`")
+ .build()
+ .run();
+ }
--- End diff --
Unfortunately this unit test does not ensure that reverse function works
correctly.
Let's replace this unit test with two unit tests:
1. test reverse result for one row like in `testSplit()` test.
2. test that reverse doesn't fail for table with several long varchars.
> Fix IOBE for reverse function
> -
>
> Key: DRILL-5424
> URL: https://issues.apache.org/jira/browse/DRILL-5424
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Minor
>
> Query with reverse function fails:
> {code:sql}
> 0: jdbc:drill:zk=local> select reverse(a) from dfs.`/tmp/test.json`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: index: 259, length: 1
> (expected: range(0, 256))
> {code}
> for table with several long varchars.
> {noformat}
> cat /tmp/test.json
> {"a": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz"}
> {"a": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz"}
> {"a": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz"}
> {"a": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz"}
> {"a": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz"}
> {noformat}
> The same query works for the table with less row number:
> {code:sql}
> 0: jdbc:drill:zk=local> select reverse(a) from dfs.`/tmp/test2.json`;
> +---+
> |EXPR$0 |
> +---+
> | zyxwvutsrqponmlkjihgfedcbazyxwvutsrqponmlkjihgfedcba |
> | zyxwvutsrqponmlkjihgfedcbazyxwvutsrqponmlkjihgfedcba |
> | zyxwvutsrqponmlkjihgfedcbazyxwvutsrqponmlkjihgfedcba |
> | zyxwvutsrqponmlkjihgfedcbazyxwvutsrqponmlkjihgfedcba |
> +---+
> {code}
> {noformat}
> cat /tmp/test2.json
> {"a": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz"}
> {"a": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz"}
> {"a": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz"}
> {"a": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz"}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (DRILL-5431) Support SSL
Sudheesh Katkam created DRILL-5431: -- Summary: Support SSL Key: DRILL-5431 URL: https://issues.apache.org/jira/browse/DRILL-5431 Project: Apache Drill Issue Type: New Feature Components: Client - Java, Client - ODBC Reporter: Sudheesh Katkam Support SSL between Drillbit and JDBC/ODBC drivers. Drill already supports HTTPS for web traffic. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (DRILL-5104) Foreman sets external sort memory allocation even for a physical plan
[
https://issues.apache.org/jira/browse/DRILL-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rahul Challapalli updated DRILL-5104:
-
Attachment: physical.json
2711af55-e1f7-bf4e-3dcf-b9f66f69fc33.sys.drill
Looks like I am still hitting this problem. Below is the query I used to
generate the attached physical plan (physical.json)
{code}
ALTER SESSION SET `exec.sort.disable_managed` = false;
alter session set `planner.width.max_per_node` = 1;
alter session set `planner.width.max_per_node` = 1;
alter session set `planner.disable_exchanges` = true;
alter session set `planner.memory.max_query_memory_per_node` = 62914560;
select count(*) from (select * from
dfs.`/drill/testdata/resource-manager/250wide-small.tbl` order by columns[0])d
where d.columns[0] = 'ljdfhwuehnoiueyf';
{code}
Now I submitted the physical plan using the below command
{code}
/opt/drill/bin/submit_plan -f /home/framework/physical.json -t physical -z
10.10.100.183:5181
{code}
>From the attached query profile, external sort consumed 104 MB before
>spilling. Can you take a look [~Paul.Rogers] ?
> Foreman sets external sort memory allocation even for a physical plan
> -
>
> Key: DRILL-5104
> URL: https://issues.apache.org/jira/browse/DRILL-5104
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Labels: ready-to-commit
> Fix For: 1.10.0
>
> Attachments: 2711af55-e1f7-bf4e-3dcf-b9f66f69fc33.sys.drill,
> physical.json
>
>
> Consider the (disabled) unit test
> {{TestSimpleExternalSort.outOfMemoryExternalSort}} which uses the physical
> plan {{xsort/oom_sort_test.json}} that contains a setting for the amount of
> memory to allocate:
> {code}
>{
> ...
> pop:"external-sort",
> ...
> initialAllocation: 100,
> maxAllocation: 3000
> },
> {code}
> When run, the amount of memory is set to 715827882. The reason is that code
> was added to {{Foreman}} to compute the memory to allocate to the external
> sort:
> {code}
> private void runPhysicalPlan(final PhysicalPlan plan) throws
> ExecutionSetupException {
> validatePlan(plan);
> MemoryAllocationUtilities.setupSortMemoryAllocations(plan, queryContext);
> {code}
> The problem is that a physical plan should execute as provided to enable
> detailed testing.
> To solve this problem, move the sort memory setup to the path taken by SQL
> queries, but not via physical plans.
> This change is necessary to re-enable the previously-disabled external sort
> tests.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (DRILL-4273) If query is cancelled during external sort memory is leaked, merge join fragment is running forever on a forman node
[
https://issues.apache.org/jira/browse/DRILL-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966237#comment-15966237
]
Rahul Challapalli commented on DRILL-4273:
--
Tested with the new managed external sort (git.commit.id.abbrev=9df3403) and I
cannot reproduce the memory leak or the dangling fragments issue. Can't rule it
out still as it could be related to timing. Will try a few more times
> If query is cancelled during external sort memory is leaked, merge join
> fragment is running forever on a forman node
>
>
> Key: DRILL-4273
> URL: https://issues.apache.org/jira/browse/DRILL-4273
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.5.0
>Reporter: Victoria Markman
>Priority: Critical
> Attachments: 2967b65e-ea42-a736-9f5d-3513914ada88.sys.drill,
> after_shutdown.log, drillbit.log.node_133, drillbit.log.node_134,
> drillbit.log.node_135, drillbit.log.node_136
>
>
> Query was cancelled during external sort.
> Here is what happened:
> 1. Query got stuck in CANCELLATION_REQUESTED mode: ( queryid =
> 2967b65e-ea42-a736-9f5d-3513914ada88 )
> 2. On a forman node 6:0 (executing merge join) fragment kept running forever
> (see stack below)
> {code}
> "2967b65e-ea42-a736-9f5d-3513914ada88:frag:6:0" daemon prio=10
> tid=0x01af5800 nid=0x769b runnable [0x7fa82fc7c000]
>java.lang.Thread.State: RUNNABLE
> at java.lang.Throwable.getStackTraceElement(Native Method)
> at java.lang.Throwable.getOurStackTrace(Throwable.java:827)
> - locked <0x0006eef95f80> (a java.lang.Exception)
> at java.lang.Throwable.getStackTrace(Throwable.java:816)
> at java.lang.Thread.getStackTrace(Thread.java:1589)
> at org.apache.drill.common.StackTrace.(StackTrace.java:33)
> at
> org.apache.drill.common.HistoricalLog$Event.(HistoricalLog.java:39)
> at
> org.apache.drill.common.HistoricalLog.recordEvent(HistoricalLog.java:95)
> - locked <0x0006eef95870> (a
> org.apache.drill.common.HistoricalLog)
> at io.netty.buffer.DrillBuf.(DrillBuf.java:84)
> at
> org.apache.drill.exec.memory.AllocatorManager$BufferLedger.newDrillBuf(AllocatorManager.java:285)
> at
> org.apache.drill.exec.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:222)
> at
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:204)
> at
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:177)
> at
> org.apache.drill.exec.vector.IntVector.allocateBytes(IntVector.java:201)
> at
> org.apache.drill.exec.vector.IntVector.allocateNew(IntVector.java:183)
> at
> org.apache.drill.exec.vector.NullableIntVector.allocateNew(NullableIntVector.java:216)
> at
> org.apache.drill.exec.vector.AllocationHelper.allocateNew(AllocationHelper.java:56)
> at
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.allocateBatch(MergeJoinBatch.java:429)
> at
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext(MergeJoinBatch.java:172)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:295)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> at
> org.apache.d
[jira] [Updated] (DRILL-5430) IOB Exception : Select on fastparquet created parquet file
[
https://issues.apache.org/jira/browse/DRILL-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-5430:
--
Summary: IOB Exception : Select on fastparquet created parquet file (was:
select on fastparquet created parquet file results in IOB Exception)
> IOB Exception : Select on fastparquet created parquet file
> --
>
> Key: DRILL-5430
> URL: https://issues.apache.org/jira/browse/DRILL-5430
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.11.0
>Reporter: Khurram Faraaz
> Attachments: frmPandas_9.parquet
>
>
> Select on a parquet file created using python's fastparquet results in IOB
> Exception.
> Parquet file was created using fastparquet (python implementation of parquet
> format)
> Parquet schema information for parquet file used in test.
> {noformat}
> [root@centos-12q parquet-tools]# ./parquet-schema frmPandas_9.parquet
> message schema {
> required int32 c_int8_signed (INT_8);
> required int32 c_uint8 (UINT_8);
> }
> {noformat}
> Apache Drill 1.11.0
> git.commit.id.abbrev=06e1522
> Select on the parquet results in DATA_READ error, stack trace shows an
> IndexOutOfBoundsException
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select * from `frmPandas_9.parquet`;
> Error: DATA_READ ERROR: Error reading from Parquet file
> File: /tmp/frmPandas_9.parquet
> Column: c_int8_signed
> Row Group Start: 4
> Fragment 0:0
> [Error Id: 32b4d0ec-4198-4b38-bdd7-a27c881072e6 on centos-01.qa.lab:31010]
> (state=,code=0)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2017-04-12 04:23:55,214 [27125424-657c-7f8f-c27b-4d1516e4bb97:frag:0:0] INFO
> o.a.d.e.s.p.c.ColumnReader - User Error Occurred: Error reading from Parquet
> file (srcIndex: 0)
> org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: Error
> reading from Parquet file
> File: /tmp/frmPandas_9.parquet
> Column: c_int8_signed
> Row Group Start: 4
> [Error Id: 32b4d0ec-4198-4b38-bdd7-a27c881072e6 ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
> ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readValues(ColumnReader.java:151)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPageData(ColumnReader.java:199)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:179)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPages(ColumnReader.java:129)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFieldsSerial(ParquetRecordReader.java:512)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFields(ParquetRecordReader.java:505)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:590)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:179)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
> [drill
[jira] [Created] (DRILL-5432) Want a memory format for PCAP files
Ted Dunning created DRILL-5432: -- Summary: Want a memory format for PCAP files Key: DRILL-5432 URL: https://issues.apache.org/jira/browse/DRILL-5432 Project: Apache Drill Issue Type: New Feature Reporter: Ted Dunning PCAP files [1] are the de facto standard for storing network capture data. In security and protocol applications, it is very common to want to extract particular packets from a capture for further analysis. At a first level, it is desirable to query and filter by source and destination IP and port or by protocol. Beyond that, however, it would be very useful to be able to group packets by TCP session and eventually to look at packet contents. For now, however, the most critical requirement is that we should be able to scan captures at very high speed. I previously wrote a (kind of working) proof of concept for a PCAP decoder that did lazy deserialization and could traverse hundreds of MB of PCAP data per second per core. This compares to roughly 2-3 MB/s for widely available Apache-compatible open source PCAP decoders. This JIRA covers the integration and extension of that proof of concept as a Drill file format. Initial work is available at https://github.com/mapr-demos/pcap-query [1] https://en.wikipedia.org/wiki/Pcap -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5433) Authentication failed: Server requires authentication using [kerberos, plain]
Parag Darji created DRILL-5433: -- Summary: Authentication failed: Server requires authentication using [kerberos, plain] Key: DRILL-5433 URL: https://issues.apache.org/jira/browse/DRILL-5433 Project: Apache Drill Issue Type: Task Components: Functions - Drill Affects Versions: 1.10.0 Environment: OS: Redhat Linux 6.7, HDP 2.5.3, Kerberos enabled, Hardware: VmWare Reporter: Parag Darji Priority: Minor Fix For: 1.10.0 I've setup Apace drill 1.10.0 on RHEL 6.7, HDP 2.5.3, kerberos enabled I'm getting below error while running "drill-conf" or sqlline as user "drill" which is configured in the "drill-override.conf" file. drill@host:/opt/drill/bin> drill-conf Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials? [Caused by javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?] (state=,code=0) java.sql.SQLException: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials? [Caused by javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?] at org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:166) at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72) at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69) at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143) at org.apache.drill.jdbc.Driver.connect(Driver.java:72) at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167) at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213) at sqlline.Commands.connect(Commands.java:1083) at sqlline.Commands.connect(Commands.java:1015) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36) at sqlline.SqlLine.dispatch(SqlLine.java:742) at sqlline.SqlLine.initArgs(SqlLine.java:528) at sqlline.SqlLine.begin(SqlLine.java:596) at sqlline.SqlLine.start(SqlLine.java:375) at sqlline.SqlLine.main(SqlLine.java:268) Caused by: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials? [Caused by javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?] at org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:157) at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432) at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379) at org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:157) ... 18 more Caused by: javax.security.sasl.SaslException: Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials? [Caused by javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?] at org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:204) at org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:197) at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85) at org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:155) ... 21 more Caused by: javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials? at org.apache.drill.exec.rpc.user.UserClient.getAuthenticatorFactory(UserClient.java:285) at org.apache.drill.exec.rpc.user.UserClient.authenticate(UserClient.java:216) ... 22 more apache drill 1.10.0 "this isn't your grandfather's sql" Same error when running below command: sqlline --maxWidth=1 -u "jdbc:drill:drillbit=host1.fqdn;auth=kerberos;principal=drill/lad...@lab.com" "Drill" user has has valid keytab/ticket. The Drill UI is working fine with local authentication. drill-override.c
[jira] [Commented] (DRILL-5433) Authentication failed: Server requires authentication using [kerberos, plain]
[ https://issues.apache.org/jira/browse/DRILL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966568#comment-15966568 ] Sudheesh Katkam commented on DRILL-5433: Can you check the error in drillbit.log file (by default in $DRILL_HOME/log directory)? > Authentication failed: Server requires authentication using [kerberos, plain] > - > > Key: DRILL-5433 > URL: https://issues.apache.org/jira/browse/DRILL-5433 > Project: Apache Drill > Issue Type: Task > Components: Functions - Drill >Affects Versions: 1.10.0 > Environment: OS: Redhat Linux 6.7, HDP 2.5.3, Kerberos enabled, > Hardware: VmWare >Reporter: Parag Darji >Priority: Minor > Labels: newbie, security > Fix For: 1.10.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > I've setup Apace drill 1.10.0 on RHEL 6.7, HDP 2.5.3, kerberos enabled > I'm getting below error while running "drill-conf" or sqlline as user "drill" > which is configured in the "drill-override.conf" file. > drill@host:/opt/drill/bin> drill-conf > Error: Failure in connecting to Drill: > org.apache.drill.exec.rpc.NonTransientRpcException: > javax.security.sasl.SaslException: Authentication failed: Server requires > authentication using [kerberos, plain]. Insufficient credentials? [Caused by > javax.security.sasl.SaslException: Server requires authentication using > [kerberos, plain]. Insufficient credentials?] (state=,code=0) > java.sql.SQLException: Failure in connecting to Drill: > org.apache.drill.exec.rpc.NonTransientRpcException: > javax.security.sasl.SaslException: Authentication failed: Server requires > authentication using [kerberos, plain]. Insufficient credentials? [Caused by > javax.security.sasl.SaslException: Server requires authentication using > [kerberos, plain]. Insufficient credentials?] > at > org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:166) > at > org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72) > at > org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69) > at > org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143) > at org.apache.drill.jdbc.Driver.connect(Driver.java:72) > at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167) > at > sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213) > at sqlline.Commands.connect(Commands.java:1083) > at sqlline.Commands.connect(Commands.java:1015) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36) > at sqlline.SqlLine.dispatch(SqlLine.java:742) > at sqlline.SqlLine.initArgs(SqlLine.java:528) > at sqlline.SqlLine.begin(SqlLine.java:596) > at sqlline.SqlLine.start(SqlLine.java:375) > at sqlline.SqlLine.main(SqlLine.java:268) > Caused by: org.apache.drill.exec.rpc.NonTransientRpcException: > javax.security.sasl.SaslException: Authentication failed: Server requires > authentication using [kerberos, plain]. Insufficient credentials? [Caused by > javax.security.sasl.SaslException: Server requires authentication using > [kerberos, plain]. Insufficient credentials?] > at > org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:157) > at > org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432) > at > org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379) > at > org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:157) > ... 18 more > Caused by: javax.security.sasl.SaslException: Authentication failed: Server > requires authentication using [kerberos, plain]. Insufficient credentials? > [Caused by javax.security.sasl.SaslException: Server requires authentication > using [kerberos, plain]. Insufficient credentials?] > at > org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:204) > at > org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:197) > at > com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85) > at > org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:155) > ... 21 more > Caused by: javax.security.sasl.SaslException: Server requires a
[jira] [Commented] (DRILL-5433) Authentication failed: Server requires authentication using [kerberos, plain]
[ https://issues.apache.org/jira/browse/DRILL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966582#comment-15966582 ] Parag Darji commented on DRILL-5433: I do not see any error in drillbit.log: 2017-04-12 16:42:49,549 [main] INFO o.a.drill.exec.server.rest.WebServer - Setting up HTTP connector for web server 2017-04-12 16:42:50,936 [main] INFO o.apache.drill.exec.server.Drillbit - Startup completed (5728 ms). However I do see this error in sqlline.log: Override File: file:/opt/drill/conf/drill-override.conf 2017-04-12 16:43:15,002 [main] WARN o.a.drill.exec.metrics.DrillMetrics - Removing old metric since name matched newly registered metric. Metric name: drill.allocator.root.used 2017-04-12 16:43:15,002 [main] WARN o.a.drill.exec.metrics.DrillMetrics - Removing old metric since name matched newly registered metric. Metric name: drill.allocator.root.peak 2017-04-12 16:43:15,053 [main] INFO o.a.drill.common.config.DrillConfig - Configuration and plugin file(s) identified in 50ms. Base Configuration: - jar:file:/opt/drill/jars/drill-common-1.10.0.jar!/drill-default.conf Intermediate Configuration and Plugin files, in order of precedence: - jar:file:/opt/drill/jars/drill-java-exec-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-common-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-storage-hive-core-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-memory-base-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-mongo-storage-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-jdbc-storage-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-logical-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-storage-hbase-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-gis-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-kudu-storage-1.10.0.jar!/drill-module.conf - jar:file:/opt/drill/jars/drill-hive-exec-shaded-1.10.0.jar!/drill-module.conf Override File: file:/opt/drill/conf/drill-override.conf 2017-04-12 16:43:15,063 [main] WARN o.a.drill.exec.metrics.DrillMetrics - Removing old metric since name matched newly registered metric. Metric name: drill.allocator.root.used 2017-04-12 16:43:15,064 [main] WARN o.a.drill.exec.metrics.DrillMetrics - Removing old metric since name matched newly registered metric. Metric name: drill.allocator.root.peak 2017-04-12 16:43:15,299 [main] ERROR o.a.drill.exec.client.DrillClient - Connection to altbthdlhdpsb01.cscdev.com:31010 failed with error javax.security.sasl.SaslException: Authentication failed: Server requires authentication using [kerberos, plain]. Insufficient credentials? [Caused by javax.security.sasl.SaslException: Server requires authentication using [kerberos, plain]. Insufficient credentials?]. Not retrying anymore > Authentication failed: Server requires authentication using [kerberos, plain] > - > > Key: DRILL-5433 > URL: https://issues.apache.org/jira/browse/DRILL-5433 > Project: Apache Drill > Issue Type: Task > Components: Functions - Drill >Affects Versions: 1.10.0 > Environment: OS: Redhat Linux 6.7, HDP 2.5.3, Kerberos enabled, > Hardware: VmWare >Reporter: Parag Darji >Priority: Minor > Labels: newbie, security > Fix For: 1.10.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > I've setup Apace drill 1.10.0 on RHEL 6.7, HDP 2.5.3, kerberos enabled > I'm getting below error while running "drill-conf" or sqlline as user "drill" > which is configured in the "drill-override.conf" file. > drill@host:/opt/drill/bin> drill-conf > Error: Failure in connecting to Drill: > org.apache.drill.exec.rpc.NonTransientRpcException: > javax.security.sasl.SaslException: Authentication failed: Server requires > authentication using [kerberos, plain]. Insufficient credentials? [Caused by > javax.security.sasl.SaslException: Server requires authentication using > [kerberos, plain]. Insufficient credentials?] (state=,code=0) > java.sql.SQLException: Failure in connecting to Drill: > org.apache.drill.exec.rpc.NonTransientRpcException: > javax.security.sasl.SaslException: Authentication failed: Server requires > authentication using [kerberos, plain]. Insufficient credentials? [Caused by > javax.security.sasl.SaslException: Server requires authentication using > [kerberos, plain]. Insufficient credentials?] > at > org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:166) > at > org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72) > at > or
[jira] [Updated] (DRILL-5433) Authentication failed: Server requires authentication using [kerberos, plain]
[
https://issues.apache.org/jira/browse/DRILL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudheesh Katkam updated DRILL-5433:
---
Description:
I've setup Apace drill 1.10.0 on RHEL 6.7, HDP 2.5.3, kerberos enabled
I'm getting below error while running "drill-conf" or sqlline as user "drill"
which is configured in the "drill-override.conf" file.
{code}
drill@host:/opt/drill/bin> drill-conf
Error: Failure in connecting to Drill:
org.apache.drill.exec.rpc.NonTransientRpcException:
javax.security.sasl.SaslException: Authentication failed: Server requires
authentication using [kerberos, plain]. Insufficient credentials? [Caused by
javax.security.sasl.SaslException: Server requires authentication using
[kerberos, plain]. Insufficient credentials?] (state=,code=0)
java.sql.SQLException: Failure in connecting to Drill:
org.apache.drill.exec.rpc.NonTransientRpcException:
javax.security.sasl.SaslException: Authentication failed: Server requires
authentication using [kerberos, plain]. Insufficient credentials? [Caused by
javax.security.sasl.SaslException: Server requires authentication using
[kerberos, plain]. Insufficient credentials?]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:166)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.NonTransientRpcException:
javax.security.sasl.SaslException: Authentication failed: Server requires
authentication using [kerberos, plain]. Insufficient credentials? [Caused by
javax.security.sasl.SaslException: Server requires authentication using
[kerberos, plain]. Insufficient credentials?]
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:157)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:157)
... 18 more
Caused by: javax.security.sasl.SaslException: Authentication failed: Server
requires authentication using [kerberos, plain]. Insufficient credentials?
[Caused by javax.security.sasl.SaslException: Server requires authentication
using [kerberos, plain]. Insufficient credentials?]
at
org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:204)
at
org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:197)
at
com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:155)
... 21 more
Caused by: javax.security.sasl.SaslException: Server requires authentication
using [kerberos, plain]. Insufficient credentials?
at
org.apache.drill.exec.rpc.user.UserClient.getAuthenticatorFactory(UserClient.java:285)
at
org.apache.drill.exec.rpc.user.UserClient.authenticate(UserClient.java:216)
... 22 more
apache drill 1.10.0
"this isn't your grandfather's sql"
{code}
Same error when running below command:
{code}
sqlline --maxWidth=1 -u
"jdbc:drill:drillbit=host1.fqdn;auth=kerberos;principal=drill/lad...@lab.com"
{code}
"Drill" user has has valid keytab/ticket.
The Drill UI is working fine with local authentication.
drill-override.conf file:
{code}
drill.exec: {
cluster-id: "drillbits1",
zk.connect: "host1.fqdn:2181,host2.fqdn:2181,host3.fqdn:2181",
security: {
user.auth.enabled: true,
user.auth.impl: "pam",
user.auth.pam_profiles: [ "sudo", "login" ],
packages += "org.apache.drill.exec.rpc.user.security",
auth.me
[jira] [Commented] (DRILL-5433) Authentication failed: Server requires authentication using [kerberos, plain]
[
https://issues.apache.org/jira/browse/DRILL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966604#comment-15966604
]
Sudheesh Katkam commented on DRILL-5433:
What's the error why you try this?
{code}
sqlline --maxWidth=1 -u
"jdbc:drill:drillbit=host1.fqdn;auth=kerberos;principal=drill/lad...@lab.com"
{code}
I would guess the error is a slight different message.
> Authentication failed: Server requires authentication using [kerberos, plain]
> -
>
> Key: DRILL-5433
> URL: https://issues.apache.org/jira/browse/DRILL-5433
> Project: Apache Drill
> Issue Type: Task
> Components: Functions - Drill
>Affects Versions: 1.10.0
> Environment: OS: Redhat Linux 6.7, HDP 2.5.3, Kerberos enabled,
> Hardware: VmWare
>Reporter: Parag Darji
>Priority: Minor
> Labels: newbie, security
> Fix For: 1.10.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I've setup Apace drill 1.10.0 on RHEL 6.7, HDP 2.5.3, kerberos enabled
> I'm getting below error while running "drill-conf" or sqlline as user "drill"
> which is configured in the "drill-override.conf" file.
> {code}
> drill@host:/opt/drill/bin> drill-conf
> Error: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?] (state=,code=0)
> java.sql.SQLException: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?]
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:166)
> at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
> at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
> at
> sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
> at sqlline.Commands.connect(Commands.java:1083)
> at sqlline.Commands.connect(Commands.java:1015)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:742)
> at sqlline.SqlLine.initArgs(SqlLine.java:528)
> at sqlline.SqlLine.begin(SqlLine.java:596)
> at sqlline.SqlLine.start(SqlLine.java:375)
> at sqlline.SqlLine.main(SqlLine.java:268)
> Caused by: org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?]
> at
> org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:157)
> at
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
> at
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:157)
> ... 18 more
> Caused by: javax.security.sasl.SaslException: Authentication failed: Server
> requires authentication using [kerberos, plain]. Insufficient credentials?
> [Caused by javax.security.sasl.SaslException: Server requires authentication
> using [kerberos, plain]. Insufficient credentials?]
> at
> org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:204)
> at
> org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:197)
> at
> com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
> at
> org.apache.drill.exec.rpc.
[jira] [Commented] (DRILL-5433) Authentication failed: Server requires authentication using [kerberos, plain]
[ https://issues.apache.org/jira/browse/DRILL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966612#comment-15966612 ] Parag Darji commented on DRILL-5433: Here is error: drill@:/home/drill> sqlline --maxWidth=1 -u "jdbc:drill:drillbit=host1.fqdn;auth=kerberos;principal=drill/lad...@lab.com" Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed unexpectedly. [Caused by java.util.concurrent.ExecutionException: org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]] (state=,code=0) java.sql.SQLException: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed unexpectedly. [Caused by java.util.concurrent.ExecutionException: org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]] at org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:166) at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72) at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69) at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143) at org.apache.drill.jdbc.Driver.connect(Driver.java:72) at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167) at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213) at sqlline.Commands.connect(Commands.java:1083) at sqlline.Commands.connect(Commands.java:1015) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36) at sqlline.SqlLine.dispatch(SqlLine.java:742) at sqlline.SqlLine.initArgs(SqlLine.java:528) at sqlline.SqlLine.begin(SqlLine.java:596) at sqlline.SqlLine.start(SqlLine.java:375) at sqlline.SqlLine.main(SqlLine.java:268) Caused by: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed unexpectedly. [Caused by java.util.concurrent.ExecutionException: org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]] at org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:157) at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432) at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379) at org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:157) ... 18 more Caused by: javax.security.sasl.SaslException: Authentication failed unexpectedly. [Caused by java.util.concurrent.ExecutionException: org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]] at org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:207) at org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:197) at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85) at org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:155) ... 21 more Caused by: java.util.concurrent.ExecutionException: org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)] at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63) at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractChecked
[jira] [Comment Edited] (DRILL-5433) Authentication failed: Server requires authentication using [kerberos, plain]
[
https://issues.apache.org/jira/browse/DRILL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966612#comment-15966612
]
Parag Darji edited comment on DRILL-5433 at 4/12/17 9:02 PM:
-
Here is error:
{code}
drill@:/home/drill> sqlline --maxWidth=1 -u
"jdbc:drill:drillbit=host1.fqdn;auth=kerberos;principal=drill/lad...@lab.com"
Error: Failure in connecting to Drill:
org.apache.drill.exec.rpc.NonTransientRpcException:
javax.security.sasl.SaslException: Authentication failed unexpectedly. [Caused
by java.util.concurrent.ExecutionException:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]]
(state=,code=0)
java.sql.SQLException: Failure in connecting to Drill:
org.apache.drill.exec.rpc.NonTransientRpcException:
javax.security.sasl.SaslException: Authentication failed unexpectedly. [Caused
by java.util.concurrent.ExecutionException:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:166)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.NonTransientRpcException:
javax.security.sasl.SaslException: Authentication failed unexpectedly. [Caused
by java.util.concurrent.ExecutionException:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]]
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:157)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:157)
... 18 more
Caused by: javax.security.sasl.SaslException: Authentication failed
unexpectedly. [Caused by java.util.concurrent.ExecutionException:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]]
at
org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:207)
at
org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:197)
at
com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:155)
... 21 more
Caused by: java.util.concurrent.ExecutionException:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
at
com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
at
com.google.common.util.conc
[jira] [Comment Edited] (DRILL-5433) Authentication failed: Server requires authentication using [kerberos, plain]
[
https://issues.apache.org/jira/browse/DRILL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966582#comment-15966582
]
Parag Darji edited comment on DRILL-5433 at 4/12/17 9:02 PM:
-
I do not see any error in drillbit.log:
{code}
2017-04-12 16:42:49,549 [main] INFO o.a.drill.exec.server.rest.WebServer -
Setting up HTTP connector for web server
2017-04-12 16:42:50,936 [main] INFO o.apache.drill.exec.server.Drillbit -
Startup completed (5728 ms).
{code}
However I do see this error in sqlline.log:
{code}
Override File: file:/opt/drill/conf/drill-override.conf
2017-04-12 16:43:15,002 [main] WARN o.a.drill.exec.metrics.DrillMetrics -
Removing old metric since name matched newly registered metric. Metric name:
drill.allocator.root.used
2017-04-12 16:43:15,002 [main] WARN o.a.drill.exec.metrics.DrillMetrics -
Removing old metric since name matched newly registered metric. Metric name:
drill.allocator.root.peak
2017-04-12 16:43:15,053 [main] INFO o.a.drill.common.config.DrillConfig -
Configuration and plugin file(s) identified in 50ms.
Base Configuration:
- jar:file:/opt/drill/jars/drill-common-1.10.0.jar!/drill-default.conf
Intermediate Configuration and Plugin files, in order of precedence:
- jar:file:/opt/drill/jars/drill-java-exec-1.10.0.jar!/drill-module.conf
- jar:file:/opt/drill/jars/drill-common-1.10.0.jar!/drill-module.conf
-
jar:file:/opt/drill/jars/drill-storage-hive-core-1.10.0.jar!/drill-module.conf
-
jar:file:/opt/drill/jars/drill-memory-base-1.10.0.jar!/drill-module.conf
-
jar:file:/opt/drill/jars/drill-mongo-storage-1.10.0.jar!/drill-module.conf
-
jar:file:/opt/drill/jars/drill-jdbc-storage-1.10.0.jar!/drill-module.conf
- jar:file:/opt/drill/jars/drill-logical-1.10.0.jar!/drill-module.conf
-
jar:file:/opt/drill/jars/drill-storage-hbase-1.10.0.jar!/drill-module.conf
- jar:file:/opt/drill/jars/drill-gis-1.10.0.jar!/drill-module.conf
-
jar:file:/opt/drill/jars/drill-kudu-storage-1.10.0.jar!/drill-module.conf
-
jar:file:/opt/drill/jars/drill-hive-exec-shaded-1.10.0.jar!/drill-module.conf
Override File: file:/opt/drill/conf/drill-override.conf
2017-04-12 16:43:15,063 [main] WARN o.a.drill.exec.metrics.DrillMetrics -
Removing old metric since name matched newly registered metric. Metric name:
drill.allocator.root.used
2017-04-12 16:43:15,064 [main] WARN o.a.drill.exec.metrics.DrillMetrics -
Removing old metric since name matched newly registered metric. Metric name:
drill.allocator.root.peak
2017-04-12 16:43:15,299 [main] ERROR o.a.drill.exec.client.DrillClient -
Connection to altbthdlhdpsb01.cscdev.com:31010 failed with error
javax.security.sasl.SaslException: Authentication failed: Server requires
authentication using [kerberos, plain]. Insufficient credentials? [Caused by
javax.security.sasl.SaslException: Server requires authentication using
[kerberos, plain]. Insufficient credentials?]. Not retrying anymore
{code}
was (Author: pd47):
I do not see any error in drillbit.log:
2017-04-12 16:42:49,549 [main] INFO o.a.drill.exec.server.rest.WebServer -
Setting up HTTP connector for web server
2017-04-12 16:42:50,936 [main] INFO o.apache.drill.exec.server.Drillbit -
Startup completed (5728 ms).
However I do see this error in sqlline.log:
Override File: file:/opt/drill/conf/drill-override.conf
2017-04-12 16:43:15,002 [main] WARN o.a.drill.exec.metrics.DrillMetrics -
Removing old metric since name matched newly registered metric. Metric name:
drill.allocator.root.used
2017-04-12 16:43:15,002 [main] WARN o.a.drill.exec.metrics.DrillMetrics -
Removing old metric since name matched newly registered metric. Metric name:
drill.allocator.root.peak
2017-04-12 16:43:15,053 [main] INFO o.a.drill.common.config.DrillConfig -
Configuration and plugin file(s) identified in 50ms.
Base Configuration:
- jar:file:/opt/drill/jars/drill-common-1.10.0.jar!/drill-default.conf
Intermediate Configuration and Plugin files, in order of precedence:
- jar:file:/opt/drill/jars/drill-java-exec-1.10.0.jar!/drill-module.conf
- jar:file:/opt/drill/jars/drill-common-1.10.0.jar!/drill-module.conf
-
jar:file:/opt/drill/jars/drill-storage-hive-core-1.10.0.jar!/drill-module.conf
-
jar:file:/opt/drill/jars/drill-memory-base-1.10.0.jar!/drill-module.conf
-
jar:file:/opt/drill/jars/drill-mongo-storage-1.10.0.jar!/drill-module.conf
-
jar:file:/opt/drill/jars/drill-jdbc-storage-1.10.0.jar!/drill-module.conf
- jar:file:/opt/drill/jars/drill-logical-1.10.0.jar!/drill-module.conf
-
jar:file:/opt/drill/jars/drill-storage-hbase-1.10.0.jar!/drill-module.conf
- jar:file:/opt/drill/jars/drill-gis-1.10.0.jar!/drill-module.conf
-
jar:file:/opt/drill/jars/drill-kudu-storage-1.10.0.jar!/drill-module.conf
[jira] [Commented] (DRILL-5433) Authentication failed: Server requires authentication using [kerberos, plain]
[
https://issues.apache.org/jira/browse/DRILL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966640#comment-15966640
]
Sudheesh Katkam commented on DRILL-5433:
The error says: *Server not found in Kerberos database (7) - UNKNOWN_SERVER*
This needs to be investigated.
Some pointers for more logging:
+ Create a file named */tmp/java.logging.txt* with contents:
{code}
javax.security.sasl.level=FINEST
handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=FINEST
{code}
+ Edit $DRILL_HOME/conf/drill-env.sh:
{code}
export SQLLINE_JAVA_OPTS = "$SQLLINE_JAVA_OPTS -Dsun.security.jgss.debug=true
-Dsun.security.krb5.debug=true
-Djava.util.logging.config.file=/tmp/java.logging.txt"
{code}
Maybe the */tmp/java.logging.txt* will have information. You could also look
into Kerberos logs; logging configuration is in */etc/krb5.conf* file.
> Authentication failed: Server requires authentication using [kerberos, plain]
> -
>
> Key: DRILL-5433
> URL: https://issues.apache.org/jira/browse/DRILL-5433
> Project: Apache Drill
> Issue Type: Task
> Components: Functions - Drill
>Affects Versions: 1.10.0
> Environment: OS: Redhat Linux 6.7, HDP 2.5.3, Kerberos enabled,
> Hardware: VmWare
>Reporter: Parag Darji
>Priority: Minor
> Labels: newbie, security
> Fix For: 1.10.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I've setup Apace drill 1.10.0 on RHEL 6.7, HDP 2.5.3, kerberos enabled
> I'm getting below error while running "drill-conf" or sqlline as user "drill"
> which is configured in the "drill-override.conf" file.
> {code}
> drill@host:/opt/drill/bin> drill-conf
> Error: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?] (state=,code=0)
> java.sql.SQLException: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?]
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:166)
> at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
> at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
> at
> sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
> at sqlline.Commands.connect(Commands.java:1083)
> at sqlline.Commands.connect(Commands.java:1015)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:742)
> at sqlline.SqlLine.initArgs(SqlLine.java:528)
> at sqlline.SqlLine.begin(SqlLine.java:596)
> at sqlline.SqlLine.start(SqlLine.java:375)
> at sqlline.SqlLine.main(SqlLine.java:268)
> Caused by: org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?]
> at
> org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:157)
> at
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
> at
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:157)
> ... 18 more
> Caused by: javax.security.sasl.SaslException: Authentication failed: Server
> requires authentication using [kerberos, plain]. Insuffici
[jira] [Comment Edited] (DRILL-5433) Authentication failed: Server requires authentication using [kerberos, plain]
[
https://issues.apache.org/jira/browse/DRILL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966640#comment-15966640
]
Sudheesh Katkam edited comment on DRILL-5433 at 4/12/17 9:18 PM:
-
The error says: *Server not found in Kerberos database (7) - UNKNOWN_SERVER*
This needs to be investigated.
Some pointers for more logging:
+ Create a file named */tmp/java.logging.txt* with contents:
{code}
javax.security.sasl.level=FINEST
handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=FINEST
{code}
+ Edit $DRILL_HOME/conf/drill-env.sh:
{code}
export SQLLINE_JAVA_OPTS = "$SQLLINE_JAVA_OPTS -Dsun.security.jgss.debug=true
-Dsun.security.krb5.debug=true
-Djava.util.logging.config.file=/tmp/java.logging.txt"
{code}
Maybe the */tmp/java.logging.txt* will have information. And there is more info
to the console was well. You could also look into Kerberos logs; logging
configuration is in */etc/krb5.conf* file.
(Also make sure you don't commit with any sensitive info here.)
was (Author: sudheeshkatkam):
The error says: *Server not found in Kerberos database (7) - UNKNOWN_SERVER*
This needs to be investigated.
Some pointers for more logging:
+ Create a file named */tmp/java.logging.txt* with contents:
{code}
javax.security.sasl.level=FINEST
handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=FINEST
{code}
+ Edit $DRILL_HOME/conf/drill-env.sh:
{code}
export SQLLINE_JAVA_OPTS = "$SQLLINE_JAVA_OPTS -Dsun.security.jgss.debug=true
-Dsun.security.krb5.debug=true
-Djava.util.logging.config.file=/tmp/java.logging.txt"
{code}
Maybe the */tmp/java.logging.txt* will have information. You could also look
into Kerberos logs; logging configuration is in */etc/krb5.conf* file.
> Authentication failed: Server requires authentication using [kerberos, plain]
> -
>
> Key: DRILL-5433
> URL: https://issues.apache.org/jira/browse/DRILL-5433
> Project: Apache Drill
> Issue Type: Task
> Components: Functions - Drill
>Affects Versions: 1.10.0
> Environment: OS: Redhat Linux 6.7, HDP 2.5.3, Kerberos enabled,
> Hardware: VmWare
>Reporter: Parag Darji
>Priority: Minor
> Labels: newbie, security
> Fix For: 1.10.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I've setup Apace drill 1.10.0 on RHEL 6.7, HDP 2.5.3, kerberos enabled
> I'm getting below error while running "drill-conf" or sqlline as user "drill"
> which is configured in the "drill-override.conf" file.
> {code}
> drill@host:/opt/drill/bin> drill-conf
> Error: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?] (state=,code=0)
> java.sql.SQLException: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?]
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:166)
> at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
> at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
> at
> sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
> at sqlline.Commands.connect(Commands.java:1083)
> at sqlline.Commands.connect(Commands.java:1015)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:742)
> at sqlline.SqlLine.initArgs(SqlLine.java:528)
> at sqlline.SqlLine.begin(SqlLine.java:596)
> at sqlline.SqlLine.start(SqlLine.java:375)
> a
[jira] [Comment Edited] (DRILL-5433) Authentication failed: Server requires authentication using [kerberos, plain]
[
https://issues.apache.org/jira/browse/DRILL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966640#comment-15966640
]
Sudheesh Katkam edited comment on DRILL-5433 at 4/12/17 9:18 PM:
-
The error says: *Server not found in Kerberos database (7) - UNKNOWN_SERVER*
This needs to be investigated.
Some pointers for more logging:
+ Create a file named */tmp/java.logging.txt* with contents:
{code}
javax.security.sasl.level=FINEST
handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=FINEST
{code}
+ Edit $DRILL_HOME/conf/drill-env.sh:
{code}
export SQLLINE_JAVA_OPTS = "$SQLLINE_JAVA_OPTS -Dsun.security.jgss.debug=true
-Dsun.security.krb5.debug=true
-Djava.util.logging.config.file=/tmp/java.logging.txt"
{code}
Maybe the */tmp/java.logging.txt* will have information. And there is more info
to the console was well. You could also look into Kerberos logs; logging
configuration is in */etc/krb5.conf* file.
(Also make sure you don't comment any sensitive info here.)
was (Author: sudheeshkatkam):
The error says: *Server not found in Kerberos database (7) - UNKNOWN_SERVER*
This needs to be investigated.
Some pointers for more logging:
+ Create a file named */tmp/java.logging.txt* with contents:
{code}
javax.security.sasl.level=FINEST
handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=FINEST
{code}
+ Edit $DRILL_HOME/conf/drill-env.sh:
{code}
export SQLLINE_JAVA_OPTS = "$SQLLINE_JAVA_OPTS -Dsun.security.jgss.debug=true
-Dsun.security.krb5.debug=true
-Djava.util.logging.config.file=/tmp/java.logging.txt"
{code}
Maybe the */tmp/java.logging.txt* will have information. And there is more info
to the console was well. You could also look into Kerberos logs; logging
configuration is in */etc/krb5.conf* file.
(Also make sure you don't commit with any sensitive info here.)
> Authentication failed: Server requires authentication using [kerberos, plain]
> -
>
> Key: DRILL-5433
> URL: https://issues.apache.org/jira/browse/DRILL-5433
> Project: Apache Drill
> Issue Type: Task
> Components: Functions - Drill
>Affects Versions: 1.10.0
> Environment: OS: Redhat Linux 6.7, HDP 2.5.3, Kerberos enabled,
> Hardware: VmWare
>Reporter: Parag Darji
>Priority: Minor
> Labels: newbie, security
> Fix For: 1.10.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I've setup Apace drill 1.10.0 on RHEL 6.7, HDP 2.5.3, kerberos enabled
> I'm getting below error while running "drill-conf" or sqlline as user "drill"
> which is configured in the "drill-override.conf" file.
> {code}
> drill@host:/opt/drill/bin> drill-conf
> Error: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?] (state=,code=0)
> java.sql.SQLException: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?]
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:166)
> at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
> at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
> at
> sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
> at sqlline.Commands.connect(Commands.java:1083)
> at sqlline.Commands.connect(Commands.java:1015)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:742)
> at sqlline.SqlLine.initArgs(SqlLine.java:528)
>
[jira] [Comment Edited] (DRILL-5433) Authentication failed: Server requires authentication using [kerberos, plain]
[
https://issues.apache.org/jira/browse/DRILL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966640#comment-15966640
]
Sudheesh Katkam edited comment on DRILL-5433 at 4/12/17 9:21 PM:
-
The error says: *Server not found in Kerberos database (7) - UNKNOWN_SERVER*
This needs to be investigated.
Some pointers for more logging:
+ Create a file named */tmp/java.logging.txt* with contents:
{code}
javax.security.sasl.level=FINEST
handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=FINEST
{code}
+ Edit $DRILL_HOME/conf/drill-env.sh:
{code}
export SQLLINE_JAVA_OPTS = "$SQLLINE_JAVA_OPTS -Dsun.security.jgss.debug=true
-Dsun.security.krb5.debug=true
-Djava.util.logging.config.file=/tmp/java.logging.txt"
{code}
The console will now print more information. You could also look into Kerberos
logs; logging configuration is in */etc/krb5.conf* file.
(Also make sure you don't comment any sensitive info here.)
was (Author: sudheeshkatkam):
The error says: *Server not found in Kerberos database (7) - UNKNOWN_SERVER*
This needs to be investigated.
Some pointers for more logging:
+ Create a file named */tmp/java.logging.txt* with contents:
{code}
javax.security.sasl.level=FINEST
handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=FINEST
{code}
+ Edit $DRILL_HOME/conf/drill-env.sh:
{code}
export SQLLINE_JAVA_OPTS = "$SQLLINE_JAVA_OPTS -Dsun.security.jgss.debug=true
-Dsun.security.krb5.debug=true
-Djava.util.logging.config.file=/tmp/java.logging.txt"
{code}
Maybe the */tmp/java.logging.txt* will have information. And there is more info
to the console was well. You could also look into Kerberos logs; logging
configuration is in */etc/krb5.conf* file.
(Also make sure you don't comment any sensitive info here.)
> Authentication failed: Server requires authentication using [kerberos, plain]
> -
>
> Key: DRILL-5433
> URL: https://issues.apache.org/jira/browse/DRILL-5433
> Project: Apache Drill
> Issue Type: Task
> Components: Functions - Drill
>Affects Versions: 1.10.0
> Environment: OS: Redhat Linux 6.7, HDP 2.5.3, Kerberos enabled,
> Hardware: VmWare
>Reporter: Parag Darji
>Priority: Minor
> Labels: newbie, security
> Fix For: 1.10.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I've setup Apace drill 1.10.0 on RHEL 6.7, HDP 2.5.3, kerberos enabled
> I'm getting below error while running "drill-conf" or sqlline as user "drill"
> which is configured in the "drill-override.conf" file.
> {code}
> drill@host:/opt/drill/bin> drill-conf
> Error: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?] (state=,code=0)
> java.sql.SQLException: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?]
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:166)
> at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
> at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
> at
> sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
> at sqlline.Commands.connect(Commands.java:1083)
> at sqlline.Commands.connect(Commands.java:1015)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:742)
> at sqlline.SqlLine.initArgs(SqlLine.java:528)
> at sqlline.SqlLine.begin(SqlLine.java:596)
> at sqllin
[jira] [Comment Edited] (DRILL-5433) Authentication failed: Server requires authentication using [kerberos, plain]
[
https://issues.apache.org/jira/browse/DRILL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966640#comment-15966640
]
Sudheesh Katkam edited comment on DRILL-5433 at 4/12/17 9:22 PM:
-
The error says: *Server not found in Kerberos database (7) - UNKNOWN_SERVER*
This needs to be investigated.
Some pointers for more logging:
+ Create a file named */tmp/java.logging.txt* with contents:
{code}
javax.security.sasl.level=FINEST
handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=FINEST
{code}
+ Edit $DRILL_HOME/conf/drill-env.sh:
{code}
export SQLLINE_JAVA_OPTS = "$SQLLINE_JAVA_OPTS -Dsun.security.jgss.debug=true
-Dsun.security.krb5.debug=true
-Djava.util.logging.config.file=/tmp/java.logging.txt"
{code}
There will be more info printed to the console. You could also look into
Kerberos logs; logging configuration is in */etc/krb5.conf* file.
(Also make sure you don't comment any sensitive info here.)
was (Author: sudheeshkatkam):
The error says: *Server not found in Kerberos database (7) - UNKNOWN_SERVER*
This needs to be investigated.
Some pointers for more logging:
+ Create a file named */tmp/java.logging.txt* with contents:
{code}
javax.security.sasl.level=FINEST
handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=FINEST
{code}
+ Edit $DRILL_HOME/conf/drill-env.sh:
{code}
export SQLLINE_JAVA_OPTS = "$SQLLINE_JAVA_OPTS -Dsun.security.jgss.debug=true
-Dsun.security.krb5.debug=true
-Djava.util.logging.config.file=/tmp/java.logging.txt"
{code}
The console will now print more information. You could also look into Kerberos
logs; logging configuration is in */etc/krb5.conf* file.
(Also make sure you don't comment any sensitive info here.)
> Authentication failed: Server requires authentication using [kerberos, plain]
> -
>
> Key: DRILL-5433
> URL: https://issues.apache.org/jira/browse/DRILL-5433
> Project: Apache Drill
> Issue Type: Task
> Components: Functions - Drill
>Affects Versions: 1.10.0
> Environment: OS: Redhat Linux 6.7, HDP 2.5.3, Kerberos enabled,
> Hardware: VmWare
>Reporter: Parag Darji
>Priority: Minor
> Labels: newbie, security
> Fix For: 1.10.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I've setup Apace drill 1.10.0 on RHEL 6.7, HDP 2.5.3, kerberos enabled
> I'm getting below error while running "drill-conf" or sqlline as user "drill"
> which is configured in the "drill-override.conf" file.
> {code}
> drill@host:/opt/drill/bin> drill-conf
> Error: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?] (state=,code=0)
> java.sql.SQLException: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.NonTransientRpcException:
> javax.security.sasl.SaslException: Authentication failed: Server requires
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by
> javax.security.sasl.SaslException: Server requires authentication using
> [kerberos, plain]. Insufficient credentials?]
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:166)
> at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
> at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
> at
> sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
> at sqlline.Commands.connect(Commands.java:1083)
> at sqlline.Commands.connect(Commands.java:1015)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:742)
> at sqlline.SqlLine.initArgs(SqlLine.java:528)
> at sqlline.SqlLine.begin(SqlLine.java:596)
> at sqlline.SqlLine.start(SqlLine.java:375)
> at sqlline.Sql
[jira] [Commented] (DRILL-5429) Cache tableStats per query for MapR DB JSON Tables
[ https://issues.apache.org/jira/browse/DRILL-5429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966970#comment-15966970 ] ASF GitHub Bot commented on DRILL-5429: --- GitHub user ppadma opened a pull request: https://github.com/apache/drill/pull/817 DRILL-5429: Cache tableStats per query for MapR DB JSON Tables You can merge this pull request into a Git repository by running: $ git pull https://github.com/ppadma/drill DRILL-5429 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/817.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #817 commit c04c7d5acbb182a0152885711817cbe72ff6582a Author: Padma Penumarthy Date: 2017-04-11T23:34:14Z DRILL-5429: Cache tableStats per query for MapR DB JSON Tables > Cache tableStats per query for MapR DB JSON Tables > -- > > Key: DRILL-5429 > URL: https://issues.apache.org/jira/browse/DRILL-5429 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization, Storage - MapRDB >Affects Versions: 1.10.0 >Reporter: Padma Penumarthy > Fix For: 1.11.0 > > > For MapR DB JSON Tables, cache (per query) and reuse tableStats. Getting > tableStats is an expensive operation. Saving it and reusing it helps reduce > query latency. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5432) Want a memory format for PCAP files
[ https://issues.apache.org/jira/browse/DRILL-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967021#comment-15967021 ] Charles Givre commented on DRILL-5432: -- Hi Ted, I started working on getting the fields from your PCAP decoder mapped to Drill. Do you want me to share that in this JIRA? -- C > Want a memory format for PCAP files > --- > > Key: DRILL-5432 > URL: https://issues.apache.org/jira/browse/DRILL-5432 > Project: Apache Drill > Issue Type: New Feature >Reporter: Ted Dunning > > PCAP files [1] are the de facto standard for storing network capture data. In > security and protocol applications, it is very common to want to extract > particular packets from a capture for further analysis. > At a first level, it is desirable to query and filter by source and > destination IP and port or by protocol. Beyond that, however, it would be > very useful to be able to group packets by TCP session and eventually to look > at packet contents. For now, however, the most critical requirement is that > we should be able to scan captures at very high speed. > I previously wrote a (kind of working) proof of concept for a PCAP decoder > that did lazy deserialization and could traverse hundreds of MB of PCAP data > per second per core. This compares to roughly 2-3 MB/s for widely available > Apache-compatible open source PCAP decoders. > This JIRA covers the integration and extension of that proof of concept as a > Drill file format. > Initial work is available at https://github.com/mapr-demos/pcap-query > [1] https://en.wikipedia.org/wiki/Pcap -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5432) Want a memory format for PCAP files
[
https://issues.apache.org/jira/browse/DRILL-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967041#comment-15967041
]
Ted Dunning commented on DRILL-5432:
Charles,
I don't understand your comment. Tug reported the following output from a
sample file:
{code}
select *
from dfs.`data`.`airtunes.pcap`
limit 10
+---+--+--+-+-+---+---++---+
| Type | Network |Timestamp | dst_ip | src_ip
| src_port | dst_port | packet_length | data |
+---+--+--+-+-+---+---++---+
| TCP | 1| 2012-03-29 22:05:41.808 | /192.168.3.123 |
/192.168.3.107 | 51594 | 5000 | 78 | []|
| TCP | 1| 2012-03-29 22:05:41.808 | /192.168.3.107 |
/192.168.3.123 | 5000 | 51594 | 78 | []|
| TCP | 1| 2012-03-29 22:05:41.808 | /192.168.3.123 |
/192.168.3.107 | 51594 | 5000 | 66 | []|
+---+--+--+-+-+---+---++---+
{code}
What is your change going to do?
> Want a memory format for PCAP files
> ---
>
> Key: DRILL-5432
> URL: https://issues.apache.org/jira/browse/DRILL-5432
> Project: Apache Drill
> Issue Type: New Feature
>Reporter: Ted Dunning
>
> PCAP files [1] are the de facto standard for storing network capture data. In
> security and protocol applications, it is very common to want to extract
> particular packets from a capture for further analysis.
> At a first level, it is desirable to query and filter by source and
> destination IP and port or by protocol. Beyond that, however, it would be
> very useful to be able to group packets by TCP session and eventually to look
> at packet contents. For now, however, the most critical requirement is that
> we should be able to scan captures at very high speed.
> I previously wrote a (kind of working) proof of concept for a PCAP decoder
> that did lazy deserialization and could traverse hundreds of MB of PCAP data
> per second per core. This compares to roughly 2-3 MB/s for widely available
> Apache-compatible open source PCAP decoders.
> This JIRA covers the integration and extension of that proof of concept as a
> Drill file format.
> Initial work is available at https://github.com/mapr-demos/pcap-query
> [1] https://en.wikipedia.org/wiki/Pcap
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (DRILL-5432) Want a memory format for PCAP files
[ https://issues.apache.org/jira/browse/DRILL-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967044#comment-15967044 ] Charles Givre commented on DRILL-5432: -- Never mind. I misread the JIRA. > Want a memory format for PCAP files > --- > > Key: DRILL-5432 > URL: https://issues.apache.org/jira/browse/DRILL-5432 > Project: Apache Drill > Issue Type: New Feature >Reporter: Ted Dunning > > PCAP files [1] are the de facto standard for storing network capture data. In > security and protocol applications, it is very common to want to extract > particular packets from a capture for further analysis. > At a first level, it is desirable to query and filter by source and > destination IP and port or by protocol. Beyond that, however, it would be > very useful to be able to group packets by TCP session and eventually to look > at packet contents. For now, however, the most critical requirement is that > we should be able to scan captures at very high speed. > I previously wrote a (kind of working) proof of concept for a PCAP decoder > that did lazy deserialization and could traverse hundreds of MB of PCAP data > per second per core. This compares to roughly 2-3 MB/s for widely available > Apache-compatible open source PCAP decoders. > This JIRA covers the integration and extension of that proof of concept as a > Drill file format. > Initial work is available at https://github.com/mapr-demos/pcap-query > [1] https://en.wikipedia.org/wiki/Pcap -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (DRILL-5432) Want a memory format for PCAP files
[ https://issues.apache.org/jira/browse/DRILL-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967044#comment-15967044 ] Charles Givre edited comment on DRILL-5432 at 4/13/17 3:05 AM: --- I thought you were just posting the PCAP reader. I didn't realize that you had it working with Drill. The repo you posted just has the gitgnore and the license. Will you be sharing the code? I'd love to try this out. Thanks, -- C was (Author: cgivre): Never mind. I misread the JIRA. > Want a memory format for PCAP files > --- > > Key: DRILL-5432 > URL: https://issues.apache.org/jira/browse/DRILL-5432 > Project: Apache Drill > Issue Type: New Feature >Reporter: Ted Dunning > > PCAP files [1] are the de facto standard for storing network capture data. In > security and protocol applications, it is very common to want to extract > particular packets from a capture for further analysis. > At a first level, it is desirable to query and filter by source and > destination IP and port or by protocol. Beyond that, however, it would be > very useful to be able to group packets by TCP session and eventually to look > at packet contents. For now, however, the most critical requirement is that > we should be able to scan captures at very high speed. > I previously wrote a (kind of working) proof of concept for a PCAP decoder > that did lazy deserialization and could traverse hundreds of MB of PCAP data > per second per core. This compares to roughly 2-3 MB/s for widely available > Apache-compatible open source PCAP decoders. > This JIRA covers the integration and extension of that proof of concept as a > Drill file format. > Initial work is available at https://github.com/mapr-demos/pcap-query > [1] https://en.wikipedia.org/wiki/Pcap -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5432) Want a memory format for PCAP files
[ https://issues.apache.org/jira/browse/DRILL-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967049#comment-15967049 ] Ted Dunning commented on DRILL-5432: Wow. Missed that. New URL: https://github.com/mapr-demos/drill-pcap-format I will update the original comment so as to limit the number of people who are confused. > Want a memory format for PCAP files > --- > > Key: DRILL-5432 > URL: https://issues.apache.org/jira/browse/DRILL-5432 > Project: Apache Drill > Issue Type: New Feature >Reporter: Ted Dunning > > PCAP files [1] are the de facto standard for storing network capture data. In > security and protocol applications, it is very common to want to extract > particular packets from a capture for further analysis. > At a first level, it is desirable to query and filter by source and > destination IP and port or by protocol. Beyond that, however, it would be > very useful to be able to group packets by TCP session and eventually to look > at packet contents. For now, however, the most critical requirement is that > we should be able to scan captures at very high speed. > I previously wrote a (kind of working) proof of concept for a PCAP decoder > that did lazy deserialization and could traverse hundreds of MB of PCAP data > per second per core. This compares to roughly 2-3 MB/s for widely available > Apache-compatible open source PCAP decoders. > This JIRA covers the integration and extension of that proof of concept as a > Drill file format. > Initial work is available at https://github.com/mapr-demos/pcap-query > [1] https://en.wikipedia.org/wiki/Pcap -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (DRILL-5432) Want a memory format for PCAP files
[ https://issues.apache.org/jira/browse/DRILL-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Dunning updated DRILL-5432: --- Description: PCAP files [1] are the de facto standard for storing network capture data. In security and protocol applications, it is very common to want to extract particular packets from a capture for further analysis. At a first level, it is desirable to query and filter by source and destination IP and port or by protocol. Beyond that, however, it would be very useful to be able to group packets by TCP session and eventually to look at packet contents. For now, however, the most critical requirement is that we should be able to scan captures at very high speed. I previously wrote a (kind of working) proof of concept for a PCAP decoder that did lazy deserialization and could traverse hundreds of MB of PCAP data per second per core. This compares to roughly 2-3 MB/s for widely available Apache-compatible open source PCAP decoders. This JIRA covers the integration and extension of that proof of concept as a Drill file format. Initial work is available at https://github.com/mapr-demos/drill-pcap-format [1] https://en.wikipedia.org/wiki/Pcap was: PCAP files [1] are the de facto standard for storing network capture data. In security and protocol applications, it is very common to want to extract particular packets from a capture for further analysis. At a first level, it is desirable to query and filter by source and destination IP and port or by protocol. Beyond that, however, it would be very useful to be able to group packets by TCP session and eventually to look at packet contents. For now, however, the most critical requirement is that we should be able to scan captures at very high speed. I previously wrote a (kind of working) proof of concept for a PCAP decoder that did lazy deserialization and could traverse hundreds of MB of PCAP data per second per core. This compares to roughly 2-3 MB/s for widely available Apache-compatible open source PCAP decoders. This JIRA covers the integration and extension of that proof of concept as a Drill file format. Initial work is available at https://github.com/mapr-demos/pcap-query [1] https://en.wikipedia.org/wiki/Pcap > Want a memory format for PCAP files > --- > > Key: DRILL-5432 > URL: https://issues.apache.org/jira/browse/DRILL-5432 > Project: Apache Drill > Issue Type: New Feature >Reporter: Ted Dunning > > PCAP files [1] are the de facto standard for storing network capture data. In > security and protocol applications, it is very common to want to extract > particular packets from a capture for further analysis. > At a first level, it is desirable to query and filter by source and > destination IP and port or by protocol. Beyond that, however, it would be > very useful to be able to group packets by TCP session and eventually to look > at packet contents. For now, however, the most critical requirement is that > we should be able to scan captures at very high speed. > I previously wrote a (kind of working) proof of concept for a PCAP decoder > that did lazy deserialization and could traverse hundreds of MB of PCAP data > per second per core. This compares to roughly 2-3 MB/s for widely available > Apache-compatible open source PCAP decoders. > This JIRA covers the integration and extension of that proof of concept as a > Drill file format. > Initial work is available at https://github.com/mapr-demos/drill-pcap-format > [1] https://en.wikipedia.org/wiki/Pcap -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (DRILL-5429) Cache tableStats per query for MapR DB JSON Tables
[ https://issues.apache.org/jira/browse/DRILL-5429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zelaine Fong reassigned DRILL-5429: --- Assignee: Padma Penumarthy Reviewer: Gautam Kumar Parai Assigned Reviewer to [~gparai] > Cache tableStats per query for MapR DB JSON Tables > -- > > Key: DRILL-5429 > URL: https://issues.apache.org/jira/browse/DRILL-5429 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization, Storage - MapRDB >Affects Versions: 1.10.0 >Reporter: Padma Penumarthy >Assignee: Padma Penumarthy > Fix For: 1.11.0 > > > For MapR DB JSON Tables, cache (per query) and reuse tableStats. Getting > tableStats is an expensive operation. Saving it and reusing it helps reduce > query latency. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5434) IllegalStateException: Memory was leaked by query.
Khurram Faraaz created DRILL-5434:
-
Summary: IllegalStateException: Memory was leaked by query.
Key: DRILL-5434
URL: https://issues.apache.org/jira/browse/DRILL-5434
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
Issue a long running COUNT query.
While the query is being executed, stop the foreman drillbit, ./drillbit.sh stop
A memory leak is reported in the drillbit.log
Apache Drill 1.11.0
git.commit.id.abbrev=06e1522
Stack trace from drillbit.log
{noformat}
2017-04-13 06:14:36,828 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
2710e8b2-d4dc-1bee-016e-b69fd4966916: SELECT COUNT(*) FROM `twoKeyJsn.json`
2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,930 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,930 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,930 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,932 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,934 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using
1 threads. Time: 2ms total, 2.102992ms avg, 2ms max.
2017-04-13 06:14:36,934 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using
1 threads. Earliest start: 0.555000 μs, Latest start: 0.555000 μs, Average
start: 0.555000 μs .
2017-04-13 06:14:36,949 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0:
State change requested AWAITING_ALLOCATION --> RUNNING
2017-04-13 06:14:36,949 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0:
State to report: RUNNING
Thu Apr 13 06:14:40 UTC 2017 Terminating drillbit pid 5107
2017-04-13 06:14:40,756 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Received shutdown request.
2017-04-13 06:14:47,819 [pool-169-thread-2] INFO
o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup
io.netty.channel.nio.NioEventLoopGroup@4f3ee67c in 1024 ms
2017-04-13 06:14:47,819 [pool-169-thread-2] INFO
o.a.drill.exec.service.ServiceEngine - closed dataPool in 1024 ms
2017-04-13 06:14:49,806 [Drillbit-ShutdownHook#0] WARN
o.apache.drill.exec.work.WorkManager - Closing WorkManager but there are 1
running fragments.
2017-04-13 06:14:49,807 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0:
State change requested RUNNING --> FAILED
2017-04-13 06:14:49,807 [Drillbit-ShutdownHook#0] INFO
o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 6964, cache miss
count: 335, hit rate: 95%
2017-04-13 06:14:49,807 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0:
State change requested FAILED --> FINISHED
2017-04-13 06:14:49,809 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
Fragment 0:0
[Error Id: 94817261-98a9-4153-8b3a-2d9c95d80cc1 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
NullPointerException
Fragment 0:0
[Error Id: 94817261-98a9-4153-8b3a-2d9c95d80cc1 on centos-01.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0
