[jira] [Commented] (DRILL-7828) Refactor Pcap and Pcapng format plugin
[ https://issues.apache.org/jira/browse/DRILL-7828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314392#comment-17314392 ] ASF GitHub Bot commented on DRILL-7828: --- luocooong commented on pull request #2192: URL: https://github.com/apache/drill/pull/2192#issuecomment-812963481 @dzamo Hello. Is it possible to add a docs (when the [DRILL-6564](https://issues.apache.org/jira/browse/DRILL-6564) or related issues not resolved) that describe the troubleshooting about `conflict upgrade` at the phase of upgrade (from old plugin config). thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refactor Pcap and Pcapng format plugin > -- > > Key: DRILL-7828 > URL: https://issues.apache.org/jira/browse/DRILL-7828 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.19.0 > > > We can check the file's magic number, If that magic number indicates PCAP > then use the existing reader, If the magic number indicates PCAPNG, use the > refactored NG reader. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-6547) IllegalStateException: Tried to remove unmanaged buffer.
[ https://issues.apache.org/jira/browse/DRILL-6547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314381#comment-17314381 ] ASF GitHub Bot commented on DRILL-6547: --- luocooong commented on pull request #2183: URL: https://github.com/apache/drill/pull/2183#issuecomment-812961546 I agree. @oleg-zinovev Thanks for the PR. Then, Is it possible to check all the buffer issues in the file that contains the concat functions? thanks again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > IllegalStateException: Tried to remove unmanaged buffer. > > > Key: DRILL-6547 > URL: https://issues.apache.org/jira/browse/DRILL-6547 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.14.0 >Reporter: Robert Hou >Assignee: Pritesh Maker >Priority: Major > > This is the query: > select * from ( > select Index, concat(BinaryValue, 'aaa') NewVarcharValue from (select * from > dfs.`/drill/testdata/batch_memory/alltypes_large_1MB.parquet`)) d where > d.Index = 1; > This is the plan: > {noformat} > | 00-00Screen > 00-01 Project(Index=[$0], NewVarcharValue=[$1]) > 00-02SelectionVectorRemover > 00-03 Filter(condition=[=($0, 1)]) > 00-04Project(Index=[$0], NewVarcharValue=[CONCAT($1, 'aaa')]) > 00-05 Scan(groupscan=[ParquetGroupScan > [entries=[ReadEntryWithPath > [path=maprfs:///drill/testdata/batch_memory/alltypes_large_1MB.parquet]], > selectionRoot=maprfs:/drill/testdata/batch_memory/alltypes_large_1MB.parquet, > numFiles=1, numRowGroups=1, usedMetadataFile=false, columns=[`Index`, > `BinaryValue`]]]) > {noformat} > Here is the stack trace from drillbit.log: > {noformat} > 2018-06-27 13:55:03,291 [24cc0659-30b7-b290-7fae-ecb1c1f15c05:frag:0:0] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Tried to remove unmanaged buffer. > Fragment 0:0 > [Error Id: bc1f2f72-c31b-4b9a-964f-96dec9e0f388 on qa-node186.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Tried to remove unmanaged buffer. > Fragment 0:0 > [Error Id: bc1f2f72-c31b-4b9a-964f-96dec9e0f388 on qa-node186.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) > ~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361) > [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216) > [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327) > [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [na:1.8.0_161] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [na:1.8.0_161] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] > Caused by: java.lang.IllegalStateException: Tried to remove unmanaged buffer. > at > org.apache.drill.exec.ops.BufferManagerImpl.replace(BufferManagerImpl.java:50) > ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] > at io.netty.buffer.DrillBuf.reallocIfNeeded(DrillBuf.java:97) > ~[drill-memory-base-1.14.0-SNAPSHOT.jar:4.0.48.Final] > at > org.apache.drill.exec.test.generated.ProjectorGen4046.doEval(ProjectorTemplate.java:77) > ~[na:na] > at > org.apache.drill.exec.test.generated.ProjectorGen4046.projectRecords(ProjectorTemplate.java:67) > ~[na:na] > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:236) > ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:117) > ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:147) > ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) > ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] >
[jira] [Commented] (DRILL-7891) Drill Cluster Authentication failed. Incorrect credentials? PLAIN PAM authentication on node-2
[ https://issues.apache.org/jira/browse/DRILL-7891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314376#comment-17314376 ] Cong Luo commented on DRILL-7891: - [~noor.ciit...@hotmail.com] There are users that have discussed the troubleshooting (with auth) on the Slack recently, you can take a look on Slack channel. > Drill Cluster Authentication failed. Incorrect credentials? PLAIN PAM > authentication on node-2 > -- > > Key: DRILL-7891 > URL: https://issues.apache.org/jira/browse/DRILL-7891 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC, Functions - Drill >Affects Versions: 1.18.0 > Environment: drill.exec: { > cluster-id: "DrillMFSYS", > zk.connect: "zk1:2181,zk2:2181,zk3:2181" > impersonation: { > enabled: true, > max_chained_user_hops: 3 > }, > security: { > auth.mechanisms : ["PLAIN"], > user.encryption.sasl.enabled: false > }, > security.user.auth: { > enabled: true, > packages += "org.apache.drill.exec.rpc.user.security", > impl: "pam", > pam_profiles: ["sudo"] > } > } >Reporter: Noor >Priority: Major > Original Estimate: 504h > Remaining Estimate: 504h > > [Client-1] ERROR o.o.a.d.e.r.s.AuthenticationOutcomeListener - Authentication > with encryption context: Encryption: disabled , MaxWrappedSize: 65536 , > WrapSizeLimit: 0 using mechanism PLAIN failed with Authentication failed. > Incorrect credentials? [Details: Encryption: disabled , MaxWrappedSize: 65536 > , WrapSizeLimit: 0] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-6268) Drill-on-YARN client obtains HDFS URL incorrectly
[ https://issues.apache.org/jira/browse/DRILL-6268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314359#comment-17314359 ] ASF GitHub Bot commented on DRILL-6268: --- paul-rogers commented on pull request #2139: URL: https://github.com/apache/drill/pull/2139#issuecomment-812928381 @cgivre, Revisiting this one. The fix is probably not correct. As explained earlier, the goal is to 1) use the YARN config by default, unless, 2) overridden in the DoY config file. Here is the default config from `drill-on-yarn-defaults.conf`: ``` drill.yarn: { ... dfs: { connection: "" app-dir: "/user/drill" } ``` The code says: ```java String dfsConnection = config.getString(DrillOnYarnConfig.DFS_CONNECTION); try { if (DoYUtil.isBlank(dfsConnection)) { fs = FileSystem.get(yarnConf); ``` So, if the `dfs.connection` property is blank, use the one from the YARN config file. Again, why might there be a different DoY value? Because some users push apps to multiple servers, and the DoY config should be sufficient to do so, without having to have multiple different YARN configs available. (If, in practice, people use only one config, we can remove these DoY configs if not needed. But, let's assume they are needed.) So, the question is, why did the user see the bug which was reported? Where did the `"hdfs://localhost/"` value come from? **That** is the bug we need to fix. The answer seems to be that someone used `drill-on-yarn-example.conf` as their config, without inspecting if the *example* values are useful. (This is an *example*, not a *default*.): ``` drill.yarn: { ... dfs: { # Connection to the distributed file system. Defaults to work with # a single-node Drill on the local machine. # Omit this if you want to get the configuration either from the # Hadoop config (set with config-dir above) or from the # $DRILL_HOME/core-site.xml. connection: "hdfs://localhost/" ``` Why is that being used? The proper "default" file is `drill-on-yarn-override.conf` from `distribution`. But, it looks like the `component.xml` file is missing a line. So, maybe the user renamed the example file to `drill-on-yarn-override.conf`. We need: ```xml src/main/resources/drill-on-yarn-override.conf conf 0640 ``` With the above Maven fix, we don't need to change the code: the code does what it is supposed to do, if given a proper (blank) config entry. An "extra for experts" fix is to add the updated port number to the example file above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Drill-on-YARN client obtains HDFS URL incorrectly > - > > Key: DRILL-6268 > URL: https://issues.apache.org/jira/browse/DRILL-6268 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.13.0 >Reporter: Paul Rogers >Assignee: Charles Givre >Priority: Major > > The Drill-on-YARN client must upload files to HDFS so that YARN can localize > them. The code that does so is in {{DfsFacade}}. This code obtains the URL > twice. The first time is correct: > {code} > private void loadYarnConfig() { > ... > URI fsUri = FileSystem.getDefaultUri( yarnConf ); > if(fsUri.toString().startsWith("file:/")) { > System.err.println("Warning: Default DFS URI is for a local file > system: " + fsUri); > } > } > } > {code} > The {{fsUri}} returned is {{hdfs://localhost:9000}}, which is the correct > value for an out-of-the-box Hadoop 2.9.0 install after following [these > instructions|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html]. > The instructions have the reader explicitly set the port number to 9000: > {code} > > > fs.defaultFS > hdfs://localhost:9000 > > > {code} > The other place that gets the URL, this time or real, is > {{DfsFacade.connect()}}: > {code} > String dfsConnection = config.getString(DrillOnYarnConfig.DFS_CONNECTION); > {code} > This value comes back as {{hdfs://localhost/}}, which causes HDFS to try to > connect on port 8020 (the Hadoop default), resulting in the following error: > {noformat} > Connecting to DFS... Connected. > Uploading /Users/paulrogers/bin/apache-drill-1.13.0.tar.gz to > /users/drill/apache-drill-1.13.0.tar.gz ... Failed. > Failed to upload Drill archive > Caused by: Failed to create DFS directory: /users/drill > Ca
[jira] [Commented] (DRILL-6547) IllegalStateException: Tried to remove unmanaged buffer.
[ https://issues.apache.org/jira/browse/DRILL-6547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314358#comment-17314358 ] ASF GitHub Bot commented on DRILL-6547: --- paul-rogers commented on pull request #2183: URL: https://github.com/apache/drill/pull/2183#issuecomment-812925090 @luocooong, thanks for the question. I raised several issues. I'll leave it to the team to decide whether to worry about those issues, or just wait for someone to report new bugs. My tendency is, once I become aware of an issue, to fix it if doing so is quick; I find that more efficient than waiting for some poor user to hit the issue, debug the problem, report the bug, get the bug prioritized, and go through another code review cycle. But, that's just me. My suggestion would be to inspect and, if necessary, fix all the buffer issues in the file that contains the concat functions. An "extra for experts" is to change the code in the concat functions that works with null values to *not* inspect the null value, as, in the general case, it can be invalid. Of course, we don't know that these are problems, probably because we have no good tests. So, if you are of the "test driven development" persuasion, I'd encourage writing a few tests to verify behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > IllegalStateException: Tried to remove unmanaged buffer. > > > Key: DRILL-6547 > URL: https://issues.apache.org/jira/browse/DRILL-6547 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.14.0 >Reporter: Robert Hou >Assignee: Pritesh Maker >Priority: Major > > This is the query: > select * from ( > select Index, concat(BinaryValue, 'aaa') NewVarcharValue from (select * from > dfs.`/drill/testdata/batch_memory/alltypes_large_1MB.parquet`)) d where > d.Index = 1; > This is the plan: > {noformat} > | 00-00Screen > 00-01 Project(Index=[$0], NewVarcharValue=[$1]) > 00-02SelectionVectorRemover > 00-03 Filter(condition=[=($0, 1)]) > 00-04Project(Index=[$0], NewVarcharValue=[CONCAT($1, 'aaa')]) > 00-05 Scan(groupscan=[ParquetGroupScan > [entries=[ReadEntryWithPath > [path=maprfs:///drill/testdata/batch_memory/alltypes_large_1MB.parquet]], > selectionRoot=maprfs:/drill/testdata/batch_memory/alltypes_large_1MB.parquet, > numFiles=1, numRowGroups=1, usedMetadataFile=false, columns=[`Index`, > `BinaryValue`]]]) > {noformat} > Here is the stack trace from drillbit.log: > {noformat} > 2018-06-27 13:55:03,291 [24cc0659-30b7-b290-7fae-ecb1c1f15c05:frag:0:0] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Tried to remove unmanaged buffer. > Fragment 0:0 > [Error Id: bc1f2f72-c31b-4b9a-964f-96dec9e0f388 on qa-node186.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Tried to remove unmanaged buffer. > Fragment 0:0 > [Error Id: bc1f2f72-c31b-4b9a-964f-96dec9e0f388 on qa-node186.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) > ~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361) > [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216) > [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327) > [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [na:1.8.0_161] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [na:1.8.0_161] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] > Caused by: java.lang.IllegalStateException: Tried to remove unmanaged buffer. > at > org.apache.drill.exec.ops.BufferManagerImpl.replace(BufferManagerImpl.java:50) > ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT] > at io.netty.buffer.DrillBuf.reallocIfNeeded(DrillBuf.java:97) > ~[drill-memory-base-1.14.0-SNAPSHOT.jar:4.0.48.Final] > at > org.apache.drill.exec.test.generated.ProjectorGen4046.doEval(Proje
[jira] [Commented] (DRILL-7828) Refactor Pcap and Pcapng format plugin
[ https://issues.apache.org/jira/browse/DRILL-7828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314357#comment-17314357 ] ASF GitHub Bot commented on DRILL-7828: --- paul-rogers commented on pull request #2192: URL: https://github.com/apache/drill/pull/2192#issuecomment-812924446 Where are we on this one? We should avoid the "boil the ocean" temptation: the desire to fix too many things in one PR. If this PR will break upgrades, then the team can decide if that is OK for the handful of people who might be affected. Maybe document recovery in release notes, or be prepared to describe the steps on Slack or the mail lists. I am concerned about table functions. They are obscure, but are supposed to work. I'd like to see a test added so we can be sure that the non-public config variables do, in fact, work. Of course, you could argue that we can skip that until someone complains, then file another bug because, again, there are likely few users of this plugin and likely none of them use table functions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refactor Pcap and Pcapng format plugin > -- > > Key: DRILL-7828 > URL: https://issues.apache.org/jira/browse/DRILL-7828 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.19.0 > > > We can check the file's magic number, If that magic number indicates PCAP > then use the existing reader, If the magic number indicates PCAPNG, use the > refactored NG reader. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7888) query.json returns an incorrect error message when the query fails
[ https://issues.apache.org/jira/browse/DRILL-7888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314241#comment-17314241 ] ASF GitHub Bot commented on DRILL-7888: --- vvysotskyi commented on pull request #2194: URL: https://github.com/apache/drill/pull/2194#issuecomment-812851057 Hi @luocooong, yes, it is ready to review. @paul-rogers, could you please look one more time? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > query.json returns an incorrect error message when the query fails > -- > > Key: DRILL-7888 > URL: https://issues.apache.org/jira/browse/DRILL-7888 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.19.0 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.19.0 > > > When submitting a query that should fail to \{{query.json}} endpoint, it > always returns the same response regardless of the actual failure reason: > {code:json} > { > "errorMessage": "Current context not Array but root" > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7888) query.json returns an incorrect error message when the query fails
[ https://issues.apache.org/jira/browse/DRILL-7888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314240#comment-17314240 ] ASF GitHub Bot commented on DRILL-7888: --- luocooong commented on pull request #2194: URL: https://github.com/apache/drill/pull/2194#issuecomment-812850694 @vvysotskyi Hi. Do we have any new revisions ready to submit? Or It's only ready to request a review again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > query.json returns an incorrect error message when the query fails > -- > > Key: DRILL-7888 > URL: https://issues.apache.org/jira/browse/DRILL-7888 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.19.0 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.19.0 > > > When submitting a query that should fail to \{{query.json}} endpoint, it > always returns the same response regardless of the actual failure reason: > {code:json} > { > "errorMessage": "Current context not Array but root" > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7828) Refactor Pcap and Pcapng format plugin
[ https://issues.apache.org/jira/browse/DRILL-7828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314237#comment-17314237 ] ASF GitHub Bot commented on DRILL-7828: --- luocooong commented on pull request #2192: URL: https://github.com/apache/drill/pull/2192#issuecomment-812849597 @vdiravka Hi. Do we have any new revisions ready to submit? AFAIK, Now there is only an issue and related to the `conflict upragde`, Is right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refactor Pcap and Pcapng format plugin > -- > > Key: DRILL-7828 > URL: https://issues.apache.org/jira/browse/DRILL-7828 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.19.0 > > > We can check the file's magic number, If that magic number indicates PCAP > then use the existing reader, If the magic number indicates PCAPNG, use the > refactored NG reader. -- This message was sent by Atlassian Jira (v8.3.4#803005)