[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144144#comment-16144144
]
Chen Liang commented on HDFS-12321:
---
The failed tests all passed in my local run. {{TestSQLQuery}} seems to complain
not able to find ozoneQuery.json file when Jenkins was running it. But
interestingly the other test {{TestKSMSQLCli}} that also uses this file seem to
have passed.
> Ozone : debug cli: add support to load user-provided SQL query
> --
>
> Key: HDFS-12321
> URL: https://issues.apache.org/jira/browse/HDFS-12321
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12321-HDFS-7240.001.patch,
> HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch,
> HDFS-12321-HDFS-7240.004.patch, HDFS-12321-HDFS-7240.005.patch,
> HDFS-12321-HDFS-7240.006.patch, HDFS-12321-HDFS-7240.007.patch,
> HDFS-12321-HDFS-7240.008.patch, HDFS-12321-HDFS-7240.009.patch,
> HDFS-12321-HDFS-7240.010.patch
>
>
> This JIRA extends SQL CLI to support loading a user-provided file that
> includes any sql query the user wants to run on the SQLite db obtained by
> converting Ozone metadata db.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142132#comment-16142132
]
Hadoop QA commented on HDFS-12321:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} hadoop-hdfs-project generated 0 new + 469 unchanged
- 2 fixed = 469 total (was 471) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 45s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 26s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}118m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.scm.TestArchive |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.ozone.ksm.TestSQLQuery |
| | hadoop.hdfs.server.namenode.ha.TestHAMetrics |
| | hadoop.ozone.scm.node.TestQueryNode |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12321 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12883776/HDFS-12321-HDFS-7240.010.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml shellcheck shelldocs |
| uname | Linux b0226698f906 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16141095#comment-16141095
]
Hadoop QA commented on HDFS-12321:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
2s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 17m
37s{color} | {color:red} root in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
52s{color} | {color:green} hadoop-hdfs-project generated 0 new + 469 unchanged
- 2 fixed = 469 total (was 471) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-hdfs-project: The patch generated 94 new
+ 0 unchanged - 0 fixed = 94 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
18s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 54s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}106m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | org.apache.hadoop.ozone.scm.cli.SQLCommand.execute(Connection,
CommandLine, Writer) may fail to clean up java.sql.ResultSet Obligation to
clean up resource created at SQLCommand.java:to clean up java.sql.ResultSet
Obligation to clean up resource created at SQLCommand.java:[line 213] is not
discharged |
| Failed junit tests | hadoop.scm.TestArchive |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplicat
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16141037#comment-16141037
]
Hadoop QA commented on HDFS-12321:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
1s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 19m
19s{color} | {color:red} root in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
19s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
16s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
9s{color} | {color:green} root generated 0 new + 1316 unchanged - 2 fixed =
1316 total (was 1318) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 40s{color} | {color:orange} root: The patch generated 96 new + 43 unchanged
- 1 fixed = 139 total (was 44) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 5s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 18s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
56s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.scm.TestArchive |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.ozone.ksm.TestSQLquery |
| | hadoop.ozone.ksm.TestKSMSQLCli |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.diskbalancer.co
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140877#comment-16140877
]
Anu Engineer commented on HDFS-12321:
-
[~vagarychen] This looks fantastic. Thanks for getting this working. Some very
minor comments on patch v8.
* DiskBalancerCLI.java, TestDBInputFormat.java, TestDbClasses.java -- There is
an accidental edit in each of these files.
* SQLQueryCLI -- Help string which embeds Json in the help string needs to be
updated to the new format.
* We should add a default OzoneQuery.json with the queries we know, feel free
to do this in a later JIRA too.
* SQLQueryCLI#parseOptions - {{.withArgName("property=value")}} -- should we
actually have the argument name here ? Probably a for loop to add each one of
them ? if this is a place holder for code review, then it is correct.
* cmds.json : nit : this comment seems to be incomplete.
{noformat}
"desc":"Count the total number of buckets under a specific volume.\nArgs :
-volumeName -- Volume name, required."
{noformat}
Can you also test stand alone on a DB and see if it works, that is independent
of the Unit tests?
> Ozone : debug cli: add support to load user-provided SQL query
> --
>
> Key: HDFS-12321
> URL: https://issues.apache.org/jira/browse/HDFS-12321
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12321-HDFS-7240.001.patch,
> HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch,
> HDFS-12321-HDFS-7240.004.patch, HDFS-12321-HDFS-7240.005.patch,
> HDFS-12321-HDFS-7240.006.patch, HDFS-12321-HDFS-7240.007.patch,
> HDFS-12321-HDFS-7240.008.patch
>
>
> This JIRA extends SQL CLI to support loading a user-provided file that
> includes any sql query the user wants to run on the SQLite db obtained by
> converting Ozone metadata db.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140449#comment-16140449
]
Anu Engineer commented on HDFS-12321:
-
bq. the CLI Parser must be able to recognize these two options "-volumeName"
and "-bucketName"
if you look at the patch v7, you will see that we first parse the SQL and pull
out the arguments out of the SQL. if we get an SQL of the form , "Text
:argument1: Text :argument2:", the SQLCommand#processSQL will return a SQL of
the form "Text ? Text ?" and will also populate an arguments array -- which
will look like this,
{{Arguments[] = [ argument1, argument2].}}
Once we have this argument array with us, we can add them to the Options via a
function called Options.addOption -- so these options are available on the
command line. It does not matter how many options we have since we will invoke
addOption on each one of these arguments.
bq. something to parse options by their given order (which would be a bit odd
to me...). In this case, things might become easier.
We don't need to read the options in a given order at all. We know the order in
which we need to bind the SQL arguments. The array of arguments already gives
us that information. So we can get the right argument from the command line and
bind them at the right location.
Please let me know if it makes sense for me to create a working code rather a
prototype? Would that be helpful?
> Ozone : debug cli: add support to load user-provided SQL query
> --
>
> Key: HDFS-12321
> URL: https://issues.apache.org/jira/browse/HDFS-12321
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12321-HDFS-7240.001.patch,
> HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch,
> HDFS-12321-HDFS-7240.004.patch, HDFS-12321-HDFS-7240.005.patch,
> HDFS-12321-HDFS-7240.006.patch, HDFS-12321-HDFS-7240.007.patch
>
>
> This JIRA extends SQL CLI to support loading a user-provided file that
> includes any sql query the user wants to run on the SQLite db obtained by
> converting Ozone metadata db.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139451#comment-16139451
]
Chen Liang commented on HDFS-12321:
---
bq. Now we need to add these two as two options of the command. Before we run
the SQL, we have to bind the arguments in the right location (JDBC limitations
where this is done via position, so we already remember the arguments
positions. So when user runs the command he/she will do it like this-- "hdfs
sqlquery commandName -volumeName myVolume -bucketName myBucket" – inside code
we will do a getOption("volumeName") which will return the myVolume to us,
which will become part of SQL query. Same thing for bucketName, and sql will
contain myBucket.
Yes, this is exactly I was talking about, the part that requires code is: the
CLI Parser *must be able to recognize these two options* "-volumeName" and
"-bucketName", such that they can be parsed by Parser class. i.e. we call
{{CommandLine cmd = BasicParser#parse(options)}} followed by
{{cmd.getOptionValue("volumeName")}}, then we obtain the corresponding command
line values of volumeName. But how is this going to happen? i.e. how can the
parser recognize this option?
Take another example, there is command like {{hdfs scm -container -create -c
Contaienr0}}. Take {{-create}} for instance. The way it worked is to explicitly
code {{-create}} as an option that can be recognized by CLI. So similarly here
"-volumeName" and "-bucketName" must also be coded I guess. But these two
options come with the user-provided sql query and are only meaningful for this
user-provided query, thus they can only be provided along with this sql query,
probably in the json file I suppose. Then CLI takes these options and adds them
as recognizable options. This is the whole point I was try to make.
Maybe there is a way for {{BasicParser}} or something to parse options by their
given order (which would be a bit odd to me...). In this case, things might
become easier.
> Ozone : debug cli: add support to load user-provided SQL query
> --
>
> Key: HDFS-12321
> URL: https://issues.apache.org/jira/browse/HDFS-12321
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12321-HDFS-7240.001.patch,
> HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch,
> HDFS-12321-HDFS-7240.004.patch, HDFS-12321-HDFS-7240.005.patch,
> HDFS-12321-HDFS-7240.006.patch, HDFS-12321-HDFS-7240.007.patch
>
>
> This JIRA extends SQL CLI to support loading a user-provided file that
> includes any sql query the user wants to run on the SQLite db obtained by
> converting Ozone metadata db.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139405#comment-16139405
]
Hadoop QA commented on HDFS-12321:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 24m
5s{color} | {color:red} root in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
19s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
5s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
19s{color} | {color:green} hadoop-hdfs-project generated 0 new + 469 unchanged
- 2 fixed = 469 total (was 471) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 57s{color} | {color:orange} hadoop-hdfs-project: The patch generated 105 new
+ 1 unchanged - 0 fixed = 106 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
33s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 59s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
26s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}164m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | org.apache.hadoop.ozone.scm.cli.SQLCommand.execute(Connection,
CommandLine) may fail to clean up java.sql.Statement Obligation to clean up
resource created at SQLCommand.java:clean up java.sql.Statement Obligation to
clean up resource created at SQLCommand.java:[line 174] is not discharged |
| Failed junit tests | hadoop.scm.TestArchive |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.T
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139360#comment-16139360
]
Anu Engineer commented on HDFS-12321:
-
bq. One thing I found tricky was what's the best way to take arguments from
command line for the sql queries. For example if I want to get count of bucket
from volume X, but ignoring volumes starts with Y. So we want to take X and Y
from command line
This is my thought on that, I was planning to create an
{code}
"args" {
"name1": "description of name1",
"name2": "description of name2",
"name3": "description of name3"
}
{code}
This becomes an args object which we can add to Options as we can verify at the
code startup. When we parse the SQL we can only recognise the arguments that
are part of the args object. Similarly, when we get command line arguments, we
can verify since the SQLCommand and the Args object will tell us what the args
is.
bq. But I as far as I remember, withLongOpt is specifying the longer version
name of the command. For example, x.withLongOpt("createContainer").create("c")
means you can type either -c or -createContainer, they both recognized as same
createContainer command. If this is correct, then this code snippet is
basically keeping overwriting withLongOpt with each of the arguments.
Quite possibly you are right on this, My patch is to illustrate the idea, last
time we had a discussion you had commented it seemed tricky.
bq. Now the issue is, given two argument from command, say M and N, what is the
best way to assign the arguments, i.e.whether it is X=M,Y=N or the other way.
Ideally, I think they should be specified with options. Such as writing on
command line with something like -x M -y N, then we know it is X=M and Y=N. But
this means json command file must also provide -x and -y options and how to
parse them. This adds some complexity to the code which I was targeting for a
separate patch.
May be I am missing something here. Let us take a concerte example. Let us say
we have two arguments, say volumeName and bucketName in a SQL.
These will now become part of args object in JSON as well as part of SQL
statement via *:volumeName:* and *:bucketName:*.
Now we need to add these two as two options of the command. Before we run the
SQL, we have to bind the arguments in the right location (JDBC limitations
where this is done via position, so we already remember the arguments
positions. So when user runs the command he/she will do it like this-- "hdfs
sqlquery commandName -volumeName myVolume -bucketName myBucket" -- inside code
we will do a getOption("volumeName") which will return the myVolume to us,
which will become part of SQL query. Same thing for bucketName, and sql will
contain myBucket.
bq. I thought of this too, but it seemed a bit tricky with the Options class.
Will need to continue looking closer on how to do this.
I was just exploring how hard this would be, and now that we have got a
prototype that explores all the JSON to command path and possibly how to deal
with args, may be you can drive this to completion? Please let me know if you
like me to fix the issues that you have already pointed out.
> Ozone : debug cli: add support to load user-provided SQL query
> --
>
> Key: HDFS-12321
> URL: https://issues.apache.org/jira/browse/HDFS-12321
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12321-HDFS-7240.001.patch,
> HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch,
> HDFS-12321-HDFS-7240.004.patch, HDFS-12321-HDFS-7240.005.patch,
> HDFS-12321-HDFS-7240.006.patch, HDFS-12321-HDFS-7240.007.patch
>
>
> This JIRA extends SQL CLI to support loading a user-provided file that
> includes any sql query the user wants to run on the SQLite db obtained by
> converting Ozone metadata db.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139316#comment-16139316
]
Chen Liang commented on HDFS-12321:
---
Thanks Anu for the patch, I thought about some of these changes also, but was
planning for another patch. One thing I found tricky was what's the best way to
take arguments from command line for the sql queries. For example if I want to
get count of bucket from volume X, but ignoring volumes starts with Y. So we
want to take X and Y from command line. I think ideally we specify in json file
a sql like "select count(*) as count from bucketInfo where volumeName=X and
bucketName!=Y" and then take X and Y from commandline.
Now the issue is, given two argument from command, say M and N, what is the
best way to assign the arguments, i.e.whether it is X=M,Y=N or the other way.
Ideally, I think they should be specified with options. Such as writing on
command line with something like -x M -y N, then we know it is X=M and Y=N. But
this means json command file must also provide -x and -y options and how to
parse them. This adds some complexity to the code which I was targeting for a
separate patch.
Seems v007 is taking arguments in order. Namely, if in sql string X comes
before Y and in command line M comes before N, then X=M,Y=N, right? I guess I'm
fine with this. But still, I'm not quite sure about this part:
{code}
command.getArguments().forEach((String argument) -> {
if (org.apache.commons.lang3.StringUtils.isNoneBlank(argument)) {
//TODO check if the Argument already exists, if so don't add it
// and just use it.
commandBuilder.withLongOpt(argument);
}
});
{code}
I might be totally wrong here. But I as far as I remember, {{withLongOpt}} is
specifying the longer version name of the command. For example,
{{x.withLongOpt("createContainer").create("c")}} means you can type either -c
or -createContainer, they both recognized as same createContainer command. If
this is correct, then this code snippet is basically keeping overwriting
withLongOpt with each of the arguments. In the end, the arguments do not seem
to be correctly taken and passed from command line to command instance.
Additionally, that change to {{TestPlanner}} seems unnecessary.
> Ozone : debug cli: add support to load user-provided SQL query
> --
>
> Key: HDFS-12321
> URL: https://issues.apache.org/jira/browse/HDFS-12321
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12321-HDFS-7240.001.patch,
> HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch,
> HDFS-12321-HDFS-7240.004.patch, HDFS-12321-HDFS-7240.005.patch,
> HDFS-12321-HDFS-7240.006.patch, HDFS-12321-HDFS-7240.007.patch
>
>
> This JIRA extends SQL CLI to support loading a user-provided file that
> includes any sql query the user wants to run on the SQLite db obtained by
> converting Ozone metadata db.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139002#comment-16139002
]
Hadoop QA commented on HDFS-12321:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 15m
33s{color} | {color:red} root in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} hadoop-hdfs-project generated 0 new + 469 unchanged
- 2 fixed = 469 total (was 471) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 37s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 42s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}111m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.scm.TestArchive |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.ozone.scm.node.TestQueryNode |
| | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.ozone.TestMiniOzoneCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12321 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12883379/HDFS-12321-HDFS-7240.006.patch
|
| Opti
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136074#comment-16136074
]
Hadoop QA commented on HDFS-12321:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
4s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} hadoop-hdfs-project generated 0 new + 449 unchanged
- 2 fixed = 449 total (was 451) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 42s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 18s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.scm.TestArchive |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
| | hadoop.ozone.scm.node.TestQueryNode |
| Timed out junit tests | org.apache.hadoop.ozone.TestMiniOzoneCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12321 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882979/HDFS-12321-HDFS-7240.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux ed76489b303e 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 7d13259 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20790/a
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135912#comment-16135912
]
Anu Engineer commented on HDFS-12321:
-
[~vagarychen] Thanks for adding this amazing capability to ozone debug tool. I
am sure this one tool will save countless hours for each one of us. Thank you
very much for taking care of this. The patch looks excellent. Some minor
comments.
* SQLQueryCLI
{code}
if (filePath == null) {
throw new IOException("Requires setting " +
OZONE_CLI_USER_COMMAND_FILE_KEY);
}
{code}
Instead of throwing here, can we please print the text that is commented out
right after this line, and return a
failure code to the OS ? So say "Requires setting " +
OZONE_CLI_USER_COMMAND_FILE_KEY, and then print out the text that you already
have.
* *At some point in the future* -- not in this patch, we might have to support
passing arguments to queries. So that json might need the argument names -- so
that user can specify them in the command line. For example, think about a
query that lists all buckets under a volume, you need a way to pass volume name
into this function.
* Just a *very wild thought*: I am not suggesting we have to do this, just
sharing a thought that I had. How about we read the json file and add the
commands to the Options -- like {{getOptions}}, no idea how hard it will be,
but it will look like we have first class commands via Json. If you would like
to explore this more, we can chat about this.
> Ozone : debug cli: add support to load user-provided SQL query
> --
>
> Key: HDFS-12321
> URL: https://issues.apache.org/jira/browse/HDFS-12321
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12321-HDFS-7240.001.patch,
> HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch,
> HDFS-12321-HDFS-7240.004.patch
>
>
> This JIRA extends SQL CLI to support loading a user-provided file that
> includes any sql query the user wants to run on the SQLite db obtained by
> converting Ozone metadata db.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135764#comment-16135764
]
Hadoop QA commented on HDFS-12321:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
11s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 38s{color}
| {color:red} hadoop-hdfs-project generated 2 new + 449 unchanged - 2 fixed =
451 total (was 451) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 41s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.scm.TestArchive |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.ozone.scm.node.TestQueryNode |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12321 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882944/HDFS-12321-HDFS-7240.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux c88924f1cc65 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 7d13259 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/20788/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/207
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133725#comment-16133725
]
Hadoop QA commented on HDFS-12321:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 49s{color}
| {color:red} hadoop-hdfs-project generated 2 new + 449 unchanged - 2 fixed =
451 total (was 451) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
27s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 7 new + 0
unchanged - 0 fixed = 7 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 41s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 58s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
22s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | org.apache.hadoop.ozone.scm.cli.SQLQueryCLI.run(String[]) may fail to
clean up java.sql.ResultSet Obligation to clean up resource created at
SQLQueryCLI.java:up java.sql.ResultSet Obligation to clean up resource created
at SQLQueryCLI.java:[line 182] is not discharged |
| | org.apache.hadoop.ozone.scm.cli.SQLQueryCLI.run(String[]) may fail to
clean up java.sql.Statement Obligation to clean up resource created at
SQLQueryCLI.java:up java.sql.Statement Obligation to clean up resource created
at SQLQueryCLI.java:[line 181] is not discharged |
| | org.apache.hadoop.ozone.scm.cli.SQLQueryCLI.run(String[]) may fail to
close Statement At SQLQueryCLI.java:Statement At SQLQueryCLI.java:[line 181] |
| | Unread field:field be static? At SQLQueryCLI.java:[line 48] |
| | Unread field:field be static? At
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133583#comment-16133583
]
Hadoop QA commented on HDFS-12321:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HDFS-12321 does not apply to HDFS-7240. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-12321 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882664/HDFS-12321-HDFS-7240.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20761/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone : debug cli: add support to load user-provided SQL query
> --
>
> Key: HDFS-12321
> URL: https://issues.apache.org/jira/browse/HDFS-12321
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12321-HDFS-7240.001.patch
>
>
> This JIRA extends SQL CLI to support loading a user-provided file that
> includes any sql query the user wants to run on the SQLite db obtained by
> converting Ozone metadata db.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
