[jira] [Commented] (HIVE-24484) Upgrade Hadoop to 3.3.1 And Tez to 0.10.2
[ https://issues.apache.org/jira/browse/HIVE-24484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601737#comment-17601737 ] Takanobu Asanuma commented on HIVE-24484: - Great work! Do you have any plan to move to Hadoop-3.3.1 & Tez-0.10.2 in Hive-3.x? > Upgrade Hadoop to 3.3.1 And Tez to 0.10.2 > -- > > Key: HIVE-24484 > URL: https://issues.apache.org/jira/browse/HIVE-24484 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 15.05h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-25920) Bump Xerce2 to 2.12.2
[ https://issues.apache.org/jira/browse/HIVE-25920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25920 started by Takanobu Asanuma. --- > Bump Xerce2 to 2.12.2 > - > > Key: HIVE-25920 > URL: https://issues.apache.org/jira/browse/HIVE-25920 > Project: Hive > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Xerce 2.12.1 has high-level vulnerabilities. > * [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-23437] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25920) Bump Xerce2 to 2.12.2
[ https://issues.apache.org/jira/browse/HIVE-25920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma reassigned HIVE-25920: --- > Bump Xerce2 to 2.12.2 > - > > Key: HIVE-25920 > URL: https://issues.apache.org/jira/browse/HIVE-25920 > Project: Hive > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > > Xerce 2.12.1 has high-level vulnerabilities. > * [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-23437] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25649) Backport HIVE-20638 and HIVE-22090 to branch-3 to upgrade Jetty to 9.3.27
[ https://issues.apache.org/jira/browse/HIVE-25649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-25649: Status: Patch Available (was: Open) > Backport HIVE-20638 and HIVE-22090 to branch-3 to upgrade Jetty to 9.3.27 > - > > Key: HIVE-25649 > URL: https://issues.apache.org/jira/browse/HIVE-25649 > Project: Hive > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Jetty in branch-3 is very old. So let's update to 9.3.27, which is the same > as the master branch. Although 9.3.27 is not the latest version, it fixes > some vulnerabilities. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25649) Backport HIVE-20638 and HIVE-22090 to branch-3 to upgrade Jetty to 9.3.27
[ https://issues.apache.org/jira/browse/HIVE-25649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma reassigned HIVE-25649: --- > Backport HIVE-20638 and HIVE-22090 to branch-3 to upgrade Jetty to 9.3.27 > - > > Key: HIVE-25649 > URL: https://issues.apache.org/jira/browse/HIVE-25649 > Project: Hive > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > > Jetty in branch-3 is very old. So let's update to 9.3.27, which is the same > as the master branch. Although 9.3.27 is not the latest version, it fixes > some vulnerabilities. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25636) Bump Xerce2 to 2.12.1
[ https://issues.apache.org/jira/browse/HIVE-25636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-25636: Status: Patch Available (was: Open) > Bump Xerce2 to 2.12.1 > - > > Key: HIVE-25636 > URL: https://issues.apache.org/jira/browse/HIVE-25636 > Project: Hive > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Xerce 2.9.1 has high-level vulnerabilities. > * [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-4002] > * [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-0881] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25636) Bump Xerce2 to 2.12.1
[ https://issues.apache.org/jira/browse/HIVE-25636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma reassigned HIVE-25636: --- > Bump Xerce2 to 2.12.1 > - > > Key: HIVE-25636 > URL: https://issues.apache.org/jira/browse/HIVE-25636 > Project: Hive > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > > Xerce 2.9.1 has high-level vulnerabilities. > * [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-4002] > * [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-0881] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-15831) LLAP: Fix a problem of the output of LlapDump
[
https://issues.apache.org/jira/browse/HIVE-15831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871377#comment-15871377
]
Takanobu Asanuma commented on HIVE-15831:
-
Thanks again, Sergey!
> LLAP: Fix a problem of the output of LlapDump
> -
>
> Key: HIVE-15831
> URL: https://issues.apache.org/jira/browse/HIVE-15831
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Fix For: 2.2.0
>
> Attachments: HIVE-15831.1.patch
>
>
> The output of llapdump doesn't include the first column at the moment.
> This is an output example:
> {noformat}
> url: jdbc:hive2://{hiveserver2_interactive}
> user: hive
> query: select column1, column2 from table_a limit 10
> column1:int,column2:float
> Results:
> , 258.0
> , 645.0
> , 652.0
> , 1336.096
> , 97.1828
> , 1864.4
> , 10428.4
> , 2346.3
> , 425.1
> , 228.9
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15831) LLAP: Fix a problem of the output of LlapDump
[
https://issues.apache.org/jira/browse/HIVE-15831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HIVE-15831:
Status: Patch Available (was: Open)
> LLAP: Fix a problem of the output of LlapDump
> -
>
> Key: HIVE-15831
> URL: https://issues.apache.org/jira/browse/HIVE-15831
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HIVE-15831.1.patch
>
>
> The output of llapdump doesn't include the first column at the moment.
> This is an output example:
> {noformat}
> url: jdbc:hive2://{hiveserver2_interactive}
> user: hive
> query: select column1, column2 from table_a limit 10
> column1:int,column2:float
> Results:
> , 258.0
> , 645.0
> , 652.0
> , 1336.096
> , 97.1828
> , 1864.4
> , 10428.4
> , 2346.3
> , 425.1
> , 228.9
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15831) LLAP: Fix a problem of the output of LlapDump
[
https://issues.apache.org/jira/browse/HIVE-15831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HIVE-15831:
Attachment: HIVE-15831.1.patch
Uploaded the first patch. It also removes unused imports.
> LLAP: Fix a problem of the output of LlapDump
> -
>
> Key: HIVE-15831
> URL: https://issues.apache.org/jira/browse/HIVE-15831
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HIVE-15831.1.patch
>
>
> The output of llapdump doesn't include the first column at the moment.
> This is an output example:
> {noformat}
> url: jdbc:hive2://{hiveserver2_interactive}
> user: hive
> query: select column1, column2 from table_a limit 10
> column1:int,column2:float
> Results:
> , 258.0
> , 645.0
> , 652.0
> , 1336.096
> , 97.1828
> , 1864.4
> , 10428.4
> , 2346.3
> , 425.1
> , 228.9
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15831) LLAP: Fix a problem of the output of LlapDump
[
https://issues.apache.org/jira/browse/HIVE-15831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HIVE-15831:
Description:
The output of llapdump doesn't include the first column at the moment.
This is an output example:
{noformat}
url: jdbc:hive2://{hiveserver2_interactive}
user: hive
query: select column1, column2 from table_a limit 10
column1:int,column2:float
Results:
, 258.0
, 645.0
, 652.0
, 1336.096
, 97.1828
, 1864.4
, 10428.4
, 2346.3
, 425.1
, 228.9
{noformat}
> LLAP: Fix a problem of the output of LlapDump
> -
>
> Key: HIVE-15831
> URL: https://issues.apache.org/jira/browse/HIVE-15831
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>
> The output of llapdump doesn't include the first column at the moment.
> This is an output example:
> {noformat}
> url: jdbc:hive2://{hiveserver2_interactive}
> user: hive
> query: select column1, column2 from table_a limit 10
> column1:int,column2:float
> Results:
> , 258.0
> , 645.0
> , 652.0
> , 1336.096
> , 97.1828
> , 1864.4
> , 10428.4
> , 2346.3
> , 425.1
> , 228.9
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15831) LLAP: Fix a problem of the output of LlapDump
[
https://issues.apache.org/jira/browse/HIVE-15831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HIVE-15831:
Environment: (was: The output of llapdump doesn't include the first
column at the moment.
This is an output example:
{noformat}
url: jdbc:hive2://{hiveserver2_interactive}
user: hive
query: select column1, column2 from table_a limit 10
column1:int,column2:float
Results:
, 258.0
, 645.0
, 652.0
, 1336.096
, 97.1828
, 1864.4
, 10428.4
, 2346.3
, 425.1
, 228.9
{noformat})
> LLAP: Fix a problem of the output of LlapDump
> -
>
> Key: HIVE-15831
> URL: https://issues.apache.org/jira/browse/HIVE-15831
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-15831) LLAP: Fix a problem of the output of LlapDump
[
https://issues.apache.org/jira/browse/HIVE-15831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma reassigned HIVE-15831:
---
> LLAP: Fix a problem of the output of LlapDump
> -
>
> Key: HIVE-15831
> URL: https://issues.apache.org/jira/browse/HIVE-15831
> Project: Hive
> Issue Type: Bug
> Components: llap
> Environment: The output of llapdump doesn't include the first column
> at the moment.
> This is an output example:
> {noformat}
> url: jdbc:hive2://{hiveserver2_interactive}
> user: hive
> query: select column1, column2 from table_a limit 10
> column1:int,column2:float
> Results:
> , 258.0
> , 645.0
> , 652.0
> , 1336.096
> , 97.1828
> , 1864.4
> , 10428.4
> , 2346.3
> , 425.1
> , 228.9
> {noformat}
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15817) Fix LlapDump classpath in llapdum.sh
[ https://issues.apache.org/jira/browse/HIVE-15817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855178#comment-15855178 ] Takanobu Asanuma commented on HIVE-15817: - Thanks for reviewing and committing, [~sershe]! > Fix LlapDump classpath in llapdum.sh > > > Key: HIVE-15817 > URL: https://issues.apache.org/jira/browse/HIVE-15817 > Project: Hive > Issue Type: Bug >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma > Labels: llap > Fix For: 2.2.0 > > Attachments: HIVE-15817.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15817) Fix LlapDump classpath in llapdum.sh
[ https://issues.apache.org/jira/browse/HIVE-15817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-15817: Status: Patch Available (was: Open) > Fix LlapDump classpath in llapdum.sh > > > Key: HIVE-15817 > URL: https://issues.apache.org/jira/browse/HIVE-15817 > Project: Hive > Issue Type: Bug >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma > Attachments: HIVE-15817.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15817) Fix LlapDump classpath in llapdum.sh
[ https://issues.apache.org/jira/browse/HIVE-15817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-15817: Attachment: HIVE-15817.1.patch Uploaded the first patch. > Fix LlapDump classpath in llapdum.sh > > > Key: HIVE-15817 > URL: https://issues.apache.org/jira/browse/HIVE-15817 > Project: Hive > Issue Type: Bug >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma > Attachments: HIVE-15817.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-15817) Fix LlapDump classpath in llapdum.sh
[ https://issues.apache.org/jira/browse/HIVE-15817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma reassigned HIVE-15817: --- > Fix LlapDump classpath in llapdum.sh > > > Key: HIVE-15817 > URL: https://issues.apache.org/jira/browse/HIVE-15817 > Project: Hive > Issue Type: Bug >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11527: Attachment: HIVE-11527.11.patch Somehow Jenkins didn't run. I uploaded the latest patch again. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.11.patch, > HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11527: Attachment: (was: HIVE-11527.11.patch) > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11527: Status: Open (was: Patch Available) > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.11.patch, > HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11527: Status: Patch Available (was: Open) > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.11.patch, > HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15354811#comment-15354811 ] Takanobu Asanuma commented on HIVE-11527: - [~thejas] I think that to extend typeDesc is a fairly large change and it should be done in another jira. (Has it already begun?) How about using typeName for simplicity at first? > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.11.patch, > HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11527: Status: Patch Available (was: Open) > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.11.patch, > HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11527: Status: Open (was: Patch Available) > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.11.patch, > HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341082#comment-15341082 ] Takanobu Asanuma commented on HIVE-11527: - BTW, somehow Jenkins did not run for HIVE-11527.10.patch. This time Jenkins likely to run for the new patch. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.11.patch, > HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11527: Status: Open (was: Patch Available) > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.11.patch, > HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11527: Status: Patch Available (was: Open) > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.11.patch, > HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11527: Attachment: HIVE-11527.11.patch [~thejas] I uploaded a new patch in this jira and RB. And I left some comments in RB. Please could you check it? > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.11.patch, > HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338896#comment-15338896 ] Takanobu Asanuma commented on HIVE-11527: - [~thejas] Thank you for your review! I will consider handling backward compatibility. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331259#comment-15331259 ] Takanobu Asanuma commented on HIVE-11527: - [~vgumashta] I uploaded the latest patch in this jira and in RB. I also replied your comments in RB. Thanks. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11527: Attachment: HIVE-11527.10.patch > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316217#comment-15316217 ] Takanobu Asanuma commented on HIVE-11527: - [~vgumashta], [~gopalv] Thank you for your review and thoughts! I will follow your decision. The last patch is conflicting the latest master branch. I will resolve the conflicts and fix some of the minor codes. Then, I will attach the patch on this jira to kick the Jenkins. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13938) TestJdbcWithMiniHA is being killed by PTEST after 1h running
[
https://issues.apache.org/jira/browse/HIVE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316100#comment-15316100
]
Takanobu Asanuma commented on HIVE-13938:
-
[~spena], [~ashutoshc]
Thank you for filing this problem and sorry for late reply. I checked
HIVE-13917.
> TestJdbcWithMiniHA is being killed by PTEST after 1h running
>
>
> Key: HIVE-13938
> URL: https://issues.apache.org/jira/browse/HIVE-13938
> Project: Hive
> Issue Type: Test
>Reporter: Sergio Peña
> Attachments: hive.log, maven-test.txt
>
>
> {{TestJdbcWithMiniHA}} does not return any results on Jenkins because it's
> been killed after 1 hour running.
> We should investigate what is happening with this test, or if it is an issue
> with ptest environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303694#comment-15303694 ] Takanobu Asanuma commented on HIVE-11527: - [~sershe], [~vgumashta] Thank you very much! I will wait. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288712#comment-15288712 ] Takanobu Asanuma commented on HIVE-11527: - [~sershe] Sorry for late reply. I updated the patch on RB. I fixed some codes and merged the latest master branch. Please review it again. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15253152#comment-15253152 ] Takanobu Asanuma commented on HIVE-11527: - Thanks! I'll check it later. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[
https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242862#comment-15242862
]
Takanobu Asanuma commented on HIVE-11527:
-
Hi, [~sershe], [~vgumashta], and other experts.
I uploaded a new patch in Review Board just now. I think I have almost finished
implementing the features. So I'd like to explain the summary of all my
implementation.
*How to use the bypass*
When {{hive.server2.webhdfs.bypass.enabled}} is true, users can use the bypass.
The default is false.
*Tests*
I added some unit tests in {{TestJdbcWithMiniHS2}}, {{TestJdbcWithMiniMr}} and
{{TestJdbcWithMiniHA}}. They will help debugging.
*Changing thrift API*
I added three optional variables as the response from HS2 to JDBC drivers after
executing a query.
* {{finalDirUri}}: a pass of the directory which has the final data
* {{haConf}}: configurations for Namenode HA
* {{typeName}}: a type name for complex columns
*Decoding data*
Decoding data in clients side is implemented in {{HiveQueryResultSet}}. In the
latest patch, to avoid complex codes, clients can use the bypass only when the
final data is SequenceFile which is the default format of final data. I think
it is rare that clients change the default format.
*Handling HA*
When Namenode is HA, clients need some configurations which are in the cluster
side. They are passed in {{Driver#getFinalDirName}}.
*Unable to use the bypass*
In some cases, it is difficult to use the bypass. I wrote the cases in
{{TestJdbcWithMiniHS2#testUnableUseBypassCase}}. {{Driver#useBypass}} judges
whether clients use the bypass.
Some optimizations and bugs may remain. Please review the patch when you are
free.
Thank you very much for reading this long comment!
> bypass HiveServer2 thrift interface for query results
> -
>
> Key: HIVE-11527
> URL: https://issues.apache.org/jira/browse/HIVE-11527
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Sergey Shelukhin
>Assignee: Takanobu Asanuma
> Attachments: HIVE-11527.WIP.patch
>
>
> Right now, HS2 reads query results and returns them to the caller via its
> thrift API.
> There should be an option for HS2 to return some pointer to results (an HDFS
> link?) and for the user to read the results directly off HDFS inside the
> cluster, or via something like WebHDFS outside the cluster
> Review board link: https://reviews.apache.org/r/40867
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13268) Add a HA mini cluster type in MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15229674#comment-15229674 ] Takanobu Asanuma commented on HIVE-13268: - Thank you very much for reviewing and committing! > Add a HA mini cluster type in MiniHS2 > - > > Key: HIVE-13268 > URL: https://issues.apache.org/jira/browse/HIVE-13268 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Minor > Fix For: 2.1.0 > > Attachments: HIVE-13268.1.patch, HIVE-13268.2.patch, > HIVE-13268.3.patch, HIVE-13268.4.patch, HIVE-13268.5.patch > > > We need a HA mini cluster for unit tests. This jira is for implimenting that > in MiniHS2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13268) Add a HA mini cluster type in MiniHS2
[
https://issues.apache.org/jira/browse/HIVE-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225487#comment-15225487
]
Takanobu Asanuma commented on HIVE-13268:
-
Hmm... the failed tests of TestMiniTezCliDriver are passed in my local computer.
{noformat}
mvn test -Dtest=TestMiniTezCliDriver
-Dqfile=alter_merge_orc.q,auto_join30.q,auto_sortmerge_join_11.q,insert_into1.q,metadata_only_queries_with_filters.q,scriptfile1.q,tez_join.q,tez_union_dynamic_partition.q,union3.q,vector_aggregate_without_gby.q,vector_data_types.q,vector_decimal_precision.q,vector_interval_arithmetic.q,vector_mapjoin_reduce.q,vectorized_math_funcs.q
---
T E S T S
---
Running org.apache.hadoop.hive.cli.TestMiniTezCliDriver
Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 663.359 sec -
in org.apache.hadoop.hive.cli.TestMiniTezCli
Driver
Results :
Tests run: 15, Failures: 0, Errors: 0, Skipped: 0
{noformat}
[~sershe]
Could you take a look at it again?
> Add a HA mini cluster type in MiniHS2
> -
>
> Key: HIVE-13268
> URL: https://issues.apache.org/jira/browse/HIVE-13268
> Project: Hive
> Issue Type: Test
> Components: Tests
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Attachments: HIVE-13268.1.patch, HIVE-13268.2.patch,
> HIVE-13268.3.patch, HIVE-13268.4.patch, HIVE-13268.5.patch
>
>
> We need a HA mini cluster for unit tests. This jira is for implimenting that
> in MiniHS2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13268) Add a HA mini cluster type in MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-13268: Attachment: HIVE-13268.4.patch Last patch(HIVE-13268.3.patch) has a problem for handling errors. I fixed it and uploaded a new patch. > Add a HA mini cluster type in MiniHS2 > - > > Key: HIVE-13268 > URL: https://issues.apache.org/jira/browse/HIVE-13268 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Minor > Attachments: HIVE-13268.1.patch, HIVE-13268.2.patch, > HIVE-13268.3.patch, HIVE-13268.4.patch > > > We need a HA mini cluster for unit tests. This jira is for implimenting that > in MiniHS2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13268) Add a HA mini cluster type in MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-13268: Attachment: HIVE-13268.3.patch [~sershe] Thanks for the review. TestPigHBaseStorageHandler was passed in my local computer. I uploaded a new patch and want to see new Jenkins results. It includes a new unit test class which is cloned from TestJdbcWithMiniMR. > Add a HA mini cluster type in MiniHS2 > - > > Key: HIVE-13268 > URL: https://issues.apache.org/jira/browse/HIVE-13268 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Minor > Attachments: HIVE-13268.1.patch, HIVE-13268.2.patch, > HIVE-13268.3.patch > > > We need a HA mini cluster for unit tests. This jira is for implimenting that > in MiniHS2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197055#comment-15197055 ] Takanobu Asanuma commented on HIVE-11527: - Hi [~sershe], [~vgumashta] Thanks to the discussion with [~jingzhao], I implemented features for handling HA. I uploaded a WIP patch on RB. Please review it. I will continue the rest of the work. ・Until now, I assumed that intermediate results' format is the text format. But we need to make jdbc clients decode other file formats. That is even more important since sequence file is the default format for intermediate results, which was currently implemented in HIVE-1608. ・Considering multiple intermediate files. ・Adding some unit tests. HA tests depend on HIVE-13268 (Could you also review this?). Thanks. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13268) Add a HA mini cluster type in MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-13268: Attachment: HIVE-13268.2.patch Sorry, the first patch includes debugging codes. I uploaded a new patch. > Add a HA mini cluster type in MiniHS2 > - > > Key: HIVE-13268 > URL: https://issues.apache.org/jira/browse/HIVE-13268 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Minor > Attachments: HIVE-13268.1.patch, HIVE-13268.2.patch > > > We need a HA mini cluster for unit tests. This jira is for implimenting that > in MiniHS2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13268) Add a HA mini cluster type in MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-13268: Attachment: HIVE-13268.1.patch Hi [~sershe] I uploaded the first patch. Please could you check it? After fixed this jira, I will write unit tests in HIVE-11527. :) > Add a HA mini cluster type in MiniHS2 > - > > Key: HIVE-13268 > URL: https://issues.apache.org/jira/browse/HIVE-13268 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Minor > Attachments: HIVE-13268.1.patch > > > We need a HA mini cluster for unit tests. This jira is for implimenting that > in MiniHS2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13268) Add a HA mini cluster type in MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-13268: Status: Patch Available (was: Open) > Add a HA mini cluster type in MiniHS2 > - > > Key: HIVE-13268 > URL: https://issues.apache.org/jira/browse/HIVE-13268 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Minor > Attachments: HIVE-13268.1.patch > > > We need a HA mini cluster for unit tests. This jira is for implimenting that > in MiniHS2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13268) Add a HA mini cluster type in MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-13268: Issue Type: Test (was: Bug) > Add a HA mini cluster type in MiniHS2 > - > > Key: HIVE-13268 > URL: https://issues.apache.org/jira/browse/HIVE-13268 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Minor > > We need a HA mini cluster for unit tests. This jira is for implimenting that > in MiniHS2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171794#comment-15171794 ] Takanobu Asanuma commented on HIVE-11527: - Hi [~sershe], [~jingzhao] Sorry for my late action. As I said in the last comment, since jdbc clients may not be able to resolve the HA namespace, HiveServer2 should resolve it with WebHdfsFileSystem and return the final FQDN to jdbc clients. But currently, WebHdfsFileSystem does not have the API like that. So I want to implement the public API in WebHdfsFileSystem. What do you think about that? > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138325#comment-15138325 ] Takanobu Asanuma commented on HIVE-11527: - Thank you for the advice, Jing! It makes sense to me. If IIUC, we should assume that jdbc client doesn't always have the same configuration files as the cluster side. So we should create the final URI in HiveServer2 while considering the cases Jing suggested, and return to jdbc client. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138323#comment-15138323 ] Takanobu Asanuma commented on HIVE-11527: - Sergey, I see. I will reconsider handling multiple files. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132194#comment-15132194 ] Takanobu Asanuma commented on HIVE-11527: - Thanks. I'd like to get Jing's advice. And I left some questions on RB about result files. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133522#comment-15133522 ] Takanobu Asanuma commented on HIVE-11527: - I forgot to publish ... I published a while ago. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133712#comment-15133712 ] Takanobu Asanuma commented on HIVE-11527: - i appreciate your support! > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15130262#comment-15130262 ] Takanobu Asanuma commented on HIVE-11527: - Hi [~sershe] I updated the patch based on your advice. Please review it. I merged latest master branch every time I updated patch. I realized that this may be not a good way since it becomes hard to review... Sorry for bother you. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123016#comment-15123016 ] Takanobu Asanuma commented on HIVE-11527: - Oh, I'm sorry. In the last patch, I changed TCLIService.thrift only and forgot to regenerate the related files... I'll fix it soon. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088825#comment-15088825 ] Takanobu Asanuma commented on HIVE-11527: - I submmit a new patch on RB. This is two addtinal updates for bypass. * Until the last patch, when bypass is enabled, JDBC driver can't handle tables which has complex types such as array, struct and map. This patch modifies thrift api to use typename to handle complex types. * This patch uses LineNumberReader to handle Statement#setFetchSize for bypass. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082626#comment-15082626 ] Takanobu Asanuma commented on HIVE-11527: - Hi, [~vgumashta] [~sershe]. I wish you a happy new year. I updated the patch on RB. * It fixes codes to use webhdfs when we use HDFS. * It improves logging in HiveQueryResultSet. * When hive.exec.compress.output is true, the final data is compressed. In this case, we use current implementation for the moment. Prease review it. Thanks. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070904#comment-15070904 ] Takanobu Asanuma commented on HIVE-11527: - [~vgumashta] [~sershe] Thank you for the comments. > it seems you're not using webhdfs to read the results as intended in the > original proposal. Sorry, I misunderstood how to use webhdfs. I will reconsider it. > there's a better overload to use in logging. Thanks. I will use it. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063628#comment-15063628 ] Takanobu Asanuma commented on HIVE-11527: - I updated the patch in Review Board. Please review it when you're free. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044688#comment-15044688 ] Takanobu Asanuma commented on HIVE-11527: - I appreciate for the detailed review. I will fix the codes. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035755#comment-15035755 ] Takanobu Asanuma commented on HIVE-11527: - Thank you for letting me know. I posted on https://reviews.apache.org/r/40867/ . Please review it when you have time. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11527: Description: Right now, HS2 reads query results and returns them to the caller via its thrift API. There should be an option for HS2 to return some pointer to results (an HDFS link?) and for the user to read the results directly off HDFS inside the cluster, or via something like WebHDFS outside the cluster Review board link: https://reviews.apache.org/r/40867 was: Right now, HS2 reads query results and returns them to the caller via its thrift API. There should be an option for HS2 to return some pointer to results (an HDFS link?) and for the user to read the results directly off HDFS inside the cluster, or via something like WebHDFS outside the cluster > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11527: Attachment: HIVE-11527.WIP.patch Hi [~sershe], [~vgumashta] I uploaded a WIP patch. * I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass. * I still have not considered security such as Kerberos and SSL at present. * I have not implement Statement#setFetchSize for bypass yet. * I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests. Please could you review it? And let me know if you have any comments for this patch. Thank you. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15023746#comment-15023746 ] Takanobu Asanuma commented on HIVE-11527: - Thank you for helping me! I understand it. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15015634#comment-15015634 ] Takanobu Asanuma commented on HIVE-11527: - I'd like to share my thoughts for this work. I'm going to add a new thrift api to get a uri of the result data. New data flow is like below: 1. When JDBC client calls HiveQueryResultSet#next(), JDBC driver calls the new thrift API. 2. HiveServer2 returns the path of the result data. IIUC, FetchWork has the uri. NOTE: Like "select * from tablename limit 10", some queries don't run MR/Tez/Spark jobs and the uri is different from the final data. In this case, we use current implementation. 3. JDBC driver gets the uri and downloads the data by WebHDFS. 4. JDBC driver decodes the data and creates RowSet. I'm writing codes and I will upload a wip patch next week. If you have any thoughts on this jira, please share it with me. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15013056#comment-15013056 ] Takanobu Asanuma commented on HIVE-11527: - [~sershe]] [~vgumashta] Thanks for taking this jira into consideration. I will check HIVE-12049 and related jiras. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962743#comment-14962743 ] Takanobu Asanuma commented on HIVE-11527: - Hi, [~sershe] Can I work on this jira? > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11822) vectorize NVL UDF
[ https://issues.apache.org/jira/browse/HIVE-11822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14961591#comment-14961591 ] Takanobu Asanuma commented on HIVE-11822: - [~sershe], [~gopalv] Thank you for all your support! > vectorize NVL UDF > - > > Key: HIVE-11822 > URL: https://issues.apache.org/jira/browse/HIVE-11822 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Fix For: 2.0.0 > > Attachments: HIVE-11822.1.patch, HIVE-11822.2.patch, > HIVE-11822.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11822) vectorize NVL UDF
[
https://issues.apache.org/jira/browse/HIVE-11822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956656#comment-14956656
]
Takanobu Asanuma commented on HIVE-11822:
-
I don't know why the difference happened...
{{mvn test -Dtest=TestMiniTezCliDriver -Dqfile=vector_coalesce.q -Phadoop-2}}
was passed in my laptop computer.
> vectorize NVL UDF
> -
>
> Key: HIVE-11822
> URL: https://issues.apache.org/jira/browse/HIVE-11822
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Takanobu Asanuma
> Attachments: HIVE-11822.1.patch, HIVE-11822.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11822) vectorize NVL UDF
[ https://issues.apache.org/jira/browse/HIVE-11822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11822: Attachment: HIVE-11822.3.patch Thanks, Sergey. I updated the patch. Does this problem often occur? Should I set the same java version on my laptop computer? > vectorize NVL UDF > - > > Key: HIVE-11822 > URL: https://issues.apache.org/jira/browse/HIVE-11822 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11822.1.patch, HIVE-11822.2.patch, > HIVE-11822.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11822) vectorize NVL UDF
[ https://issues.apache.org/jira/browse/HIVE-11822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11822: Attachment: HIVE-11822.1.patch I uploaded an initial patch for vectorized NVL. This patch also includes qtests for NVL and Coalesce. Please could you review it? While reading the codes, I realized we can completely reuse VectorCoalesce for vectorized NVL. So I didn't write VectorNvl in this patch. I think there are not many overheads even if we use VectorCoalesce as VectorNvl. What do you think? > vectorize NVL UDF > - > > Key: HIVE-11822 > URL: https://issues.apache.org/jira/browse/HIVE-11822 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11822.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11822) vectorize NVL UDF
[ https://issues.apache.org/jira/browse/HIVE-11822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941025#comment-14941025 ] Takanobu Asanuma commented on HIVE-11822: - Sorry for late. Please could you confirm my plan? - implement VectorNVL like VectorCoalesce. - modify Vectorizer and VectorizationContext for VectorNVL. - add some unit tests and qtests as Gopal said. > vectorize NVL UDF > - > > Key: HIVE-11822 > URL: https://issues.apache.org/jira/browse/HIVE-11822 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11822) vectorize NVL UDF
[ https://issues.apache.org/jira/browse/HIVE-11822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747076#comment-14747076 ] Takanobu Asanuma commented on HIVE-11822: - [~gopalv] Thanks for the assignment and kind advice. > vectorize NVL UDF > - > > Key: HIVE-11822 > URL: https://issues.apache.org/jira/browse/HIVE-11822 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11822) vectorize NVL UDF
[ https://issues.apache.org/jira/browse/HIVE-11822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744874#comment-14744874 ] Takanobu Asanuma commented on HIVE-11822: - Hi [~sershe], [~gopalv] Thank you for creating this jira. I'd like to work on it. Please could you assign it to me? I agree with [~gopalv]. COALESCE is a generalization of the NVL function. Thanks. > vectorize NVL UDF > - > > Key: HIVE-11822 > URL: https://issues.apache.org/jira/browse/HIVE-11822 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11538) Add an option to skip init script while running tests
[
https://issues.apache.org/jira/browse/HIVE-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712378#comment-14712378
]
Takanobu Asanuma commented on HIVE-11538:
-
Actually the problem was solved. When importing the maven project in IntelliJ,
I was asked to Select profile and chose both hadoop-2 and hadoop-1. This
caused the problem. Then I chose only hadoop-2, it was solved. I'm sorry to
have caused trouble. Tahnks anyway.
Add an option to skip init script while running tests
-
Key: HIVE-11538
URL: https://issues.apache.org/jira/browse/HIVE-11538
Project: Hive
Issue Type: Improvement
Components: Testing Infrastructure
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Fix For: 2.0.0
Attachments: HIVE-11538.2.patch, HIVE-11538.3.patch, HIVE-11538.patch
{{q_test_init.sql}} has grown over time. Now, it takes substantial amount of
time. When debugging a particular query which doesn't need such
initialization, this delay is annoyance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11538) Add an option to skip init script while running tests
[
https://issues.apache.org/jira/browse/HIVE-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710690#comment-14710690
]
Takanobu Asanuma commented on HIVE-11538:
-
Hi, [~ashutoshc].
I'm using IntelliJ and it can't resolve {{FileStatus#isDirectory}} in
{{QTestUtil}}. I built the project with {{mvn clean install -DskipTests
-Phadoop-2}}. The build was passed and hadoop-core-1.2.1 which has
{{FileStatus}} was imported. But {{FileStatus}} doesn't have the method in
version 1.2.1. Could you please check this problem?
Add an option to skip init script while running tests
-
Key: HIVE-11538
URL: https://issues.apache.org/jira/browse/HIVE-11538
Project: Hive
Issue Type: Improvement
Components: Testing Infrastructure
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Fix For: 2.0.0
Attachments: HIVE-11538.2.patch, HIVE-11538.3.patch, HIVE-11538.patch
{{q_test_init.sql}} has grown over time. Now, it takes substantial amount of
time. When debugging a particular query which doesn't need such
initialization, this delay is annoyance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10602) optimize PTF for GC
[ https://issues.apache.org/jira/browse/HIVE-10602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696708#comment-14696708 ] Takanobu Asanuma commented on HIVE-10602: - Hi, [~sershe]. I'd like to try this jira. Could you assign it to me? Thanks. optimize PTF for GC --- Key: HIVE-10602 URL: https://issues.apache.org/jira/browse/HIVE-10602 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin see HIVE-10600 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
