[jira] [Updated] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-9153:
-
Attachment: HIVE-9153.1-spark.patch
This patch should further improve spark performance by avoid retrieving MapWork
from plan file.
Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]
-
Key: HIVE-9153
URL: https://issues.apache.org/jira/browse/HIVE-9153
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Brock Noland
Assignee: Rui Li
Attachments: HIVE-9153.1-spark.patch, screenshot.PNG
The default InputFormat is {{CombineHiveInputFormat}} and thus HOS uses this.
However, Tez uses {{HiveInputFormat}}. Since tasks are relatively cheap in
Spark, it might make sense for us to use {{HiveInputFormat}} as well. We
should evaluate this on a query which has many input splits such as {{select
count(\*) from store_sales where something is not null}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-9153:
-
Status: Patch Available (was: Open)
Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]
-
Key: HIVE-9153
URL: https://issues.apache.org/jira/browse/HIVE-9153
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Brock Noland
Assignee: Rui Li
Attachments: HIVE-9153.1-spark.patch, screenshot.PNG
The default InputFormat is {{CombineHiveInputFormat}} and thus HOS uses this.
However, Tez uses {{HiveInputFormat}}. Since tasks are relatively cheap in
Spark, it might make sense for us to use {{HiveInputFormat}} as well. We
should evaluate this on a query which has many input splits such as {{select
count(\*) from store_sales where something is not null}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-9153:
---
Attachment: HIVE-9153.1-spark.patch
Uploading the patch again to test some change I made to ptest.
Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]
-
Key: HIVE-9153
URL: https://issues.apache.org/jira/browse/HIVE-9153
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Brock Noland
Assignee: Rui Li
Attachments: HIVE-9153.1-spark.patch, HIVE-9153.1-spark.patch,
screenshot.PNG
The default InputFormat is {{CombineHiveInputFormat}} and thus HOS uses this.
However, Tez uses {{HiveInputFormat}}. Since tasks are relatively cheap in
Spark, it might make sense for us to use {{HiveInputFormat}} as well. We
should evaluate this on a query which has many input splits such as {{select
count(\*) from store_sales where something is not null}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-9153:
-
Attachment: HIVE-9153.2.patch
Upload trunk patch
Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]
-
Key: HIVE-9153
URL: https://issues.apache.org/jira/browse/HIVE-9153
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Brock Noland
Assignee: Rui Li
Attachments: HIVE-9153.1-spark.patch, HIVE-9153.1-spark.patch,
HIVE-9153.2.patch, screenshot.PNG
The default InputFormat is {{CombineHiveInputFormat}} and thus HOS uses this.
However, Tez uses {{HiveInputFormat}}. Since tasks are relatively cheap in
Spark, it might make sense for us to use {{HiveInputFormat}} as well. We
should evaluate this on a query which has many input splits such as {{select
count(\*) from store_sales where something is not null}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-9153:
-
Attachment: HIVE-9153.3.patch
Seems the redundant code in {{Utilities.getBasework}} has been taken care of in
trunk. Revert that part for trunk patch.
Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]
-
Key: HIVE-9153
URL: https://issues.apache.org/jira/browse/HIVE-9153
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Brock Noland
Assignee: Rui Li
Attachments: HIVE-9153.1-spark.patch, HIVE-9153.1-spark.patch,
HIVE-9153.2.patch, HIVE-9153.3.patch, screenshot.PNG
The default InputFormat is {{CombineHiveInputFormat}} and thus HOS uses this.
However, Tez uses {{HiveInputFormat}}. Since tasks are relatively cheap in
Spark, it might make sense for us to use {{HiveInputFormat}} as well. We
should evaluate this on a query which has many input splits such as {{select
count(\*) from store_sales where something is not null}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-9153:
-
Attachment: screenshot.PNG
Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]
-
Key: HIVE-9153
URL: https://issues.apache.org/jira/browse/HIVE-9153
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Brock Noland
Assignee: Rui Li
Attachments: screenshot.PNG
The default InputFormat is {{CombineHiveInputFormat}} and thus HOS uses this.
However, Tez uses {{HiveInputFormat}}. Since tasks are relatively cheap in
Spark, it might make sense for us to use {{HiveInputFormat}} as well. We
should evaluate this on a query which has many input splits such as {{select
count(\*) from store_sales where something is not null}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-9153:
---
Description: The default InputFormat is {{CombineHiveInputFormat}} and thus
HOS uses this. However, Tez uses {{HiveInputFormat}}. Since tasks are
relatively cheap in Spark, it might make sense for us to use
{{HiveInputFormat}} as well. We should evaluate this on a query which has many
input splits such as {{select count(\*) from store_sales where something is not
null}}. (was: The default InputFormat is {{CombineHiveInputFormat}} and thus
HOS uses this. However, Tez uses {{HiveInputFormat}}. Since tasks are
relatively cheap in Spark, it might make sense for us to use
{{HiveInputFormat}} as well. We should evaluate this on a query which has many
input splits such as {{select count(*) from store_sales where something is not
null}}.)
Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]
-
Key: HIVE-9153
URL: https://issues.apache.org/jira/browse/HIVE-9153
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Brock Noland
Assignee: Rui Li
The default InputFormat is {{CombineHiveInputFormat}} and thus HOS uses this.
However, Tez uses {{HiveInputFormat}}. Since tasks are relatively cheap in
Spark, it might make sense for us to use {{HiveInputFormat}} as well. We
should evaluate this on a query which has many input splits such as {{select
count(\*) from store_sales where something is not null}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
