[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172752#comment-16172752
]
Alexander Behm commented on HIVE-15653:
---
[~ctang.ma], am I understanding correctly that there is no interest in fixing
this issue on the Metastore side? I understand that all clients can pass
DO_NOT_UPDATE_STATS.
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Statistics
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Fix For: 2.3.0
>
> Attachments: HIVE-15653.1.patch, HIVE-15653.2.patch,
> HIVE-15653.3.patch, HIVE-15653.4.patch, HIVE-15653.5.patch,
> HIVE-15653.6.patch, HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15844101#comment-15844101
]
Chaoyu Tang commented on HIVE-15653:
The test failures are not related to this patch. Three are aged failed tests,
one (vector_varchar_simple.q) is a flaky test (see HIVE-15745). [~pxiong],
would you like to take a look again? Thanks
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-15653.1.patch, HIVE-15653.2.patch,
> HIVE-15653.3.patch, HIVE-15653.4.patch, HIVE-15653.5.patch, HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15844000#comment-15844000
]
Hive QA commented on HIVE-15653:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12849781/HIVE-15653.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 11004 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_varchar_simple]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=93)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3236/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3236/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3236/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12849781 - PreCommit-HIVE-Build
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-15653.1.patch, HIVE-15653.2.patch,
> HIVE-15653.3.patch, HIVE-15653.4.patch, HIVE-15653.5.patch, HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843583#comment-15843583
]
Hive QA commented on HIVE-15653:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12849637/HIVE-15653.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11004 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_coltype_literals]
(batchId=12)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=93)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3224/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3224/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3224/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12849637 - PreCommit-HIVE-Build
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-15653.1.patch, HIVE-15653.2.patch,
> HIVE-15653.3.patch, HIVE-15653.4.patch, HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840902#comment-15840902
]
Pengcheng Xiong commented on HIVE-15653:
LGTM, pending tests run. Thanks for your patch!
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-15653.1.patch, HIVE-15653.2.patch,
> HIVE-15653.3.patch, HIVE-15653.4.patch, HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838957#comment-15838957
]
Chaoyu Tang commented on HIVE-15653:
[~pxiong] It is a good catch. Yes, the test has renamed the column "ALTER TABLE
calendar CHANGE year year1 INT." I will take a look at that.
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-15653.1.patch, HIVE-15653.2.patch,
> HIVE-15653.3.patch, HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838620#comment-15838620
]
Hive QA commented on HIVE-15653:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12849352/HIVE-15653.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11000 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_char_simple]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=93)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3181/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3181/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3181/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12849352 - PreCommit-HIVE-Build
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-15653.1.patch, HIVE-15653.2.patch,
> HIVE-15653.3.patch, HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838570#comment-15838570
]
Pengcheng Xiong commented on HIVE-15653:
i just go over view of all the changes. It sounds like only changes in
columnStatsUpdateForStatsOptimizer_2.q.out have problem.
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-15653.1.patch, HIVE-15653.2.patch,
> HIVE-15653.3.patch, HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838556#comment-15838556
]
Pengcheng Xiong commented on HIVE-15653:
[~ctang.ma], could u double check if the updated output files are legitimate?
For example, columnStatsUpdateForStatsOptimizer_2.q, L175, we only have a
column with name year1 after column renaming, but now
\"COLUMN_STATS\":{\"year\":\"true\"}.
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-15653.1.patch, HIVE-15653.2.patch,
> HIVE-15653.3.patch, HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837412#comment-15837412
]
Hive QA commented on HIVE-15653:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12849211/HIVE-15653.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 10983 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=120)
[groupby4_noskew.q,groupby3_map_skew.q,join_cond_pushdown_2.q,union19.q,union24.q,union_remove_5.q,groupby7_noskew_multi_single_reducer.q,vectorization_1.q,index_auto_self_join.q,auto_smb_mapjoin_14.q,script_env_var2.q,pcr.q,auto_join_filters.q,join0.q,join37.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_file_format]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_skewed_table]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_not_sorted]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnStatsUpdateForStatsOptimizer_2]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_alter_list_bucketing_table1]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_like] (batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[describe_comment_nonascii]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_tblproperties]
(batchId=76)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_invalidation]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[unset_table_view_property]
(batchId=47)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[offset_limit_ppd_optimizer]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_predicate_pushdown]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_table]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_vec_table]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_nonvec_table]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_table]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_varchar_simple]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[unset_table_property]
(batchId=86)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema[0]
(batchId=173)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteDate2 (batchId=173)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3167/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3167/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3167/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12849211 - PreCommit-HIVE-Build
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-15653.1.patch, HIVE-15653.2.patch, HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837089#comment-15837089
]
Pengcheng Xiong commented on HIVE-15653:
LGTM +1 pending tests. Thanks [~ctang.ma] for the patch.
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-15653.1.patch, HIVE-15653.2.patch, HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832679#comment-15832679
]
Hive QA commented on HIVE-15653:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12848395/HIVE-15653.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 10966 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_file_format]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_skewed_table]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_not_sorted]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnStatsUpdateForStatsOptimizer_2]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_alter_list_bucketing_table1]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_like] (batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[describe_comment_nonascii]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_tblproperties]
(batchId=76)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_invalidation]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[unset_table_view_property]
(batchId=47)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver[cascade_dbdrop]
(batchId=226)
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver[generatehfiles_require_family_path]
(batchId=226)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=135)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=136)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[escape1]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[escape2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_predicate_pushdown]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_table]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_vec_table]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_nonvec_table]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_table]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_varchar_simple]
(batchId=152)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[unset_table_property]
(batchId=86)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3079/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3079/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3079/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12848395 - PreCommit-HIVE-Build
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832119#comment-15832119
]
Greg Rahn commented on HIVE-15653:
--
I would add to the comments from [~alex.behm], stating it even more strongly --
statistics should never be explicitly removed unless it absolutely is
justified, as they are very expensive to compute. For example in RDBMS
systems, TRUNCATE, although it removes all rows, never removes statistics. I
would strongly recommend such behavior.
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830771#comment-15830771
]
Alexander Behm commented on HIVE-15653:
---
Good to know thanks. Regarding ALTER TABLE SET LOCATION: I suppose it's
arguable. Imo, we should not add side-effects to ALTER TABLE especially when it
comes to stats because they are expensive to compute. This is more of a product
question, so maybe [~grahn] or [~skumar] can weigh in.
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830768#comment-15830768
]
Chaoyu Tang commented on HIVE-15653:
[~alex.behm] I think the "ALTER TABLE SET LOCATION" should change table stats
as expected since the underlying table files could change. For others (except
SET CACHED which I need verify), this patch has fixed the issue and they won't
change table stats. Yes, the ALTER TABLE RENAME worked before.
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830719#comment-15830719
]
Alexander Behm commented on HIVE-15653:
---
[~ctang.ma] Impala calls the Metastore API alter_table(). I tried the following
alterations and those did wipe the table stats:
ALTER TABLE ADD COLUMNS
ALTER TABLE CHANGE COLUMN
ALTER TABLE SET TBLPROPERTIES
ALTER TABLE SET SERDEPROPERTIES
ALTER TABLE SET LOCATION
ALTER TABLE SET FILEFORMAT
ALTER TABLE SET CACHED
So I would say most ALTER commands do wipe the stats. Just trying to make sure
the fix on the Hive side is complete (i.e. the alter_table() API call on the
Metastore).
The ALTER TABLE RENAME command worked fine (preserved table stats).
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-15653.patch
>
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827334#comment-15827334
]
Alexander Behm commented on HIVE-15653:
---
Note that this problem seems to be specific to unpartitioned tables.
Partitioned tables work ok as far as I can tell.
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Priority: Critical
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
