Re: Re: delta file compact take no effect
My hive version is 1.1.0 r7raul1...@163.com From: Alan Gates Date: 2015-06-18 23:25 To: user Subject: Re: delta file compact take no effect Which version of Hive are you running? A number of deadlock issues were resolved in HIVE-10500 which was released in Hive 1.2. Based on your log it appears it recovered properly from the deadlocks and did manage to compact. Alan. r7raul1...@163.com June 17, 2015 at 18:09 It's work~~ But I see some ERROR and Deadlock . 2015-06-18 09:06:06,509 ERROR [test.oracle-22]: txn.CompactionTxnHandler (CompactionTxnHandler.java:findNextToCompact(194)) - Unable to select next element for compaction, ERROR: could not serialize access due to concurrent update 2015-06-18 09:06:06,509 ERROR [test.oracle-27]: txn.CompactionTxnHandler (CompactionTxnHandler.java:findNextToCompact(194)) - Unable to select next element for compaction, ERROR: could not serialize access due to concurrent update 2015-06-18 09:06:06,509 ERROR [test.oracle-28]: txn.CompactionTxnHandler (CompactionTxnHandler.java:findNextToCompact(194)) - Unable to select next element for compaction, ERROR: could not serialize access due to concurrent update 2015-06-18 09:06:06,509 WARN [test.oracle-22]: txn.TxnHandler (TxnHandler.java:checkRetryable(916)) - Deadlock detected in findNextToCompact, trying again. 2015-06-18 09:06:06,509 WARN [test.oracle-27]: txn.TxnHandler (TxnHandler.java:checkRetryable(916)) - Deadlock detected in findNextToCompact, trying again. 2015-06-18 09:06:06,509 WARN [test.oracle-28]: txn.TxnHandler (TxnHandler.java:checkRetryable(916)) - Deadlock detected in findNextToCompact, trying again. 2015-06-18 09:06:06,544 INFO [test.oracle-26]: compactor.Worker (Worker.java:run(140)) - Starting MAJOR compaction for default.u_data_txn 2015-06-18 09:06:06,874 INFO [test.oracle-26]: impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(123)) - Timeline service address: http://192.168.117.117:8188/ws/v1/timeline/ 2015-06-18 09:06:06,960 INFO [test.oracle-26]: client.RMProxy (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at localhost/127.0.0.1:8032 2015-06-18 09:06:07,175 INFO [test.oracle-26]: impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(123)) - Timeline service address: http://192.168.117.117:8188/ws/v1/timeline/ 2015-06-18 09:06:07,176 INFO [test.oracle-26]: client.RMProxy (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at localhost/127.0.0.1:8032 2015-06-18 09:06:07,298 WARN [test.oracle-26]: mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(150)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2015-06-18 09:06:07,777 INFO [test.oracle-26]: mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(401)) - number of splits:2 2015-06-18 09:06:07,876 INFO [test.oracle-26]: mapreduce.JobSubmitter (JobSubmitter.java:printTokens(484)) - Submitting tokens for job: job_1433398549746_0035 2015-06-18 09:06:08,021 INFO [test.oracle-26]: impl.YarnClientImpl (YarnClientImpl.java:submitApplication(236)) - Submitted application application_1433398549746_0035 2015-06-18 09:06:08,052 INFO [test.oracle-26]: mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8088/proxy/application_1433398549746_0035/ 2015-06-18 09:06:08,052 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1344)) - Running job: job_1433398549746_0035 2015-06-18 09:06:18,174 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1365)) - Job job_1433398549746_0035 running in uber mode : false 2015-06-18 09:06:18,176 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1372)) - map 0% reduce 0% 2015-06-18 09:06:23,232 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1372)) - map 50% reduce 0% 2015-06-18 09:06:28,262 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1372)) - map 100% reduce 0% 2015-06-18 09:06:28,273 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1383)) - Job job_1433398549746_0035 completed successfully 2015-06-18 09:06:28,327 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1390)) - Counters: 30 r7raul1...@163.com r7raul1...@163.com June 10, 2015 at 22:10 I use hive 1.1.0 on hadoop 2.5.0 After I do some update operation on table u_data_txn. My table create many delta file like: drwxr-xr-x - hdfs hive 0 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001 -rw-r--r-- 3 hdfs supergroup 346453 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001/bucket_0 -rw-r--r-- 3 hdfs supergroup 415924 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 22:58 /user/hive/warehouse/u_data_txn/delta_002_002 -rw-r--r-- 3 hdfs supergroup 807 2015-02-06 22:58
Re: delta file compact take no effect
Which version of Hive are you running? A number of deadlock issues were resolved in HIVE-10500 which was released in Hive 1.2. Based on your log it appears it recovered properly from the deadlocks and did manage to compact. Alan. r7raul1...@163.com mailto:r7raul1...@163.com June 17, 2015 at 18:09 It's work~~ But I see some ERROR and Deadlock . 2015-06-18 09:06:06,509 ERROR [test.oracle-22]: txn.CompactionTxnHandler (CompactionTxnHandler.java:findNextToCompact(194)) - Unable to select next element for compaction, ERROR: could not serialize access due to concurrent update 2015-06-18 09:06:06,509 ERROR [test.oracle-27]: txn.CompactionTxnHandler (CompactionTxnHandler.java:findNextToCompact(194)) - Unable to select next element for compaction, ERROR: could not serialize access due to concurrent update 2015-06-18 09:06:06,509 ERROR [test.oracle-28]: txn.CompactionTxnHandler (CompactionTxnHandler.java:findNextToCompact(194)) - Unable to select next element for compaction, ERROR: could not serialize access due to concurrent update 2015-06-18 09:06:06,509 WARN [test.oracle-22]: txn.TxnHandler (TxnHandler.java:checkRetryable(916)) - Deadlock detected in findNextToCompact, trying again. 2015-06-18 09:06:06,509 WARN [test.oracle-27]: txn.TxnHandler (TxnHandler.java:checkRetryable(916)) - Deadlock detected in findNextToCompact, trying again. 2015-06-18 09:06:06,509 WARN [test.oracle-28]: txn.TxnHandler (TxnHandler.java:checkRetryable(916)) - Deadlock detected in findNextToCompact, trying again. 2015-06-18 09:06:06,544 INFO [test.oracle-26]: compactor.Worker (Worker.java:run(140)) - Starting MAJOR compaction for default.u_data_txn 2015-06-18 09:06:06,874 INFO [test.oracle-26]: impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(123)) - Timeline service address: http://192.168.117.117:8188/ws/v1/timeline/ 2015-06-18 09:06:06,960 INFO [test.oracle-26]: client.RMProxy (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at localhost/127.0.0.1:8032 2015-06-18 09:06:07,175 INFO [test.oracle-26]: impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(123)) - Timeline service address: http://192.168.117.117:8188/ws/v1/timeline/ 2015-06-18 09:06:07,176 INFO [test.oracle-26]: client.RMProxy (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at localhost/127.0.0.1:8032 2015-06-18 09:06:07,298 WARN [test.oracle-26]: mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(150)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2015-06-18 09:06:07,777 INFO [test.oracle-26]: mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(401)) - number of splits:2 2015-06-18 09:06:07,876 INFO [test.oracle-26]: mapreduce.JobSubmitter (JobSubmitter.java:printTokens(484)) - Submitting tokens for job: job_1433398549746_0035 2015-06-18 09:06:08,021 INFO [test.oracle-26]: impl.YarnClientImpl (YarnClientImpl.java:submitApplication(236)) - Submitted application application_1433398549746_0035 2015-06-18 09:06:08,052 INFO [test.oracle-26]: mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8088/proxy/application_1433398549746_0035/ 2015-06-18 09:06:08,052 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1344)) - Running job: job_1433398549746_0035 2015-06-18 09:06:18,174 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1365)) - Job job_1433398549746_0035 running in uber mode : false 2015-06-18 09:06:18,176 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1372)) - map 0% reduce 0% 2015-06-18 09:06:23,232 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1372)) - map 50% reduce 0% 2015-06-18 09:06:28,262 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1372)) - map 100% reduce 0% 2015-06-18 09:06:28,273 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1383)) - Job job_1433398549746_0035 completed successfully 2015-06-18 09:06:28,327 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1390)) - Counters: 30 r7raul1...@163.com r7raul1...@163.com mailto:r7raul1...@163.com June 10, 2015 at 22:10 I use hive 1.1.0 on hadoop 2.5.0 After I do some update operation on table u_data_txn. My table create many delta file like: drwxr-xr-x - hdfs hive 0 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001 -rw-r--r-- 3 hdfs supergroup 346453 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001/bucket_0 -rw-r--r-- 3 hdfs supergroup 415924 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 22:58 /user/hive/warehouse/u_data_txn/delta_002_002 -rw-r--r-- 3 hdfs supergroup 807 2015-02-06 22:58
Re: delta file compact take no effect
See https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration Compaction is initiated by the thrift metastore server. You need to set the values labeled metastore in the above page in the hive-site.xml for your metastore server. Alan. r7raul1...@163.com mailto:r7raul1...@163.com June 16, 2015 at 23:33 My config is on my client. What is metastore config? r7raul1...@163.com
Re: Re: delta file compact take no effect
It's work~~ But I see some ERROR and Deadlock . 2015-06-18 09:06:06,509 ERROR [test.oracle-22]: txn.CompactionTxnHandler (CompactionTxnHandler.java:findNextToCompact(194)) - Unable to select next element for compaction, ERROR: could not serialize access due to concurrent update 2015-06-18 09:06:06,509 ERROR [test.oracle-27]: txn.CompactionTxnHandler (CompactionTxnHandler.java:findNextToCompact(194)) - Unable to select next element for compaction, ERROR: could not serialize access due to concurrent update 2015-06-18 09:06:06,509 ERROR [test.oracle-28]: txn.CompactionTxnHandler (CompactionTxnHandler.java:findNextToCompact(194)) - Unable to select next element for compaction, ERROR: could not serialize access due to concurrent update 2015-06-18 09:06:06,509 WARN [test.oracle-22]: txn.TxnHandler (TxnHandler.java:checkRetryable(916)) - Deadlock detected in findNextToCompact, trying again. 2015-06-18 09:06:06,509 WARN [test.oracle-27]: txn.TxnHandler (TxnHandler.java:checkRetryable(916)) - Deadlock detected in findNextToCompact, trying again. 2015-06-18 09:06:06,509 WARN [test.oracle-28]: txn.TxnHandler (TxnHandler.java:checkRetryable(916)) - Deadlock detected in findNextToCompact, trying again. 2015-06-18 09:06:06,544 INFO [test.oracle-26]: compactor.Worker (Worker.java:run(140)) - Starting MAJOR compaction for default.u_data_txn 2015-06-18 09:06:06,874 INFO [test.oracle-26]: impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(123)) - Timeline service address: http://192.168.117.117:8188/ws/v1/timeline/ 2015-06-18 09:06:06,960 INFO [test.oracle-26]: client.RMProxy (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at localhost/127.0.0.1:8032 2015-06-18 09:06:07,175 INFO [test.oracle-26]: impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(123)) - Timeline service address: http://192.168.117.117:8188/ws/v1/timeline/ 2015-06-18 09:06:07,176 INFO [test.oracle-26]: client.RMProxy (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at localhost/127.0.0.1:8032 2015-06-18 09:06:07,298 WARN [test.oracle-26]: mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(150)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2015-06-18 09:06:07,777 INFO [test.oracle-26]: mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(401)) - number of splits:2 2015-06-18 09:06:07,876 INFO [test.oracle-26]: mapreduce.JobSubmitter (JobSubmitter.java:printTokens(484)) - Submitting tokens for job: job_1433398549746_0035 2015-06-18 09:06:08,021 INFO [test.oracle-26]: impl.YarnClientImpl (YarnClientImpl.java:submitApplication(236)) - Submitted application application_1433398549746_0035 2015-06-18 09:06:08,052 INFO [test.oracle-26]: mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8088/proxy/application_1433398549746_0035/ 2015-06-18 09:06:08,052 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1344)) - Running job: job_1433398549746_0035 2015-06-18 09:06:18,174 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1365)) - Job job_1433398549746_0035 running in uber mode : false 2015-06-18 09:06:18,176 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1372)) - map 0% reduce 0% 2015-06-18 09:06:23,232 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1372)) - map 50% reduce 0% 2015-06-18 09:06:28,262 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1372)) - map 100% reduce 0% 2015-06-18 09:06:28,273 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1383)) - Job job_1433398549746_0035 completed successfully 2015-06-18 09:06:28,327 INFO [test.oracle-26]: mapreduce.Job (Job.java:monitorAndPrintJob(1390)) - Counters: 30 r7raul1...@163.com From: r7raul1...@163.com Date: 2015-06-18 08:37 To: user Subject: Re: Re: delta file compact take no effect Thank you! I will try r7raul1...@163.com From: Alan Gates Date: 2015-06-18 08:33 To: user Subject: Re: delta file compact take no effect See https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration Compaction is initiated by the thrift metastore server. You need to set the values labeled metastore in the above page in the hive-site.xml for your metastore server. Alan. r7raul1...@163.com June 16, 2015 at 23:33 My config is on my client. What is metastore config? r7raul1...@163.com
Re: Re: delta file compact take no effect
My config is on my client. What is metastore config? r7raul1...@163.com From: Alan Gates Date: 2015-06-17 13:42 To: user Subject: Re: delta file compact take no effect Is the config you give on your metastore or your client? The worker thread and initiator must be started on the metastore. Alan. r7raul1...@163.com June 16, 2015 at 22:38 Any help? r7raul1...@163.com
Re: Re: delta file compact take no effect
Any help? r7raul1...@163.com From: r7raul1...@163.com Date: 2015-06-11 15:53 To: user Subject: Re: Re: delta file compact take no effect SHOW COMPACTIONS; I can see some info Database Table Partition Type State Worker Start Time default u_data_txn NULL MAJOR initiated NULL 0 Time taken: 0.024 seconds, Fetched: 2 row(s) But after that I still see many delta file. r7raul1...@163.com From: Elliot West Date: 2015-06-11 15:25 To: user@hive.apache.org Subject: Re: delta file compact take no effect What do you see if you issue: SHOW COMPACTIONS; On Thursday, 11 June 2015, r7raul1...@163.com r7raul1...@163.com wrote: I use hive 1.1.0 on hadoop 2.5.0 After I do some update operation on table u_data_txn. My table create many delta file like: drwxr-xr-x - hdfs hive 0 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001 -rw-r--r-- 3 hdfs supergroup 346453 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001/bucket_0 -rw-r--r-- 3 hdfs supergroup 415924 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 22:58 /user/hive/warehouse/u_data_txn/delta_002_002 -rw-r--r-- 3 hdfs supergroup 807 2015-02-06 22:58 /user/hive/warehouse/u_data_txn/delta_002_002/bucket_0 -rw-r--r-- 3 hdfs supergroup 779 2015-02-06 22:58 /user/hive/warehouse/u_data_txn/delta_002_002/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 22:59 /user/hive/warehouse/u_data_txn/delta_003_003 -rw-r--r-- 3 hdfs supergroup 817 2015-02-06 22:59 /user/hive/warehouse/u_data_txn/delta_003_003/bucket_0 -rw-r--r-- 3 hdfs supergroup 767 2015-02-06 22:59 /user/hive/warehouse/u_data_txn/delta_003_003/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 23:01 /user/hive/warehouse/u_data_txn/delta_004_004 -rw-r--r-- 3 hdfs supergroup 817 2015-02-06 23:01 /user/hive/warehouse/u_data_txn/delta_004_004/bucket_0 -rw-r--r-- 3 hdfs supergroup 779 2015-02-06 23:01 /user/hive/warehouse/u_data_txn/delta_004_004/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 23:03 /user/hive/warehouse/u_data_txn/delta_005_005 -rw-r--r-- 3 hdfs supergroup 817 2015-02-06 23:03 /user/hive/warehouse/u_data_txn/delta_005_005/bucket_0 -rw-r--r-- 3 hdfs supergroup 779 2015-02-06 23:03 /user/hive/warehouse/u_data_txn/delta_005_005/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-10 21:34 /user/hive/warehouse/u_data_txn/delta_006_006 -rw-r--r-- 3 hdfs supergroup 821 2015-02-10 21:34 /user/hive/warehouse/u_data_txn/delta_006_006/bucket_0 drwxr-xr-x - hdfs hive 0 2015-02-10 21:35 /user/hive/warehouse/u_data_txn/delta_007_007 -rw-r--r-- 3 hdfs supergroup 821 2015-02-10 21:35 /user/hive/warehouse/u_data_txn/delta_007_007/bucket_0 drwxr-xr-x - hdfs hive 0 2015-03-24 01:16 /user/hive/warehouse/u_data_txn/delta_008_008 -rw-r--r-- 3 hdfs supergroup 1670 2015-03-24 01:16 /user/hive/warehouse/u_data_txn/delta_008_008/bucket_0 -rw-r--r-- 3 hdfs supergroup 1767 2015-03-24 01:16 /user/hive/warehouse/u_data_txn/delta_008_008/bucket_1 I try ALTER TABLE u_data_txn COMPACT 'MAJOR'; The delta still exist. Then I try ALTER TABLE u_data_txn COMPACT 'MINOR'; The delta still exist. How to merge delta file? My config is: property namehive.support.concurrency/name valuetrue/value /property property namehive.enforce.bucketing/name valuetrue/value /property property namehive.exe.dynamic.partition.mode/name valuenonstrict/value /property property namehive.txn.manager/name valueorg.apache.hadoop.hive.ql.lockmgr.DbTxnManager/value /property property namehive.compactor.initiator.on/name valuetrue/value /property property namehive.compactor.worker.threads/name value4/value /property r7raul1...@163.com
Re: delta file compact take no effect
Is the config you give on your metastore or your client? The worker thread and initiator must be started on the metastore. Alan. r7raul1...@163.com mailto:r7raul1...@163.com June 16, 2015 at 22:38 Any help? r7raul1...@163.com
Re: delta file compact take no effect
Delta files that are no longer needed are deleted asynchronously. For example, you may have some query using delta_002_002. A minor compaction, for example, can run concurrently and create delta_001_003 but it will leave delta_001_001, delta_002_002, delta_003_003 to be cleaned later. A query that starts after this, will use delta_001_003 and ignore delta_001_001, delta_002_002, delta_003_003, thus it has fewer files to read and merge. delta_001_001, delta_002_002, delta_003_003 will be deleted when the system determines that no query can be using them. Judging by the directory listing you sent no major or minor compactions have ran. From: r7raul1...@163.commailto:r7raul1...@163.com r7raul1...@163.commailto:r7raul1...@163.com Reply-To: user@hive.apache.orgmailto:user@hive.apache.org user@hive.apache.orgmailto:user@hive.apache.org Date: Thursday, June 11, 2015 at 12:53 AM To: user@hive.apache.orgmailto:user@hive.apache.org user@hive.apache.orgmailto:user@hive.apache.org Subject: Re: Re: delta file compact take no effect SHOW COMPACTIONS; I can see some info Database Table Partition Type State Worker Start Time default u_data_txn NULL MAJOR initiated NULL 0 Time taken: 0.024 seconds, Fetched: 2 row(s) But after that I still see many delta file. r7raul1...@163.commailto:r7raul1...@163.com From: Elliot Westmailto:tea...@gmail.com Date: 2015-06-11 15:25 To: user@hive.apache.orgmailto:user@hive.apache.org Subject: Re: delta file compact take no effect What do you see if you issue: SHOW COMPACTIONS; On Thursday, 11 June 2015, r7raul1...@163.commailto:r7raul1...@163.com r7raul1...@163.commailto:r7raul1...@163.com wrote: I use hive 1.1.0 on hadoop 2.5.0 After I do some update operation on table u_data_txn. My table create many delta file like: drwxr-xr-x - hdfs hive 0 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001 -rw-r--r-- 3 hdfs supergroup 346453 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001/bucket_0 -rw-r--r-- 3 hdfs supergroup 415924 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 22:58 /user/hive/warehouse/u_data_txn/delta_002_002 -rw-r--r-- 3 hdfs supergroup 807 2015-02-06 22:58 /user/hive/warehouse/u_data_txn/delta_002_002/bucket_0 -rw-r--r-- 3 hdfs supergroup 779 2015-02-06 22:58 /user/hive/warehouse/u_data_txn/delta_002_002/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 22:59 /user/hive/warehouse/u_data_txn/delta_003_003 -rw-r--r-- 3 hdfs supergroup 817 2015-02-06 22:59 /user/hive/warehouse/u_data_txn/delta_003_003/bucket_0 -rw-r--r-- 3 hdfs supergroup 767 2015-02-06 22:59 /user/hive/warehouse/u_data_txn/delta_003_003/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 23:01 /user/hive/warehouse/u_data_txn/delta_004_004 -rw-r--r-- 3 hdfs supergroup 817 2015-02-06 23:01 /user/hive/warehouse/u_data_txn/delta_004_004/bucket_0 -rw-r--r-- 3 hdfs supergroup 779 2015-02-06 23:01 /user/hive/warehouse/u_data_txn/delta_004_004/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 23:03 /user/hive/warehouse/u_data_txn/delta_005_005 -rw-r--r-- 3 hdfs supergroup 817 2015-02-06 23:03 /user/hive/warehouse/u_data_txn/delta_005_005/bucket_0 -rw-r--r-- 3 hdfs supergroup 779 2015-02-06 23:03 /user/hive/warehouse/u_data_txn/delta_005_005/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-10 21:34 /user/hive/warehouse/u_data_txn/delta_006_006 -rw-r--r-- 3 hdfs supergroup 821 2015-02-10 21:34 /user/hive/warehouse/u_data_txn/delta_006_006/bucket_0 drwxr-xr-x - hdfs hive 0 2015-02-10 21:35 /user/hive/warehouse/u_data_txn/delta_007_007 -rw-r--r-- 3 hdfs supergroup 821 2015-02-10 21:35 /user/hive/warehouse/u_data_txn/delta_007_007/bucket_0 drwxr-xr-x - hdfs hive 0 2015-03-24 01:16 /user/hive/warehouse/u_data_txn/delta_008_008 -rw-r--r-- 3 hdfs supergroup 1670 2015-03-24 01:16 /user/hive/warehouse/u_data_txn/delta_008_008/bucket_0 -rw-r--r-- 3 hdfs supergroup 1767 2015-03-24 01:16 /user/hive/warehouse/u_data_txn/delta_008_008/bucket_1 I try ALTER TABLE u_data_txn COMPACT 'MAJOR'; The delta still exist. Then I try ALTER TABLE u_data_txn COMPACT 'MINOR'; The delta still exist. How to merge delta file? My config is: property namehive.support.concurrency/name valuetrue/value /property property namehive.enforce.bucketing/name valuetrue/value /property property namehive.exe.dynamic.partition.mode/name valuenonstrict/value /property property namehive.txn.manager/name valueorg.apache.hadoop.hive.ql.lockmgr.DbTxnManager/value /property property namehive.compactor.initiator.on/name valuetrue/value /property property namehive.compactor.worker.threads/name value4/value
Re: delta file compact take no effect
What do you see if you issue: SHOW COMPACTIONS; On Thursday, 11 June 2015, r7raul1...@163.com r7raul1...@163.com wrote: I use hive 1.1.0 on hadoop 2.5.0 After I do some update operation on table u_data_txn. My table create many delta file like: drwxr-xr-x - hdfs hive 0 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001 -rw-r--r-- 3 hdfs supergroup 346453 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001/bucket_0 -rw-r--r-- 3 hdfs supergroup 415924 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 22:58 /user/hive/warehouse/u_data_txn/delta_002_002 -rw-r--r-- 3 hdfs supergroup 807 2015-02-06 22:58 /user/hive/warehouse/u_data_txn/delta_002_002/bucket_0 -rw-r--r-- 3 hdfs supergroup 779 2015-02-06 22:58 /user/hive/warehouse/u_data_txn/delta_002_002/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 22:59 /user/hive/warehouse/u_data_txn/delta_003_003 -rw-r--r-- 3 hdfs supergroup 817 2015-02-06 22:59 /user/hive/warehouse/u_data_txn/delta_003_003/bucket_0 -rw-r--r-- 3 hdfs supergroup 767 2015-02-06 22:59 /user/hive/warehouse/u_data_txn/delta_003_003/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 23:01 /user/hive/warehouse/u_data_txn/delta_004_004 -rw-r--r-- 3 hdfs supergroup 817 2015-02-06 23:01 /user/hive/warehouse/u_data_txn/delta_004_004/bucket_0 -rw-r--r-- 3 hdfs supergroup 779 2015-02-06 23:01 /user/hive/warehouse/u_data_txn/delta_004_004/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 23:03 /user/hive/warehouse/u_data_txn/delta_005_005 -rw-r--r-- 3 hdfs supergroup 817 2015-02-06 23:03 /user/hive/warehouse/u_data_txn/delta_005_005/bucket_0 -rw-r--r-- 3 hdfs supergroup 779 2015-02-06 23:03 /user/hive/warehouse/u_data_txn/delta_005_005/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-10 21:34 /user/hive/warehouse/u_data_txn/delta_006_006 -rw-r--r-- 3 hdfs supergroup 821 2015-02-10 21:34 /user/hive/warehouse/u_data_txn/delta_006_006/bucket_0 drwxr-xr-x - hdfs hive 0 2015-02-10 21:35 /user/hive/warehouse/u_data_txn/delta_007_007 -rw-r--r-- 3 hdfs supergroup 821 2015-02-10 21:35 /user/hive/warehouse/u_data_txn/delta_007_007/bucket_0 drwxr-xr-x - hdfs hive 0 2015-03-24 01:16 /user/hive/warehouse/u_data_txn/delta_008_008 -rw-r--r-- 3 hdfs supergroup 1670 2015-03-24 01:16 /user/hive/warehouse/u_data_txn/delta_008_008/bucket_0 -rw-r--r-- 3 hdfs supergroup 1767 2015-03-24 01:16 /user/hive/warehouse/u_data_txn/delta_008_008/bucket_1 *I try ALTER TABLE u_data_txn COMPACT 'MAJOR'; The delta still exist. Then I try ALTER TABLE u_data_txn COMPACT 'MINOR'; The delta still exist. How to merge delta file?* *My config is:* property namehive.support.concurrency/name valuetrue/value /property property namehive.enforce.bucketing/name valuetrue/value /property property namehive.exe.dynamic.partition.mode/name valuenonstrict/value /property property namehive.txn.manager/name valueorg.apache.hadoop.hive.ql.lockmgr.DbTxnManager/value /property property namehive.compactor.initiator.on/name valuetrue/value /property property namehive.compactor.worker.threads/name value4/value /property -- r7raul1...@163.com javascript:_e(%7B%7D,'cvml','r7raul1...@163.com');
Re: Re: delta file compact take no effect
SHOW COMPACTIONS; I can see some info Database Table Partition Type State Worker Start Time default u_data_txn NULL MAJOR initiated NULL 0 Time taken: 0.024 seconds, Fetched: 2 row(s) But after that I still see many delta file. r7raul1...@163.com From: Elliot West Date: 2015-06-11 15:25 To: user@hive.apache.org Subject: Re: delta file compact take no effect What do you see if you issue: SHOW COMPACTIONS; On Thursday, 11 June 2015, r7raul1...@163.com r7raul1...@163.com wrote: I use hive 1.1.0 on hadoop 2.5.0 After I do some update operation on table u_data_txn. My table create many delta file like: drwxr-xr-x - hdfs hive 0 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001 -rw-r--r-- 3 hdfs supergroup 346453 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001/bucket_0 -rw-r--r-- 3 hdfs supergroup 415924 2015-02-06 22:52 /user/hive/warehouse/u_data_txn/delta_001_001/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 22:58 /user/hive/warehouse/u_data_txn/delta_002_002 -rw-r--r-- 3 hdfs supergroup 807 2015-02-06 22:58 /user/hive/warehouse/u_data_txn/delta_002_002/bucket_0 -rw-r--r-- 3 hdfs supergroup 779 2015-02-06 22:58 /user/hive/warehouse/u_data_txn/delta_002_002/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 22:59 /user/hive/warehouse/u_data_txn/delta_003_003 -rw-r--r-- 3 hdfs supergroup 817 2015-02-06 22:59 /user/hive/warehouse/u_data_txn/delta_003_003/bucket_0 -rw-r--r-- 3 hdfs supergroup 767 2015-02-06 22:59 /user/hive/warehouse/u_data_txn/delta_003_003/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 23:01 /user/hive/warehouse/u_data_txn/delta_004_004 -rw-r--r-- 3 hdfs supergroup 817 2015-02-06 23:01 /user/hive/warehouse/u_data_txn/delta_004_004/bucket_0 -rw-r--r-- 3 hdfs supergroup 779 2015-02-06 23:01 /user/hive/warehouse/u_data_txn/delta_004_004/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-06 23:03 /user/hive/warehouse/u_data_txn/delta_005_005 -rw-r--r-- 3 hdfs supergroup 817 2015-02-06 23:03 /user/hive/warehouse/u_data_txn/delta_005_005/bucket_0 -rw-r--r-- 3 hdfs supergroup 779 2015-02-06 23:03 /user/hive/warehouse/u_data_txn/delta_005_005/bucket_1 drwxr-xr-x - hdfs hive 0 2015-02-10 21:34 /user/hive/warehouse/u_data_txn/delta_006_006 -rw-r--r-- 3 hdfs supergroup 821 2015-02-10 21:34 /user/hive/warehouse/u_data_txn/delta_006_006/bucket_0 drwxr-xr-x - hdfs hive 0 2015-02-10 21:35 /user/hive/warehouse/u_data_txn/delta_007_007 -rw-r--r-- 3 hdfs supergroup 821 2015-02-10 21:35 /user/hive/warehouse/u_data_txn/delta_007_007/bucket_0 drwxr-xr-x - hdfs hive 0 2015-03-24 01:16 /user/hive/warehouse/u_data_txn/delta_008_008 -rw-r--r-- 3 hdfs supergroup 1670 2015-03-24 01:16 /user/hive/warehouse/u_data_txn/delta_008_008/bucket_0 -rw-r--r-- 3 hdfs supergroup 1767 2015-03-24 01:16 /user/hive/warehouse/u_data_txn/delta_008_008/bucket_1 I try ALTER TABLE u_data_txn COMPACT 'MAJOR'; The delta still exist. Then I try ALTER TABLE u_data_txn COMPACT 'MINOR'; The delta still exist. How to merge delta file? My config is: property namehive.support.concurrency/name valuetrue/value /property property namehive.enforce.bucketing/name valuetrue/value /property property namehive.exe.dynamic.partition.mode/name valuenonstrict/value /property property namehive.txn.manager/name valueorg.apache.hadoop.hive.ql.lockmgr.DbTxnManager/value /property property namehive.compactor.initiator.on/name valuetrue/value /property property namehive.compactor.worker.threads/name value4/value /property r7raul1...@163.com
