RE: Clear up Hive scratch directory
Hi Nitin, Certainly – it was pretty straightforward! The process was just: 1) Stop Hive (this is to ensure that there are no sessions open which may be expecting to read/write to the scratch directory at that time) 2) I think that by default the scratch directories are named tmp/hive-*username*. In my case it was tmp/hive-hive that contained a load of bloat. If you’re unsure of the full path you’ll be able to find it in the Hive configuration. 3) Do a recursive delete to clear the contents of the scratch directory that you want to empty using the hdfs command line 4) Re-start Hive Thanks, Martin. From: Nitin Pawar [mailto:nitinpawar...@gmail.com] Sent: 24 April 2015 09:44 To: user@hive.apache.org Subject: Re: Clear up Hive scratch directory Thanks Martin Can you also mention steps you did to reclaim the hdfs data from temporary data ? On Fri, Apr 24, 2015 at 12:21 PM, Martin Benson mailto:martin.ben...@jaywing.com>> wrote: Hi All, I just wanted to feedback that it does appear to be safe - I emptied the directory manually, without adverse consequences. Thanks, Martin. From: Martin Benson<mailto:martin.ben...@jaywing.com> Sent: 20/04/2015 18:06 To: user@hive.apache.org<mailto:user@hive.apache.org> Subject: Clear up Hive scratch directory Hi, One of my users tried to run an HUGE join, which failed due to a lack of space in HDFS. This has resulted in a large amount of data remaining in the Hive scratch directory which I need to clear down. I've tried setting hive.start.cleanup.scratchdir to true and restarting Hive, but it didn't tidy it up. So, I'm wondering if it is safe to just delete the content of the directory in HDFS (while Hive is stopped). Could anyone advise please? Many thanks, Martin. Registered in England and Wales at Players House, 300 Attercliffe Common, Sheffield, S9 2AG. Company number 05935923. This email and its attachments are confidential and are intended solely for the use of the addressed recipient. Any views or opinions expressed are those of the author and do not necessarily represent Jaywing. If you are not the intended recipient, you must not forward or show this to anyone or take any action based upon it. Please contact the sender if you received this in error. Registered in England and Wales at Players House, 300 Attercliffe Common, Sheffield, S9 2AG. Company number 05935923. This email and its attachments are confidential and are intended solely for the use of the addressed recipient. Any views or opinions expressed are those of the author and do not necessarily represent Jaywing. If you are not the intended recipient, you must not forward or show this to anyone or take any action based upon it. Please contact the sender if you received this in error. -- Nitin Pawar Registered in England and Wales at Players House, 300 Attercliffe Common, Sheffield, S9 2AG. Company number 05935923. This email and its attachments are confidential and are intended solely for the use of the addressed recipient. Any views or opinions expressed are those of the author and do not necessarily represent Jaywing. If you are not the intended recipient, you must not forward or show this to anyone or take any action based upon it. Please contact the sender if you received this in error.
Re: Clear up Hive scratch directory
Thanks Martin Can you also mention steps you did to reclaim the hdfs data from temporary data ? On Fri, Apr 24, 2015 at 12:21 PM, Martin Benson wrote: > Hi All, > > I just wanted to feedback that it does appear to be safe - I emptied the > directory manually, without adverse consequences. > > Thanks, > > Martin. > -- > From: Martin Benson > Sent: 20/04/2015 18:06 > To: user@hive.apache.org > Subject: Clear up Hive scratch directory > > Hi, > > One of my users tried to run an HUGE join, which failed due to a lack of > space in HDFS. This has resulted in a large amount of data remaining in the > Hive scratch directory which I need to clear down. I've tried setting > hive.start.cleanup.scratchdir to true and restarting Hive, but it didn't > tidy it up. So, I'm wondering if it is safe to just delete the content of > the directory in HDFS (while Hive is stopped). Could anyone advise please? > > Many thanks, > > Martin. > > > Registered in England and Wales at Players House, 300 Attercliffe > Common, Sheffield, S9 2AG. Company number 05935923. > > This email and its attachments are confidential and are intended solely > for the use of the addressed recipient. > Any views or opinions expressed are those of the author and do not > necessarily represent Jaywing. If you are not > the intended recipient, you must not forward or show this to anyone or > take any action based upon it. > Please contact the sender if you received this in error. > > > Registered in England and Wales at Players House, 300 Attercliffe > Common, Sheffield, S9 2AG. Company number 05935923. > > This email and its attachments are confidential and are intended solely > for the use of the addressed recipient. > Any views or opinions expressed are those of the author and do not > necessarily represent Jaywing. If you are not > the intended recipient, you must not forward or show this to anyone or > take any action based upon it. > Please contact the sender if you received this in error. > -- Nitin Pawar
RE: Clear up Hive scratch directory
Hi All, I just wanted to feedback that it does appear to be safe - I emptied the directory manually, without adverse consequences. Thanks, Martin. From: Martin Benson<mailto:martin.ben...@jaywing.com> Sent: 20/04/2015 18:06 To: user@hive.apache.org<mailto:user@hive.apache.org> Subject: Clear up Hive scratch directory Hi, One of my users tried to run an HUGE join, which failed due to a lack of space in HDFS. This has resulted in a large amount of data remaining in the Hive scratch directory which I need to clear down. I've tried setting hive.start.cleanup.scratchdir to true and restarting Hive, but it didn't tidy it up. So, I'm wondering if it is safe to just delete the content of the directory in HDFS (while Hive is stopped). Could anyone advise please? Many thanks, Martin. Registered in England and Wales at Players House, 300 Attercliffe Common, Sheffield, S9 2AG. Company number 05935923. This email and its attachments are confidential and are intended solely for the use of the addressed recipient. Any views or opinions expressed are those of the author and do not necessarily represent Jaywing. If you are not the intended recipient, you must not forward or show this to anyone or take any action based upon it. Please contact the sender if you received this in error. Registered in England and Wales at Players House, 300 Attercliffe Common, Sheffield, S9 2AG. Company number 05935923. This email and its attachments are confidential and are intended solely for the use of the addressed recipient. Any views or opinions expressed are those of the author and do not necessarily represent Jaywing. If you are not the intended recipient, you must not forward or show this to anyone or take any action based upon it. Please contact the sender if you received this in error.
Clear up Hive scratch directory
Hi, One of my users tried to run an HUGE join, which failed due to a lack of space in HDFS. This has resulted in a large amount of data remaining in the Hive scratch directory which I need to clear down. I've tried setting hive.start.cleanup.scratchdir to true and restarting Hive, but it didn't tidy it up. So, I'm wondering if it is safe to just delete the content of the directory in HDFS (while Hive is stopped). Could anyone advise please? Many thanks, Martin. Registered in England and Wales at Players House, 300 Attercliffe Common, Sheffield, S9 2AG. Company number 05935923. This email and its attachments are confidential and are intended solely for the use of the addressed recipient. Any views or opinions expressed are those of the author and do not necessarily represent Jaywing. If you are not the intended recipient, you must not forward or show this to anyone or take any action based upon it. Please contact the sender if you received this in error.
