How big are your 50 files? How long are the reducers taking?
- HC
On Jul 30, 2013, at 10:26 PM, Something Something mailinglist...@gmail.com
wrote:
Hello,
One of our pig scripts creates over 500 small part files. To save on
namespace, we need to cut down the # of files, so instead of
It is file size based, not file count based. For fewer files up the
max-file-blocks setting.
On Wed, Jul 31, 2013 at 12:21 PM, Something Something
mailinglist...@gmail.com wrote:
Thanks, John. But I don't see an option to specify the # of output files.
How does Crush decide how many files to
How big are your 50 files? How long are the reducers taking?
On Jul 30, 2013, at 10:26 PM, Something Something mailinglist...@gmail.com
wrote:
Hello,
One of our pig scripts creates over 500 small part files. To save on
namespace, we need to cut down the # of files, so instead of saving
