Re: alter table ... partition() concatenate for ORC

[Nitin Pawar]

it does not work
alter table on orc also sometimes ends in problem

On Fri, Jul 31, 2015 at 11:04 AM, Du Lam  wrote:

> Hi all
>
> not sure if this is discussed before, but does  the merging for ORC works:
>
> alter table ... partition() concatenate for ORC
>
> , tried several times never been successful and some google search show
> some failure encountered by others as well. is this best way to merge small
> ORC files??
>
> many thanks!
>



-- 
Nitin Pawar

alter table ... partition() concatenate for ORC

[Du Lam]

Hi all

not sure if this is discussed before, but does  the merging for ORC works:

alter table ... partition() concatenate for ORC

, tried several times never been successful and some google search show
some failure encountered by others as well. is this best way to merge small
ORC files??

many thanks!

SLive Tests

[dlmarion]

  I have been running some SLive tests over the past couple of days to test
NameNode performance on different filesystems. Our setup includes:

 

. CentOS 6.6

. CDH 5.3.0 with Java 7

. 2 NameNodes in an HA configuration

. Both NameNodes are on the same type of hardware

. Journal Nodes are running on the NameNode servers, and one Journal
Node running on another node (3 total)

. NameNodes and JournalNodes are writing to different devices on the
NameNode machines

 

For the tests I used SLive to create a large number of files to create a
rather large NameNode data structure. Then, I ran some tests varying the #
of mappers and # of total operations, but always limiting the test to 30
minutes. The tests ran with a 70% read ratio and a mix of the other
operations to get to 100%. When changing the filesystems, we saved off the
fsimage and other information, reformatted the name, edits, checkpoints, and
journal directories, mounted them and put the fsimage and other information
back in their original location and restarted the NameNodes. We made sure to
change nothing else during the tests.

 

We tested ext3, ext4, and xfs filesystems. When we changed the filesystem
type, we changed it to the same type on all 3 machines (2NN + 1JN). Using
the total # of operations completed during the 30 minute test we found that
ext4 seemed to be the best choice. Using ext3 we completed 1% less
operations on average and with xfs it was about 30% less. I was a little
shocked by this, so we ran the xfs tests three times, attempting to tune the
XFS filesystem and mount options with no success. Thinking about this a
little, XFS has superior performance for parallel writes due to multiple
inode tables. So, some questions: 

 

1.   Is XFS known to be slower for single-threaded write performance
with multiple inode tables?

2.   are the writes to the edits and journals multi-threaded? I know
that they are sync'd, but is it a single writer?

3.   Is using the total # of ops the correct way to use SLive?

 

Next, I wanted to test the penalty for HA NameNode. So, we took the best
performing test (ext4) and changed the setup to non-HA, with a single NN on
one of the same NN machines. Running the same tests, the single NN exhibited
much less (like 10x less) total operations in the same amount of time. This
does not seem correct. What is different about the I/O paths for a single NN
(other than the writes to the JNs)? What did I do incorrectly?

 

Thanks,

Dave

Re: Sorting the inputSplits

[Harsh J]

If you meant 'scheduled' first perhaps thats doable by following (almost)
what Gera says. The framework actually explicitly sorts your InputSplits
list by its reported lengths, which would serve as the hack point for
inducing a reordering. See
https://github.com/apache/hadoop-common/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java#L498-L503

On Thu, Jul 30, 2015 at 10:34 PM Niels Basjes  wrote:

> MapReduce is based on the premise that several parts of a task can be
> processed independently in parallel.
> If you "require" an order of processing then these files are depending on
> each other. Why use MapReduce at all?
> With your requirement you cannot use more than one CPU anyway.
>
> Niels
>
> On Thu, 30 Jul 2015 01:31 Gera Shegalov  wrote:
>
>> Can you clarify the requirement "processed first"? Maps run in parallel
>> without any ordering guarantees. If you want to affect the mapping
>> file->split number, you can implement your own getSplits in the custom
>> input format and return splits ordered anyway your like.
>>
>> On Wed, Jul 22, 2015 at 12:06 PM, Nishanth S 
>> wrote:
>>
>>> Hey folks,
>>>
>>> Is their a way to sort the input splits  in map reduce.We have a case
>>> where there are two files file1 and file2 in the input directory.Since we
>>>  have custominputformat which   has issplittable return false always each
>>> of  these files would be processed  by  a different mapper.How could I make
>>> sure that  file1 is processed   before  file2(I want the oldest file to  be
>>> processed first).Is this possible?.
>>>
>>> Thanks,
>>> Nishan
>>>
>>
>>

Re: Sorting the inputSplits

[Niels Basjes]

MapReduce is based on the premise that several parts of a task can be
processed independently in parallel.
If you "require" an order of processing then these files are depending on
each other. Why use MapReduce at all?
With your requirement you cannot use more than one CPU anyway.

Niels

On Thu, 30 Jul 2015 01:31 Gera Shegalov  wrote:

> Can you clarify the requirement "processed first"? Maps run in parallel
> without any ordering guarantees. If you want to affect the mapping
> file->split number, you can implement your own getSplits in the custom
> input format and return splits ordered anyway your like.
>
> On Wed, Jul 22, 2015 at 12:06 PM, Nishanth S 
> wrote:
>
>> Hey folks,
>>
>> Is their a way to sort the input splits  in map reduce.We have a case
>> where there are two files file1 and file2 in the input directory.Since we
>>  have custominputformat which   has issplittable return false always each
>> of  these files would be processed  by  a different mapper.How could I make
>> sure that  file1 is processed   before  file2(I want the oldest file to  be
>> processed first).Is this possible?.
>>
>> Thanks,
>> Nishan
>>
>
>

Re: HDFS datanode used space is increasing without any writes

[Akmal Abbasov]

I am running HBase snapshot exporting, but I stopped it, and still the capacity 
used is increasing.

Here you can see that it is increased to 60 GB, and mostly it is because of 1 
detanode.
Moreover I am checking the directories sizes using bin/hdfs dfs -du -h /
and while the HDFS used capacity is increasing, the directories sizes in HDFS 
is not changing.
Any ideas?

p.s. I started a hdfs balancer several days ago, then stopped it after running 
for about 10 minutes, after reading that it is not a good idea to run it on the 
same cluster with HBase.
Could it be because of this?

Thank you.


> On 28 Jul 2015, at 19:08, Harshit Mathur  wrote:
> 
> Are there any map reduce jobs running?
> 
> On Jul 28, 2015 10:11 PM, "Akmal Abbasov"  > wrote:
> Hi, I’m observing strange behaviour in HDFS/HBase cluster.
> The disk space of one of datanodes is increasing very fast even when there 
> are no write requests. 
> It is 8GB per hour in average. Here is the graph which shows it.
> 
> I am using hbase-0.98.7-hadoop2 and hadoop-2.5.1.
> 
> And this is logs from the node
> 2015-07-28 15:40:38,795 INFO 
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: 
> /10.32.1.12:50010 , dest: /10.32.0.140:38699 
> , bytes: 1071, op: HDFS_READ, cliID: 
> DFSClient_NONMAPREDUCE_-689748537_1, offset: 0, srvID: 
> 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: 
> BP-439084760-10.32.0.180-1387281790961:blk_1074784244_1045663, duration: 
> 17759797
> 2015-07-28 15:41:15,111 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311714_1574450 src: 
> /10.0.0.21:60540  dest: /10.32.1.12:50010 
> 
> 2015-07-28 15:41:15,304 INFO 
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: 
> /10.0.0.21:59054 , dest: /10.32.1.12:50010 
> , bytes: 124121, op: HDFS_WRITE, cliID: 
> DFSClient_hb_rs_hbase-rs4,60020,1438094355024_530940245_35, offset: 0, srvID: 
> 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: 
> BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, duration: 
> 3600203675041
> 2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> PacketResponder: 
> BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, 
> type=HAS_DOWNSTREAM_IN_PIPELINE terminating
> 2015-07-28 15:50:40,745 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Slow BlockReceiver write data to disk cost:6099ms (threshold=300ms)
> 2015-07-28 15:59:21,130 INFO 
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool 
> BP-439084760-10.32.0.180-1387281790961 Total blocks: 65856, missing metadata 
> files:0, missing block files:0, missing blocks in memory:0, mismatched 
> blocks:0
> 2015-07-28 16:00:16,770 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311715_1574451 src: 
> /10.32.1.12:36998  dest: /10.32.1.12:50010 
> 
> 2015-07-28 16:00:17,469 INFO 
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: 
> /10.32.1.12:36150 , dest: /10.32.1.12:50010 
> , bytes: 32688, op: HDFS_WRITE, cliID: 
> DFSClient_hb_rs_hbase-rs5,60020,1438088401479_1146354759_35, offset: 0, 
> srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: 
> BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, duration: 
> 3601152263901
> 2015-07-28 16:00:17,472 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> PacketResponder: 
> BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, 
> type=HAS_DOWNSTREAM_IN_PIPELINE terminating
> 2015-07-28 16:03:44,011 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311716_1574452 src: 
> /10.0.0.19:35851  dest: /10.32.1.12:50010 
> 
> 2015-07-28 16:03:44,169 INFO 
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: 
> /10.0.0.20:40176 , dest: /10.32.1.12:50010 
> , bytes: 316062, op: HDFS_WRITE, cliID: 
> DFSClient_hb_rs_hbase-rs1,60020,1438092204868_-99326843_35, offset: 0, srvID: 
> 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: 
> BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, duration: 
> 3600482062810
> 2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> PacketResponder: 
> BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, 
> type=LAST_IN_PIPELINE, downstreams=0:[] terminating
> 2015-07-28 16:11:10,961 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Slow BlockReceiver write data to disk cost:7884ms (threshold=300ms)
> 2015-07-28 16:11:14,122 WARN org.apache.hadoop.hdfs.server.