[
https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659802#action_12659802
]
Edward J. Yoon commented on HAMA-129:
-------------------------------------
3000 * 3000 matrices multiplication result:
[d8g053:/root/hama-trunk]# bin/hama examples mult AAA BBB 25
08/12/30 20:54:11 INFO hama.AbstractMatrix: Initializing the matrix storage.
08/12/30 20:54:15 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randwzruq
08/12/30 20:54:15 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
e arguments. Applications should implement Tool for the same.
08/12/30 20:54:15 WARN mapred.JobClient: Use genericOptions for the option -libj
ars
08/12/30 20:54:15 WARN mapred.JobClient: No job jar file set. User classes may
not be found. See JobConf(Class) or JobConf#setJar(String).
08/12/30 20:54:15 INFO mapred.TableInputFormatBase: split: 0->d8g054.nhncorp.com
:,000000000001678
08/12/30 20:54:15 INFO mapred.TableInputFormatBase: split: 1->d8g054.nhncorp.com
:000000000001678,
08/12/30 20:54:15 INFO mapred.JobClient: Running job: job_200812301027_0061
08/12/30 20:54:16 INFO mapred.JobClient: map 0% reduce 0%
08/12/30 20:56:03 INFO mapred.JobClient: map 50% reduce 0%
08/12/30 20:56:16 INFO mapred.JobClient: map 50% reduce 8%
08/12/30 20:56:17 INFO mapred.JobClient: map 50% reduce 16%
08/12/30 20:56:18 INFO mapred.JobClient: map 100% reduce 16%
08/12/30 20:56:28 INFO mapred.JobClient: map 100% reduce 58%
08/12/30 20:56:32 INFO mapred.JobClient: map 100% reduce 100%
08/12/30 20:56:41 INFO mapred.JobClient: Job complete: job_200812301027_0061
08/12/30 20:56:41 INFO mapred.JobClient: Counters: 15
08/12/30 20:56:41 INFO mapred.JobClient: File Systems
08/12/30 20:56:41 INFO mapred.JobClient: Local bytes read=606272934
08/12/30 20:56:41 INFO mapred.JobClient: Local bytes written=909388562
08/12/30 20:56:41 INFO mapred.JobClient: Job Counters
08/12/30 20:56:41 INFO mapred.JobClient: Launched reduce tasks=3
08/12/30 20:56:41 INFO mapred.JobClient: Rack-local map tasks=2
08/12/30 20:56:41 INFO mapred.JobClient: Launched map tasks=3
08/12/30 20:56:41 INFO mapred.JobClient: Data-local map tasks=1
08/12/30 20:56:41 INFO mapred.JobClient: Map-Reduce Framework
08/12/30 20:56:41 INFO mapred.JobClient: Reduce input groups=25
08/12/30 20:56:41 INFO mapred.JobClient: Combine output records=0
08/12/30 20:56:41 INFO mapred.JobClient: Map input records=3000
08/12/30 20:56:41 INFO mapred.JobClient: Reduce output records=0
08/12/30 20:56:41 INFO mapred.JobClient: Map output bytes=303069450
08/12/30 20:56:41 INFO mapred.JobClient: Map input bytes=0
08/12/30 20:56:41 INFO mapred.JobClient: Combine input records=0
08/12/30 20:56:41 INFO mapred.JobClient: Map output records=15000
08/12/30 20:56:41 INFO mapred.JobClient: Reduce input records=15000
08/12/30 20:56:41 INFO hama.AbstractMatrix: Initializing the matrix storage.
08/12/30 20:56:45 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randqkczr
08/12/30 20:56:45 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
e arguments. Applications should implement Tool for the same.
08/12/30 20:56:45 WARN mapred.JobClient: Use genericOptions for the option -libj
ars
08/12/30 20:56:45 WARN mapred.JobClient: No job jar file set. User classes may
not be found. See JobConf(Class) or JobConf#setJar(String).
08/12/30 20:56:45 INFO mapred.TableInputFormatBase: split: 0->d8g053.nhncorp.com
:,000000000001547
08/12/30 20:56:45 INFO mapred.TableInputFormatBase: split: 1->d8g053.nhncorp.com
:000000000001547,
08/12/30 20:56:45 INFO mapred.JobClient: Running job: job_200812301027_0062
08/12/30 20:56:46 INFO mapred.JobClient: map 0% reduce 0%
08/12/30 20:58:40 INFO mapred.JobClient: map 50% reduce 0%
08/12/30 20:58:41 INFO mapred.JobClient: map 100% reduce 0%
08/12/30 20:58:50 INFO mapred.JobClient: map 100% reduce 41%
08/12/30 20:58:51 INFO mapred.JobClient: map 100% reduce 92%
08/12/30 20:58:53 INFO mapred.JobClient: map 100% reduce 100%
08/12/30 20:59:03 INFO mapred.JobClient: Job complete: job_200812301027_0062
08/12/30 20:59:03 INFO mapred.JobClient: Counters: 15
08/12/30 20:59:03 INFO mapred.JobClient: File Systems
08/12/30 20:59:03 INFO mapred.JobClient: Local bytes read=606270874
08/12/30 20:59:03 INFO mapred.JobClient: Local bytes written=909388550
08/12/30 20:59:03 INFO mapred.JobClient: Job Counters
08/12/30 20:59:03 INFO mapred.JobClient: Launched reduce tasks=3
08/12/30 20:59:03 INFO mapred.JobClient: Rack-local map tasks=1
08/12/30 20:59:03 INFO mapred.JobClient: Launched map tasks=3
08/12/30 20:59:03 INFO mapred.JobClient: Data-local map tasks=2
08/12/30 20:59:03 INFO mapred.JobClient: Map-Reduce Framework
08/12/30 20:59:03 INFO mapred.JobClient: Reduce input groups=25
08/12/30 20:59:03 INFO mapred.JobClient: Combine output records=0
08/12/30 20:59:03 INFO mapred.JobClient: Map input records=3000
08/12/30 20:59:03 INFO mapred.JobClient: Reduce output records=0
08/12/30 20:59:03 INFO mapred.JobClient: Map output bytes=303069450
08/12/30 20:59:03 INFO mapred.JobClient: Map input bytes=0
08/12/30 20:59:03 INFO mapred.JobClient: Combine input records=0
08/12/30 20:59:03 INFO mapred.JobClient: Map output records=15000
08/12/30 20:59:03 INFO mapred.JobClient: Reduce input records=15000
08/12/30 20:59:03 INFO hama.AbstractMatrix: Initializing the matrix storage.
08/12/30 20:59:07 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randluapj
08/12/30 20:59:07 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
e arguments. Applications should implement Tool for the same.
08/12/30 20:59:07 WARN mapred.JobClient: Use genericOptions for the option -libj
ars
08/12/30 20:59:08 WARN mapred.JobClient: No job jar file set. User classes may
not be found. See JobConf(Class) or JobConf#setJar(String).
08/12/30 20:59:08 INFO mapred.TableInputFormatBase: split: 0->d8g053.nhncorp.com
:,
08/12/30 20:59:08 INFO mapred.JobClient: Running job: job_200812301027_0063
08/12/30 20:59:09 INFO mapred.JobClient: map 0% reduce 0%
08/12/30 21:03:37 INFO mapred.JobClient: map 100% reduce 0%
08/12/30 21:03:44 INFO mapred.JobClient: map 100% reduce 33%
08/12/30 21:03:47 INFO mapred.JobClient: map 100% reduce 67%
08/12/30 21:03:49 INFO mapred.JobClient: map 100% reduce 69%
08/12/30 21:03:54 INFO mapred.JobClient: map 100% reduce 70%
08/12/30 21:03:59 INFO mapred.JobClient: map 100% reduce 71%
08/12/30 21:04:03 INFO mapred.JobClient: map 100% reduce 72%
08/12/30 21:04:13 INFO mapred.JobClient: map 100% reduce 75%
08/12/30 21:04:17 INFO mapred.JobClient: map 100% reduce 76%
08/12/30 21:04:18 INFO mapred.JobClient: map 100% reduce 78%
08/12/30 21:04:28 INFO mapred.JobClient: map 100% reduce 79%
08/12/30 21:04:33 INFO mapred.JobClient: map 100% reduce 80%
08/12/30 21:04:38 INFO mapred.JobClient: map 100% reduce 81%
08/12/30 21:04:43 INFO mapred.JobClient: map 100% reduce 83%
08/12/30 21:04:46 INFO mapred.JobClient: map 100% reduce 84%
08/12/30 21:04:48 INFO mapred.JobClient: map 100% reduce 85%
08/12/30 21:04:49 INFO mapred.JobClient: map 100% reduce 86%
08/12/30 21:05:26 INFO mapred.JobClient: map 100% reduce 88%
08/12/30 21:05:56 INFO mapred.JobClient: map 100% reduce 89%
08/12/30 21:06:27 INFO mapred.JobClient: map 100% reduce 90%
08/12/30 21:06:57 INFO mapred.JobClient: map 100% reduce 92%
08/12/30 21:07:27 INFO mapred.JobClient: map 100% reduce 93%
08/12/30 21:07:52 INFO mapred.JobClient: map 100% reduce 95%
08/12/30 21:08:22 INFO mapred.JobClient: map 100% reduce 96%
08/12/30 21:08:47 INFO mapred.JobClient: map 100% reduce 97%
08/12/30 21:09:17 INFO mapred.JobClient: map 100% reduce 99%
08/12/30 21:09:44 INFO mapred.JobClient: map 100% reduce 100%
08/12/30 21:09:53 INFO mapred.JobClient: Job complete: job_200812301027_0063
08/12/30 21:09:53 INFO mapred.JobClient: Counters: 14
08/12/30 21:09:53 INFO mapred.JobClient: File Systems
08/12/30 21:09:53 INFO mapred.JobClient: Local bytes read=720028247
08/12/30 21:09:53 INFO mapred.JobClient: Local bytes written=1080011761
08/12/30 21:09:53 INFO mapred.JobClient: Job Counters
08/12/30 21:09:53 INFO mapred.JobClient: Launched reduce tasks=2
08/12/30 21:09:53 INFO mapred.JobClient: Launched map tasks=1
08/12/30 21:09:53 INFO mapred.JobClient: Data-local map tasks=1
08/12/30 21:09:53 INFO mapred.JobClient: Map-Reduce Framework
08/12/30 21:09:53 INFO mapred.JobClient: Reduce input groups=25
08/12/30 21:09:53 INFO mapred.JobClient: Combine output records=0
08/12/30 21:09:53 INFO mapred.JobClient: Map input records=25
08/12/30 21:09:53 INFO mapred.JobClient: Reduce output records=15000
08/12/30 21:09:53 INFO mapred.JobClient: Map output bytes=360003250
08/12/30 21:09:53 INFO mapred.JobClient: Map input bytes=0
08/12/30 21:09:53 INFO mapred.JobClient: Combine input records=0
08/12/30 21:09:53 INFO mapred.JobClient: Map output records=125
08/12/30 21:09:53 INFO mapred.JobClient: Reduce input records=125
08/12/30 21:10:43 INFO client.HBaseAdmin: Disabled DenseMatrix_randluapj
08/12/30 21:10:43 INFO client.HBaseAdmin: Deleted DenseMatrix_randluapj
[d8g053:/root/hama-trunk]#
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch,
> HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch,
> HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes
> "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
