Ok, I have github account and clone mahout in my local workdir.
I revised the code and run test: mvn test, however, there are 3 test failure: Failed tests: LocalSSVDPCASparseTest.runPCATest1:87->runSSVDSolver:222->Assert.assertTrue:52->Assert.assertTrue:41->Assert.fail:86 null LocalSSVDSolverDenseTest.testSSVDSolverPowerIterations1:59->runSSVDSolver:172->Assert.assertTrue:52->Assert.assertTrue:41->Assert.fail:86 null LocalSSVDSolverSparseSequentialTest.testSSVDSolverPowerIterations1:69->runSSVDSolver:177->Assert.assertTrue:52->Assert.assertTrue:41->Assert.fail:86 null Now, my question is, how can I run a specified test with maven? For "mvn test" is so slow, then if I can do like "mvn test LocalSSVDPCASparseTest", my efficiency will be improved. At 2015-04-29 01:25:34, "Dmitriy Lyubimov" <dlie...@gmail.com> wrote: >Just Dmitriy is fine. > >In order to create a pull request, please check out the process page >http://mahout.apache.org/developers/github.html. Note that it is written >for both committers and contributors, so you need to ignore the details for >committers. > >Basically, you just need a github account, clone (fork) apache/mahout in >your account, (optionally) create a patch branch, commit your modifications >there, and then use github UI to create a pull request against >apache/mahout. > >thanks. > >-d > >On Mon, Apr 27, 2015 at 8:39 PM, lastarsenal <lastarse...@163.com> wrote: > >> Hi, Dmitriy Lyubimov >> >> >> OK, I have submitted a JIRA issue at >> https://issues.apache.org/jira/browse/MAHOUT-1700 >> >> >> I'm a newbie for mahout, so, what should I do next for this issue? Thank >> you! >> >> At 2015-04-28 02:16:37, "Dmitriy Lyubimov" <dlie...@gmail.com> wrote: >> >Thank you for this analysis. I can't immediately confirm this since it's >> >been a while but this sounds credible. >> > >> >Do you mind to file a jira with all this information, and even perhaps do >> a >> >PR on github? >> > >> >thank you. >> > >> >On Mon, Apr 27, 2015 at 4:32 AM, lastarsenal <lastarse...@163.com> wrote: >> > >> >> Hi, All, >> >> >> >> >> >> Recently, I tried mahout's hadoop ssvd(mahout-0.9 or mahout-1.0) >> >> job. There's a java heap space out of memory problem in >> ABtDenseOutJob. I >> >> found the reason, the ABtDenseOutJob map code is as below: >> >> >> >> >> >> protected void map(Writable key, VectorWritable value, Context >> context) >> >> throws IOException, InterruptedException { >> >> >> >> >> >> Vector vec = value.get(); >> >> >> >> >> >> int vecSize = vec.size(); >> >> if (aCols == null) { >> >> aCols = new Vector[vecSize]; >> >> } else if (aCols.length < vecSize) { >> >> aCols = Arrays.copyOf(aCols, vecSize); >> >> } >> >> >> >> >> >> if (vec.isDense()) { >> >> for (int i = 0; i < vecSize; i++) { >> >> extendAColIfNeeded(i, aRowCount + 1); >> >> aCols[i].setQuick(aRowCount, vec.getQuick(i)); >> >> } >> >> } else if (vec.size() > 0) { >> >> for (Vector.Element vecEl : vec.nonZeroes()) { >> >> int i = vecEl.index(); >> >> extendAColIfNeeded(i, aRowCount + 1); >> >> aCols[i].setQuick(aRowCount, vecEl.get()); >> >> } >> >> } >> >> aRowCount++; >> >> } >> >> >> >> >> >> If the input is RandomAccessSparseVector, usually with big data, it's >> >> vec.size() is Integer.MAX_VALUE, which is 2^31, then aCols = new >> >> Vector[vecSize] will introduce the OutOfMemory problem. The settlement >> of >> >> course should be enlarge every tasktracker's maximum memory: >> >> <property> >> >> <name>mapred.child.java.opts</name> >> >> <value>-Xmx1024m</value> >> >> </property> >> >> However, if you are NOT hadoop administrator or ops, you have no >> >> permission to modify the config. So, I try to modify ABtDenseOutJob map >> >> code to support RandomAccessSparseVector situation, I use hashmap to >> >> represent aCols instead of the original Vector[] aCols array, the >> modified >> >> code is as below: >> >> >> >> >> >> private Map<Integer, Vector> aColsMap = new HashMap<Integer, Vector>(); >> >> protected void map(Writable key, VectorWritable value, Context >> context) >> >> throws IOException, InterruptedException { >> >> >> >> >> >> Vector vec = value.get(); >> >> if (vec.isDense()) { >> >> for (int i = 0; i < vecSize; i++) { >> >> //extendAColIfNeeded(i, aRowCount + 1); >> >> if (aColsMap.get(i) == null) { >> >> aColsMap.put(i, new RandomAccessSparseVector(Integer.MAX_VALUE, >> >> 100)); >> >> } >> >> aColsMap.get(i).setQuick(aRowCount, vec.getQuick(i)); >> >> //aCols[i].setQuick(aRowCount, vec.getQuick(i)); >> >> } >> >> } else if (vec.size() > 0) { >> >> for (Vector.Element vecEl : vec.nonZeroes()) { >> >> int i = vecEl.index(); >> >> //extendAColIfNeeded(i, aRowCount + 1); >> >> if (aColsMap.get(i) == null) { >> >> aColsMap.put(i, new RandomAccessSparseVector(Integer.MAX_VALUE, >> >> 100)); >> >> } >> >> aColsMap.get(i).setQuick(aRowCount, vecEl.get()); >> >> //aCols[i].setQuick(aRowCount, vecEl.get()); >> >> } >> >> } >> >> aRowCount++; >> >> } >> >> >> >> >> >> Then the OutofMemory problem is dismissed. >> >> >> >> >> >> Thank you! >> >> >> >> >>
