I am running it in 1.5.2. I will try running it in small standalone
cluster to see whether it's correct.
On Sat, Jan 9, 2016 at 6:21 AM, Bryan Cutler <cutl...@gmail.com> wrote:
> Hi Li,
>
> I tried out your code and sample data in both local mode and Spark
> Standalone and it ran correctly with output that looks good. Sorry, I don't
> have a YARN cluster setup right now, so maybe the error you are seeing is
> specific to that. Btw, I am running the latest Spark code from the master
> branch. Hope that helps some!
>
> Bryan
>
> On Mon, Jan 4, 2016 at 8:42 PM, Li Li <fancye...@gmail.com> wrote:
>>
>> anyone could help? the problem is very easy to reproduce. What's wrong?
>>
>> On Wed, Dec 30, 2015 at 8:59 PM, Li Li <fancye...@gmail.com> wrote:
>> > I use a small data and reproduce the problem.
>> > But I don't know my codes are correct or not because I am not familiar
>> > with spark.
>> > So I first post my codes here. If it's correct, then I will post the
>> > data.
>> > one line of my data like:
>> >
>> > { "time":"08-09-17","cmtUrl":"2094361"
>> >
>> > ,"rvId":"rev_10000020","webpageUrl":"http://www.dianping.com/shop/2094361","word_vec":[0,1,2,3,4,5,6,2,7,8,9
>> >
>> > ,10,11,12,13,14,15,16,8,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,32,35,36,37,38,15,39,40,41,42,5,43,44,17,45,46,42,47,26,48,49]}
>> >
>> > it's a json file which contains webpageUrl and word_vec which is the
>> > encoded words.
>> > The first step is to prase the input rdd to a rdd of VectorUrl.
>> > BTW, if public VectorUrl call(String s) return null, is it ok?
>> > Then follow the example Index documents with unique IDs
>> > Then I create a rdd to map id to url so after lda training, I can find
>> > the url of the document. Then save this rdd to hdfs.
>> > Then create corpus rdd and train
>> >
>> > The exception stack is
>> >
>> > 15/12/30 20:45:42 ERROR yarn.ApplicationMaster: User class threw
>> > exception: java.lang.IndexOutOfBoundsException: (454,0) not in
>> > [-58,58) x [-100,100)
>> > java.lang.IndexOutOfBoundsException: (454,0) not in [-58,58) x
>> > [-100,100)
>> > at breeze.linalg.DenseMatrix$mcD$sp.update$mcD$sp(DenseMatrix.scala:112)
>> > at
>> > org.apache.spark.mllib.clustering.DistributedLDAModel$$anonfun$topicsMatrix$1.apply(LDAModel.scala:534)
>> > at
>> > org.apache.spark.mllib.clustering.DistributedLDAModel$$anonfun$topicsMatrix$1.apply(LDAModel.scala:531)
>> > at
>> > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>> > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>> > at
>> > org.apache.spark.mllib.clustering.DistributedLDAModel.topicsMatrix$lzycompute(LDAModel.scala:531)
>> > at
>> > org.apache.spark.mllib.clustering.DistributedLDAModel.topicsMatrix(LDAModel.scala:523)
>> > at
>> > com.mobvoi.knowledgegraph.textmining.lda.ReviewLDA.main(ReviewLDA.java:89)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > at
>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:606)
>> > at
>> > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:525)
>> >
>> >
>> > ==========here is my codes==============
>> >
>> > SparkConf conf = new SparkConf().setAppName(ReviewLDA.class.getName());
>> >
>> > JavaSparkContext sc = new JavaSparkContext(conf);
>> >
>> >
>> > // Load and parse the data
>> >
>> > JavaRDD<String> data = sc.textFile(inputDir + "/*");
>> >
>> > JavaRDD<VectorUrl> parsedData = data.map(new Function<String,
>> > VectorUrl>() {
>> >
>> > public VectorUrl call(String s) {
>> >
>> > JsonParser parser = new JsonParser();
>> >
>> > JsonObject jo = parser.parse(s).getAsJsonObject();
>> >
>> > if (!jo.has("word_vec") || !jo.has("webpageUrl")) {
>> >
>> > return null;
>> >
>> > }
>> >
>> > JsonArray word_vec = jo.get("word_vec").getAsJsonArray();
>> >
>> > String url = jo.get("webpageUrl").getAsString();
>> >
>> > double[] values = new double[word_vec.size()];
>> >
>> > for (int i = 0; i < values.length; i++)
>> >
>> > values[i] = word_vec.get(i).getAsInt();
>> >
>> > return new VectorUrl(Vectors.dense(values), url);
>> >
>> > }
>> >
>> > });
>> >
>> >
>> >
>> > // Index documents with unique IDs
>> >
>> > JavaPairRDD<Long, VectorUrl> id2doc =
>> > JavaPairRDD.fromJavaRDD(parsedData.zipWithIndex().map(
>> >
>> > new Function<Tuple2<VectorUrl, Long>, Tuple2<Long, VectorUrl>>()
>> > {
>> >
>> > public Tuple2<Long, VectorUrl> call(Tuple2<VectorUrl, Long>
>> > doc_id) {
>> >
>> > return doc_id.swap();
>> >
>> > }
>> >
>> > }));
>> >
>> > JavaPairRDD<Long, String> id2Url = JavaPairRDD.fromJavaRDD(id2doc
>> >
>> > .map(new Function<Tuple2<Long, VectorUrl>, Tuple2<Long,
>> > String>>() {
>> >
>> > @Override
>> >
>> > public Tuple2<Long, String> call(Tuple2<Long, VectorUrl>
>> > id2doc) throws Exception {
>> >
>> > return new Tuple2(id2doc._1, id2doc._2.url);
>> >
>> > }
>> >
>> > }));
>> >
>> > id2Url.saveAsTextFile(id2UrlPath);
>> >
>> > JavaPairRDD<Long, Vector> corpus = JavaPairRDD.fromJavaRDD(id2doc
>> >
>> > .map(new Function<Tuple2<Long, VectorUrl>, Tuple2<Long,
>> > Vector>>() {
>> >
>> > @Override
>> >
>> > public Tuple2<Long, Vector> call(Tuple2<Long, VectorUrl>
>> > id2doc) throws Exception {
>> >
>> > return new Tuple2(id2doc._1, id2doc._2.vec);
>> >
>> > }
>> >
>> > }));
>> >
>> > corpus.cache();
>> >
>> >
>> > // Cluster the documents into three topics using LDA
>> >
>> > DistributedLDAModel ldaModel = (DistributedLDAModel) new
>> > LDA().setMaxIterations(iterNumber)
>> >
>> > .setK(topicNumber).run(corpus);
>> >
>> > On Wed, Dec 30, 2015 at 3:34 PM, Li Li <fancye...@gmail.com> wrote:
>> >> I will use a portion of data and try. will the hdfs block affect
>> >> spark?(if so, it's hard to reproduce)
>> >>
>> >> On Wed, Dec 30, 2015 at 3:22 AM, Joseph Bradley <jos...@databricks.com>
>> >> wrote:
>> >>> Hi Li,
>> >>>
>> >>> I'm wondering if you're running into the same bug reported here:
>> >>> https://issues.apache.org/jira/browse/SPARK-12488
>> >>>
>> >>> I haven't figured out yet what is causing it. Do you have a small
>> >>> corpus
>> >>> which reproduces this error, and which you can share on the JIRA? If
>> >>> so,
>> >>> that would help a lot in debugging this failure.
>> >>>
>> >>> Thanks!
>> >>> Joseph
>> >>>
>> >>> On Sun, Dec 27, 2015 at 7:26 PM, Li Li <fancye...@gmail.com> wrote:
>> >>>>
>> >>>> I ran my lda example in a yarn 2.6.2 cluster with spark 1.5.2.
>> >>>> it throws exception in line: Matrix topics =
>> >>>> ldaModel.topicsMatrix();
>> >>>> But in yarn job history ui, it's successful. What's wrong with it?
>> >>>> I submit job with
>> >>>> .bin/spark-submit --class Myclass \
>> >>>> --master yarn-client \
>> >>>> --num-executors 2 \
>> >>>> --driver-memory 4g \
>> >>>> --executor-memory 4g \
>> >>>> --executor-cores 1 \
>> >>>>
>> >>>>
>> >>>> My codes:
>> >>>>
>> >>>> corpus.cache();
>> >>>>
>> >>>>
>> >>>> // Cluster the documents into three topics using LDA
>> >>>>
>> >>>> DistributedLDAModel ldaModel = (DistributedLDAModel) new
>> >>>>
>> >>>>
>> >>>> LDA().setOptimizer("em").setMaxIterations(iterNumber).setK(topicNumber).run(corpus);
>> >>>>
>> >>>>
>> >>>> // Output topics. Each is a distribution over words (matching
>> >>>> word
>> >>>> count vectors)
>> >>>>
>> >>>> System.out.println("Learned topics (as distributions over vocab
>> >>>> of
>> >>>> " + ldaModel.vocabSize()
>> >>>>
>> >>>> + " words):");
>> >>>>
>> >>>> //Line81, exception here: Matrix topics =
>> >>>> ldaModel.topicsMatrix();
>> >>>>
>> >>>> for (int topic = 0; topic < topicNumber; topic++) {
>> >>>>
>> >>>> System.out.print("Topic " + topic + ":");
>> >>>>
>> >>>> for (int word = 0; word < ldaModel.vocabSize(); word++) {
>> >>>>
>> >>>> System.out.print(" " + topics.apply(word, topic));
>> >>>>
>> >>>> }
>> >>>>
>> >>>> System.out.println();
>> >>>>
>> >>>> }
>> >>>>
>> >>>>
>> >>>> ldaModel.save(sc.sc(), modelPath);
>> >>>>
>> >>>>
>> >>>> Exception in thread "main" java.lang.IndexOutOfBoundsException:
>> >>>> (1025,0) not in [-58,58) x [-100,100)
>> >>>>
>> >>>> at
>> >>>> breeze.linalg.DenseMatrix$mcD$sp.update$mcD$sp(DenseMatrix.scala:112)
>> >>>>
>> >>>> at
>> >>>>
>> >>>> org.apache.spark.mllib.clustering.DistributedLDAModel$$anonfun$topicsMatrix$1.apply(LDAModel.scala:534)
>> >>>>
>> >>>> at
>> >>>>
>> >>>> org.apache.spark.mllib.clustering.DistributedLDAModel$$anonfun$topicsMatrix$1.apply(LDAModel.scala:531)
>> >>>>
>> >>>> at
>> >>>>
>> >>>> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>> >>>>
>> >>>> at
>> >>>> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>> >>>>
>> >>>> at
>> >>>>
>> >>>> org.apache.spark.mllib.clustering.DistributedLDAModel.topicsMatrix$lzycompute(LDAModel.scala:531)
>> >>>>
>> >>>> at
>> >>>>
>> >>>> org.apache.spark.mllib.clustering.DistributedLDAModel.topicsMatrix(LDAModel.scala:523)
>> >>>>
>> >>>> at
>> >>>>
>> >>>> com.mobvoi.knowledgegraph.textmining.lda.ReviewLDA.main(ReviewLDA.java:81)
>> >>>>
>> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>>
>> >>>> at
>> >>>>
>> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> >>>>
>> >>>> at
>> >>>>
>> >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >>>>
>> >>>> at java.lang.reflect.Method.invoke(Method.java:606)
>> >>>>
>> >>>> at
>> >>>>
>> >>>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
>> >>>>
>> >>>> at
>> >>>>
>> >>>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
>> >>>>
>> >>>> at
>> >>>> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
>> >>>>
>> >>>> at
>> >>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>> >>>>
>> >>>> at
>> >>>> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>> >>>>
>> >>>> 15/12/23 00:01:16 INFO spark.SparkContext: Invoking stop() from
>> >>>> shutdown
>> >>>> hook
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> >>>> For additional commands, e-mail: dev-h...@spark.apache.org
>> >>>>
>> >>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
