Re: When querying ElasticSearch, score is 0

2015-04-18 Thread Andrejs Abele
/elasticsearch/reference/current/search-request-scroll.html#scroll-scan — Sent from Mailbox https://www.dropbox.com/mailbox On Thu, Apr 16, 2015 at 10:46 PM, Andrejs Abele andrejs.ab...@insight-centre.org mailto:andrejs.ab...@insight-centre.org wrote: Hi, I have data in my ElasticSearch

When querying ElasticSearch, score is 0

2015-04-16 Thread Andrejs Abele
Hi, I have data in my ElasticSearch server, when I query it using rest interface, I get results and score for each result, but when I run the same query in spark using ElasticSearch API, I get results and meta data, but the score is shown 0 for each record. My configuration is ... val conf = new

save as JSON objects

2014-11-04 Thread Andrejs Abele
Hi, Can some one pleas sugest me, what is the best way to output spark data as JSON file. (File where each line is a JSON object) Cheers, Andrejs

Re: how idf is calculated

2014-10-31 Thread Andrejs Abele
I found my problem. I assumed based on TF-IDF in Wikipedia , that log base 10 is used, but as I found in this discussion https://groups.google.com/forum/#!topic/scala-language/K5tbYSYqQc8, in scala it is actually ln (natural logarithm). Regards, Andrejs On Thu, Oct 30, 2014 at 10:49 PM, Ashic

Getting vector values

2014-10-30 Thread Andrejs Abele
Hi, I'm new to Mllib and spark. I'm trying to use tf-idf and use those values for term ranking. I'm getting tf values in vector format, but how can get the values of vector? val sc = new SparkContext(conf) val documents: RDD[Seq[String]] =

how idf is calculated

2014-10-30 Thread Andrejs Abele
Hi, I'm writing a paper and I need to calculate tf-idf. Whit your help I managed to get results, I needed, but the problem is that I need to be able to explain how each number was gotten. So I tried to understand how idf was calculated and the numbers i get don't correspond to those I should get .