RE: Using Spark like a search engine

Hi, I am sure you can use spark for this but it seems like a problem that should be delegated to a text based indexing technology like elastic search or something based on lucene to serve the requests. Spark can be used to prepare the data that can be fed to the indexing service. Using spark

Re: Using Spark like a search engine

Yes, spark will be useful for following areas of your application: 1. Running same function on every CV in parallel and score 2. Improve scoring function by better access to classification and clustering algorithms, within and beyond mllib. These are first benefits you can start with and then thin

Re: Using Spark like a search engine

Hi, ankur! Thanks for your reply! CVs are a just bunch of IDs, each ID represents some object of some class (eg. class=JOB, object=SW Developer). We have already processed texts and extracted all facts. So we don't need to do any text processing in Spark, just to run scoring function on many many C

Re: Using Spark like a search engine

Сергей, A simple implementation would be to create a DataFrame of CVs by issuing a Spark SQL query against your Postgres database, persist it in memory, and then to map F over it at query time and return the top

Re: Using Spark like a search engine

Thanks I'll give it a try! С Уважением, Сергей Мелехин. 2015-05-26 12:56 GMT+10:00 Alex Chavez : > Сергей, > A simple implementation would be to create a DataFrame of CVs by issuing a > Spark SQL query against your Postgres database, persist it in memory, and > then to map F over it at query tim