Need help for Custom Score On Array Fields
Hi All,
Currently I have an array type, and I need to calculate score base on num
matched terms filters.
For example:
Here is my mappings :
{
"tweet" : {
"properties" : {
"tags" : {"type" : "string", "index_name" : "tag"},
}
}
}
My data will be indexed like that :
{
"tweet" : {
"tags" : ["USA","VN","GM"]
}
}
So if I query with terms filter for : "USA", and "GM"
My score will be 2/3 (it's mean num matched / tags array). (actually the
score will be calculated with complex formal, but I just one to focus on
the problem)
Thanks in advance.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3d047dbe-20d4-43ac-92c1-d1e21322c13c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Not analyzed for all field for all Index
Hi all, currently my index will create automatically when a new doc indexed. so I couldn't use CreateMapping request to set mapping property. I would like to know whether i could set it by default for store only bc i really need to increase my indexing perfomance (expect 1M req/sec) P/S if there is any tips for improve indexing throughput, please share me, thanks :) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d424fe6a-06ea-4e74-af60-a219b62c98f6%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Using Pig/Spark on ElasticSearch (as External Storage)
Hi Costin Leau, Currently I just pull all data in one index (INDEX_NAME_DATE) In my benmark, I just do two function, count and count distinct field. P/S: Thanks for your fast response, I would really happy to see you at IRC (just give me the time). On Sunday, October 12, 2014 8:02:57 PM UTC+7, Costin Leau wrote: > > It depends on various factors. Do you put all the data under one index or > is it one index per day/month/hour? What type of script and performance > degradation do you see? If it's easier feel free to reach out on irc. I'll > be traveling this week but we'll be back the next one. > Cheers > On Oct 12, 2014 2:51 PM, "Sang Dang" > > wrote: > >> Hi All, >> Currently I am using ElasticSearch for a logging system. >> My first solution is that every log will put on ES and index will rolling >> by date. >> To do real time stats, I will use Aggregation. >> To do statistic I will use Spark (or Hive, Shark whatever) on ES data >> (thanks to ElasticSearch-Hadoop plugin >> All is fine, but when my data grows (currently 17M record/index/date), >> Spark (Hive also) becomes very slow. >> >> I did benmark with the same data on ES and Hadoop, and I saw that Spark >> (Hive) run on Hadoop is much faster. >> >> Is there something I have missed ? >> >> Currently I maintain two storage, one is ES for realtime stats, another >> is Hadoop for other statistics. Is it fine ? >> >> I really appreciate any idea or solution., >> >> Thanks. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com . >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/103fb68e-65e8-4b1c-9e75-b34d393b7210%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/103fb68e-65e8-4b1c-9e75-b34d393b7210%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/efc5ac1e-863c-49ee-b13a-211fb03a54c6%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Using Pig/Spark on ElasticSearch (as External Storage)
Hi All, Currently I am using ElasticSearch for a logging system. My first solution is that every log will put on ES and index will rolling by date. To do real time stats, I will use Aggregation. To do statistic I will use Spark (or Hive, Shark whatever) on ES data (thanks to ElasticSearch-Hadoop plugin All is fine, but when my data grows (currently 17M record/index/date), Spark (Hive also) becomes very slow. I did benmark with the same data on ES and Hadoop, and I saw that Spark (Hive) run on Hadoop is much faster. Is there something I have missed ? Currently I maintain two storage, one is ES for realtime stats, another is Hadoop for other statistics. Is it fine ? I really appreciate any idea or solution., Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/103fb68e-65e8-4b1c-9e75-b34d393b7210%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Need some advice to build a log central.
Hello All, I have selected #2 as my solution. I write data to ES, and use kibana+ to realtime monitor. For stats, I use Hive. Each project, I will create a index, for each type of log I will put in a ES Type, ex: ProjectX>>log_debug >>log_error >>Stats_API >>Stats_PageView >>Stats_XYZ I am wonder whether it's good ? Should I separate by time for each type of project ? Regards. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/35487688-4204-4f4d-aa2e-2a9b6a43aa82%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Need some advice to build a log central.
Hi Vineeth Mohan, My log central will contain 2 type of log, one is for log to debug/monitor, other is for stats. I have 2 ways to achieve it: #1 , I use only ES, it's ok to log for debug/monitor (using kibana). To do stats, I will build some extra api (base on filter/facet/agregration...) #2, I use ES as external data storage, and write data to ES use Apache Hive (https://github.com/elasticsearch/elasticsearch-hadoop#apache-hive) this approach will help me alot in doing stats, but I don't know whether it's good for logging other info ( to debug/monitor purpose). I really appreciate your help :) Best Regards. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20091177-e41d-45be-bd0a-c535d7c65871%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Need some advice to build a log central.
Hi All, I am going to build a log central using ElasticSearch. I need some advice from anyone who have built it already. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5af88543-5806-4021-83a5-41abc5b2bed6%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
