I’m curious to see these benchmarks run on the latest Solr version, as the
numbers you quoted were over two years ago. Also, it’d be useful to see the
indexing and searching benchmark code to make sure it takes advantage of best
practices. I’ve indexed 10M docs into Solr in only a few minutes. 500K, say
in CSV format, for basic e-commerce product data, would likely take a minute or
so. The searching differences you present seem fairly negligible - 100ms is
the blink of an eye, so anything under is considered quite acceptable by the
largest e-commerce vendors in the world. Along with that, perhaps an even more
important benchmark is relevancy or in some way measure how good the search
results are.
As Mike put so well, competition is a great thing so by all means I encourage
you to carry on with your endeavor. Sounds like you’ve built some powerful
stuff and have extensive experience. +1
Erik
On Nov 15, 2014, at 6:23 PM, swsong_dev <[email protected]> wrote:
> Thank you for your sincere reply, Mr. McCandless.
>
> When I posted an email in a mailing-list, I was afraid for not getting a
> considerable reply, but I’m now so glad I might find a way.
>
> I agree that a new search engine in Go would be competitive. I think we all
> need a next generation search engine core redesigned from the start.
>
> And, I understand Lucene’s limitations you mentioned. They are good points to
> get started.
>
> I have been used a search engine for first 4 years and developed a search
> engines for last 6 years from the bottom, and I got feedback “It’s faster
> than Solr in indexing and searching”. (http://ddakker.tistory.com/248
> <http://ddakker.tistory.com/248>)
>
> ===Result===
> Data size : 529,188
> Fastcat indexing time : 1m 26s
> Solr3.5 indexing time : 5m 30s
> Fastcat searching time : 48ms
> Solr3.5 searching time : 73ms
>
> It applied to Korea’s greatest shopping service(http://danawa.com/
> <http://danawa.com/>) a month ago to my delight.
>
> But my goal has been making a globally-used open source search engine.
>
> As you suggested, now I want to make a whole-new search engine in Go.
>
> I have made my first search engine alone, but I would not make a new search
> engine alone. I want to make it with global developers together.
>
> If you plan to make a new search engine in Go, or know someone around you,
> could you help me gathering members for a new search engine, and guide us
> technically(feature requirement, efficient design)?
>
> Or if there is already a new search engine project in Go, could you let me
> know?
>
> In Korea, no one develops a search engine except people who work at a search
> engine solution company, and even they are very few and do not spend time to
> an open source project.
>
> In my case, I found a tiny venture company for making time to develop an open
> source search engine 4 year ago.
>
> I want to be involved in a next-generation search engine project. I would be
> happy to make a new search engine itself.
>
> Your little help could be great for me.
>
> Thank you.
>
> Sang Song
>
>
>> 2014. 11. 15., 오후 8:22, Michael McCandless <[email protected]> 작성:
>>
>> Actually I think competing projects is very healthy for open source
>> development.
>>
>> There are many things you could explore to "contrast" with Lucene,
>> e.g. write your new search engine in Go not Java: Java has many
>> problems, maybe Go fixes them. Go also has a low-latency garbage
>> collector in development ... and Java's GC options still can't scale
>> to the heap sizes that are practical now.
>>
>> Lucene has many limitations, so your competing engine could focus on
>> them. E.g. the "schemalessness" of Lucene has become a big problem,
>> and near impossible to fix at this point, and prevents new important
>> features like LUCENE-5879 from being possible, so you could give your
>> engine a "gentle" schema from the start.
>>
>> The Lucene Filter/Query situation is a mess: one should extend the other.
>>
>> Lucene has weak support for proximity queries (SpanQuery is slow and
>> does not get much attention).
>>
>> Lucene is showing its age, missing some compelling features like a
>> builtin transaction log, "core" support for numerics (they are sort of
>> hacked on top), optimistic concurrency support (sequence ids,
>> versions, something), distributed support (near real time replication,
>> etc.), multi-tenancy, an example server implementation, so the search
>> servers on top of Lucene have had to fill these gaps. Maybe you could
>> make your engine distributed from the start (Go is a great match for
>> that, from what little I know).
>>
>> All 3 highlighter options have problems.
>>
>> The analysis chain (attributes) is overly complex.
>>
>> In your competing engine you can borrow/copy/steal from Lucene's good
>> parts to get started...
>>
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[email protected]> wrote:
>>> I’m developing search engine, Fastcatsearch. http://github
>>> <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>>>
>>> Lucene is widely known and famous project and I cannot beat Lucene for now.
>>>
>>> But is there any chance to beat Lucene?
>>>
>>> Anything like features, performance.
>>>
>>> Please, let me know what to do to make better product than Lucene.
>>>
>>> Thank you.
>
