I only know some characteristics of the openAI ada-002
vectors, although they are a very popular as
embeddings/text-characterisations as they allow more
accurate/"human meaningful" semantic search results with
fewer dimensions than their predecessors - I've evaluated a
few different embedding models, including some BERT
variants, CLIP ViT-L-14 (with 768 dims, which was quite
good), openAI's ada-001 (1024 dims) and babbage-001 (2048
dims), and ada-002 are qualitatively the best, although
that will certainly change!
In any case, ada-002 vectors have interesting
characteristics that I think mean you could confidently
create synthetic vectors which would be hard to distinguish
from "real" vectors. I found this from looking at 47K
ada-002 vectors generated across a full year (1994) of
newspaper articles from the Canberra Times and 200K
wikipedia articles:
- there is no discernible/significant correlation between
values in any pair of dimensions
- all but 5 of the 1536 dimensions have an almost identical
distribution of values shown in the central blob on these
graphs (that just show a few of these 1531 dimensions with
clumped values and the 5 "outlier" dimensions, but all 1531
non-outlier dims are in there, which makes for some easy
quantisation from float to byte if you dont want to go the
full kmeans/clustering/Lloyds-algorithm approach):
https://docs.google.com/spreadsheets/d/1DyyBCbirETZSUAEGcMK__mfbUNzsU_L48V9E0SyJYGg/edit?usp=sharing
https://docs.google.com/spreadsheets/d/1czEAlzYdyKa6xraRLesXjNZvEzlj27TcDGiEFS1-MPs/edit?usp=sharing
https://docs.google.com/spreadsheets/d/1RxTjV7Sj14etCNLk1GB-m44CXJVKdXaFlg2Y6yvj3z4/edit?usp=sharing
- the variance of the value of each dimension is
characteristic:
https://docs.google.com/spreadsheets/d/1w5LnRUXt1cRzI9Qwm07LZ6UfszjMOgPaJot9cOGLHok/edit#gid=472178228
This probably represents something significant about how
the ada-002 embeddings are created, but I think it also
means creating "realistic" values is possible. I did not
use this information when testing recall & performance on
Lucene's HNSW implementation on 192m documents, as I
slightly dithered the values of a "real" set on 47K docs
and stored other fields in the doc that referenced the
"base" document that the dithers were made from, and used
different dithering magnitudes so that I could test recall
with different neighbour sizes ("M"),
construction-beamwidth and search-beamwidths.
best regards
Kent Fitch
On Wed, Apr 12, 2023 at 5:08 AM Michael Wechner
<michael.wech...@wyona.com> wrote:
I understand what you mean that it seems to be
artificial, but I don't
understand why this matters to test performance and
scalability of the
indexing?
Let's assume the limit of Lucene would be 4 instead of
1024 and there
are only open source models generating vectors with 4
dimensions, for
example
0.02150459587574005,0.11223817616701126,-0.007903356105089188,0.03795722872018814
0.026009393855929375,0.006306684575974941,0.020492585375905037,-0.029064252972602844
-0.08239810913801193,-0.01947402022778988,0.03827739879488945,-0.020566290244460106
-0.007012288551777601,-0.026665858924388885,0.044495150446891785,-0.038030195981264114
and now I concatenate them to vectors with 8 dimensions
0.02150459587574005,0.11223817616701126,-0.007903356105089188,0.03795722872018814,0.026009393855929375,0.006306684575974941,0.020492585375905037,-0.029064252972602844
-0.08239810913801193,-0.01947402022778988,0.03827739879488945,-0.020566290244460106,-0.007012288551777601,-0.026665858924388885,0.044495150446891785,-0.038030195981264114
and normalize them to length 1.
Why should this be any different to a model which is
acting like a black
box generating vectors with 8 dimensions?
Am 11.04.23 um 19:05 schrieb Michael Sokolov:
>> What exactly do you consider real vector data?
Vector data which is based on texts written by humans?
> We have plenty of text; the problem is coming up with
a realistic
> vector model that requires as many dimensions as
people seem to be
> demanding. As I said above, after surveying
huggingface I couldn't
> find any text-based model using more than 768
dimensions. So far we
> have some ideas of generating higher-dimensional data
by dithering or
> concatenating existing data, but it seems artificial.
>
> On Tue, Apr 11, 2023 at 9:31 AM Michael Wechner
> <michael.wech...@wyona.com> wrote:
>> What exactly do you consider real vector data?
Vector data which is based on texts written by humans?
>>
>> I am asking, because I recently attended the
following presentation by Anastassia Shaitarova (UZH
Institute for Computational Linguistics,
https://www.cl.uzh.ch/de/people/team/compling/shaitarova.html)
>>
>> ----
>>
>> Can we Identify Machine-Generated Text? An Overview
of Current Approaches
>> by Anastassia Shaitarova (UZH Institute for
Computational Linguistics)
>>
>> The detection of machine-generated text has become
increasingly important due to the prevalence of
automated content generation and its potential for
misuse. In this talk, we will discuss the motivation
for automatic detection of generated text. We will
present the currently available methods, including
feature-based classification as a “first
line-of-defense.” We will provide an overview of the
detection tools that have been made available so far
and discuss their limitations. Finally, we will reflect
on some open problems associated with the automatic
discrimination of generated texts.
>>
>> ----
>>
>> and her conclusion was that it has become basically
impossible to differentiate between text generated by
humans and text generated by for example ChatGPT.
>>
>> Whereas others have a slightly different opinion,
see for example
>>
>>
https://www.wired.com/story/how-to-spot-generative-ai-text-chatgpt/
>>
>> But I would argue that real world and synthetic have
become close enough that testing performance and
scalability of indexing should be possible with
synthetic data.
>>
>> I completely agree that we have to base our
discussions and decisions on scientific methods and
that we have to make sure that Lucene performs and
scales well and that we understand the limits and what
is going on under the hood.
>>
>> Thanks
>>
>> Michael W
>>
>>
>>
>>
>>
>> Am 11.04.23 um 14:29 schrieb Michael McCandless:
>>
>> +1 to test on real vector data -- if you test on
synthetic data you draw synthetic conclusions.
>>
>> Can someone post the theoretical performance (CPU
and RAM required) of HNSW construction? Do we
know/believe our HNSW implementation has achieved that
theoretical big-O performance? Maybe we have some
silly performance bug that's causing it not to?
>>
>> As I understand it, HNSW makes the tradeoff of
costly construction for faster searching, which is
typically the right tradeoff for search use cases. We
do this in other parts of the Lucene index too.
>>
>> Lucene will do a logarithmic number of merges over
time, i.e. each doc will be merged O(log(N)) times in
its lifetime in the index. We need to multiply that by
the cost of re-building the whole HNSW graph on each
merge. BTW, other things in Lucene, like
BKD/dimensional points, also rebuild the whole data
structure on each merge, I think? But, as Rob pointed
out, stored fields merging do indeed do some sneaky
tricks to avoid excessive block decompress/recompress
on each merge.
>>
>>> As I understand it, vetoes must have technical
merit. I'm not sure that this veto rises to "technical
merit" on 2 counts:
>> Actually I think Robert's veto stands on its
technical merit already. Robert's take on technical
matters very much resonate with me, even if he is
sometimes prickly in how he expresses them ;)
>>
>> His point is that we, as a dev community, are not
paying enough attention to the indexing performance of
our KNN algo (HNSW) and implementation, and that it is
reckless to increase / remove limits in that state. It
is indeed a one-way door decision and one must confront
such decisions with caution, especially for such a
widely used base infrastructure as Lucene. We don't
even advertise today in our javadocs that you need XXX
heap if you index vectors with dimension Y, fanout X,
levels Z, etc.
>>
>> RAM used during merging is unaffected by
dimensionality, but is affected by fanout, because the
HNSW graph (not the raw vectors) is memory resident, I
think? Maybe we could move it off-heap and let the OS
manage the memory (and still document the RAM
requirements)? Maybe merge RAM costs should be
accounted for in IW's RAM buffer accounting? It is not
today, and there are some other things that use
non-trivial RAM, e.g. the doc mapping (to compress
docid space when deletions are reclaimed).
>>
>> When we added KNN vector testing to Lucene's nightly
benchmarks, the indexing time massively increased --
see annotations DH and DP here:
https://home.apache.org/~mikemccand/lucenebench/indexing.html.
Nightly benchmarks now start at 6 PM and don't finish
until ~14.5 hours later. Of course, that is using a
single thread for indexing (on a box that has 128
cores!) so we produce a deterministic index every night ...
>>
>> Stepping out (meta) a bit ... this discussion is
precisely one of the awesome benefits of the (informed)
veto. It means risky changes to the software, as
determined by any single informed developer on the
project, can force a healthy discussion about the
problem at hand. Robert is legitimately concerned
about a real issue and so we should use our creative
energies to characterize our HNSW implementation's
performance, document it clearly for users, and uncover
ways to improve it.
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Mon, Apr 10, 2023 at 6:41 PM Alessandro Benedetti
<a.benede...@sease.io> wrote:
>>> I think Gus points are on target.
>>>
>>> I recommend we move this forward in this way:
>>> We stop any discussion and everyone interested
proposes an option with a motivation, then we aggregate
the options and we create a Vote maybe?
>>>
>>> I am also on the same page on the fact that a veto
should come with a clear and reasonable technical
merit, which also in my opinion has not come yet.
>>>
>>> I also apologise if any of my words sounded harsh
or personal attacks, never meant to do so.
>>>
>>> My proposed option:
>>>
>>> 1) remove the limit and potentially make it
configurable,
>>> Motivation:
>>> The system administrator can enforce a limit its
users need to respect that it's in line with whatever
the admin decided to be acceptable for them.
>>> Default can stay the current one.
>>>
>>> That's my favourite at the moment, but I agree that
potentially in the future this may need to change, as
we may optimise the data structures for certain
dimensions. I am a big fan of Yagni (you aren't going
to need it) so I am ok we'll face a different
discussion if that happens in the future.
>>>
>>>
>>>
>>> On Sun, 9 Apr 2023, 18:46 Gus Heck,
<gus.h...@gmail.com> wrote:
>>>> What I see so far:
>>>>
>>>> Much positive support for raising the limit
>>>> Slightly less support for removing it or making it
configurable
>>>> A single veto which argues that a (as yet
undefined) performance standard must be met before
raising the limit
>>>> Hot tempers (various) making this discussion difficult
>>>>
>>>> As I understand it, vetoes must have technical
merit. I'm not sure that this veto rises to "technical
merit" on 2 counts:
>>>>
>>>> No standard for the performance is given so it
cannot be technically met. Without hard criteria it's a
moving target.
>>>> It appears to encode a valuation of the user's
time, and that valuation is really up to the user. Some
users may consider 2hours useless and not worth it, and
others might happily wait 2 hours. This is not a
technical decision, it's a business decision regarding
the relative value of the time invested vs the value of
the result. If I can cure cancer by indexing for a
year, that might be worth it... (hyperbole of course).
>>>>
>>>> Things I would consider to have technical merit
that I don't hear:
>>>>
>>>> Impact on the speed of **other** indexing
operations. (devaluation of other functionality)
>>>> Actual scenarios that work when the limit is low
and fail when the limit is high (new failure on the
same data with the limit raised).
>>>>
>>>> One thing that might or might not have technical merit
>>>>
>>>> If someone feels there is a lack of documentation
of the costs/performance implications of using large
vectors, possibly including reproducible benchmarks
establishing the scaling behavior (there seems to be
disagreement on O(n) vs O(n^2)).
>>>>
>>>> The users *should* know what they are getting
into, but if the cost is worth it to them, they should
be able to pay it without forking the project. If this
veto causes a fork that's not good.
>>>>
>>>> On Sun, Apr 9, 2023 at 7:55 AM Michael Sokolov
<msoko...@gmail.com> wrote:
>>>>> We do have a dataset built from Wikipedia in
luceneutil. It comes in 100 and 300 dimensional
varieties and can easily enough generate large numbers
of vector documents from the articles data. To go
higher we could concatenate vectors from that and I
believe the performance numbers would be plausible.
>>>>>
>>>>> On Sun, Apr 9, 2023, 1:32 AM Dawid Weiss
<dawid.we...@gmail.com> wrote:
>>>>>> Can we set up a branch in which the limit is
bumped to 2048, then have
>>>>>> a realistic, free data set (wikipedia sample or
something) that has,
>>>>>> say, 5 million docs and vectors created using
public data (glove
>>>>>> pre-trained embeddings or the like)? We then
could run indexing on the
>>>>>> same hardware with 512, 1024 and 2048 and see
what the numbers, limits
>>>>>> and behavior actually are.
>>>>>>
>>>>>> I can help in writing this but not until after
Easter.
>>>>>>
>>>>>>
>>>>>> Dawid
>>>>>>
>>>>>> On Sat, Apr 8, 2023 at 11:29 PM Adrien Grand
<jpou...@gmail.com> wrote:
>>>>>>> As Dawid pointed out earlier on this thread,
this is the rule for
>>>>>>> Apache projects: a single -1 vote on a code
change is a veto and
>>>>>>> cannot be overridden. Furthermore, Robert is
one of the people on this
>>>>>>> project who worked the most on debugging subtle
bugs, making Lucene
>>>>>>> more robust and improving our test framework,
so I'm listening when he
>>>>>>> voices quality concerns.
>>>>>>>
>>>>>>> The argument against removing/raising the limit
that resonates with me
>>>>>>> the most is that it is a one-way door. As MikeS
highlighted earlier on
>>>>>>> this thread, implementations may want to take
advantage of the fact
>>>>>>> that there is a limit at some point too. This
is why I don't want to
>>>>>>> remove the limit and would prefer a slight
increase, such as 2048 as
>>>>>>> suggested in the original issue, which would
enable most of the things
>>>>>>> that users who have been asking about raising
the limit would like to
>>>>>>> do.
>>>>>>>
>>>>>>> I agree that the merge-time memory usage and
slow indexing rate are
>>>>>>> not great. But it's still possible to index
multi-million vector
>>>>>>> datasets with a 4GB heap without hitting OOMEs
regardless of the
>>>>>>> number of dimensions, and the feedback I'm
seeing is that many users
>>>>>>> are still interested in indexing multi-million
vector datasets despite
>>>>>>> the slow indexing rate. I wish we could do
better, and vector indexing
>>>>>>> is certainly more expert than text indexing,
but it still is usable in
>>>>>>> my opinion. I understand how giving Lucene more
information about
>>>>>>> vectors prior to indexing (e.g. clustering
information as Jim pointed
>>>>>>> out) could help make merging faster and more
memory-efficient, but I
>>>>>>> would really like to avoid making it a
requirement for indexing
>>>>>>> vectors as it also makes this feature much
harder to use.
>>>>>>>
>>>>>>> On Sat, Apr 8, 2023 at 9:28 PM Alessandro Benedetti
>>>>>>> <a.benede...@sease.io> wrote:
>>>>>>>> I am very attentive to listen opinions but I
am un-convinced here and I an not sure that a single
person opinion should be allowed to be detrimental for
such an important project.
>>>>>>>>
>>>>>>>> The limit as far as I know is literally just
raising an exception.
>>>>>>>> Removing it won't alter in any way the current
performance for users in low dimensional space.
>>>>>>>> Removing it will just enable more users to use
Lucene.
>>>>>>>>
>>>>>>>> If new users in certain situations will be
unhappy with the performance, they may contribute
improvements.
>>>>>>>> This is how you make progress.
>>>>>>>>
>>>>>>>> If it's a reputation thing, trust me that not
allowing users to play with high dimensional space will
equally damage it.
>>>>>>>>
>>>>>>>> To me it's really a no brainer.
>>>>>>>> Removing the limit and enable people to use
high dimensional vectors will take minutes.
>>>>>>>> Improving the hnsw implementation can take months.
>>>>>>>> Pick one to begin with...
>>>>>>>>
>>>>>>>> And there's no-one paying me here, no company
interest whatsoever, actually I pay people to
contribute, I am just convinced it's a good idea.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, 8 Apr 2023, 18:57 Robert Muir,
<rcm...@gmail.com> wrote:
>>>>>>>>> I disagree with your categorization. I put in
plenty of work and
>>>>>>>>> experienced plenty of pain myself, writing
tests and fighting these
>>>>>>>>> issues, after i saw that, two releases in a
row, vector indexing fell
>>>>>>>>> over and hit integer overflows etc on small
datasets:
>>>>>>>>>
>>>>>>>>> https://github.com/apache/lucene/pull/11905
>>>>>>>>>
>>>>>>>>> Attacking me isn't helping the situation.
>>>>>>>>>
>>>>>>>>> PS: when i said the "one guy who wrote the
code" I didn't mean it in
>>>>>>>>> any kind of demeaning fashion really. I meant
to describe the current
>>>>>>>>> state of usability with respect to indexing a
few million docs with
>>>>>>>>> high dimensions. You can scroll up the thread
and see that at least
>>>>>>>>> one other committer on the project
experienced similar pain as me.
>>>>>>>>> Then, think about users who aren't committers
trying to use the
>>>>>>>>> functionality!
>>>>>>>>>
>>>>>>>>> On Sat, Apr 8, 2023 at 12:51 PM Michael
Sokolov <msoko...@gmail.com> wrote:
>>>>>>>>>> What you said about increasing dimensions
requiring a bigger ram buffer on merge is wrong. That's
the point I was trying to make. Your concerns about
merge costs are not wrong, but your conclusion that we
need to limit dimensions is not justified.
>>>>>>>>>>
>>>>>>>>>> You complain that hnsw sucks it doesn't
scale, but when I show it scales linearly with
dimension you just ignore that and complain about
something entirely different.
>>>>>>>>>>
>>>>>>>>>> You demand that people run all kinds of
tests to prove you wrong but when they do, you don't
listen and you won't put in the work yourself or
complain that it's too hard.
>>>>>>>>>>
>>>>>>>>>> Then you complain about people not meeting
you half way. Wow
>>>>>>>>>>
>>>>>>>>>> On Sat, Apr 8, 2023, 12:40 PM Robert Muir
<rcm...@gmail.com> wrote:
>>>>>>>>>>> On Sat, Apr 8, 2023 at 8:33 AM Michael Wechner
>>>>>>>>>>> <michael.wech...@wyona.com> wrote:
>>>>>>>>>>>> What exactly do you consider reasonable?
>>>>>>>>>>> Let's begin a real discussion by being
HONEST about the current
>>>>>>>>>>> status. Please put politically correct or
your own company's wishes
>>>>>>>>>>> aside, we know it's not in a good state.
>>>>>>>>>>>
>>>>>>>>>>> Current status is the one guy who wrote the
code can set a
>>>>>>>>>>> multi-gigabyte ram buffer and index a small
dataset with 1024
>>>>>>>>>>> dimensions in HOURS (i didn't ask what
hardware).
>>>>>>>>>>>
>>>>>>>>>>> My concerns are everyone else except the
one guy, I want it to be
>>>>>>>>>>> usable. Increasing dimensions just means
even bigger multi-gigabyte
>>>>>>>>>>> ram buffer and bigger heap to avoid OOM on
merge.
>>>>>>>>>>> It is also a permanent backwards
compatibility decision, we have to
>>>>>>>>>>> support it once we do this and we can't
just say "oops" and flip it
>>>>>>>>>>> back.
>>>>>>>>>>>
>>>>>>>>>>> It is unclear to me, if the multi-gigabyte
ram buffer is really to
>>>>>>>>>>> avoid merges because they are so slow and
it would be DAYS otherwise,
>>>>>>>>>>> or if its to avoid merges so it doesn't hit
OOM.
>>>>>>>>>>> Also from personal experience, it takes
trial and error (means
>>>>>>>>>>> experiencing OOM on merge!!!) before you
get those heap values correct
>>>>>>>>>>> for your dataset. This usually means
starting over which is
>>>>>>>>>>> frustrating and wastes more time.
>>>>>>>>>>>
>>>>>>>>>>> Jim mentioned some ideas about the memory
usage in IndexWriter, seems
>>>>>>>>>>> to me like its a good idea. maybe the
multigigabyte ram buffer can be
>>>>>>>>>>> avoided in this way and performance
improved by writing bigger
>>>>>>>>>>> segments with lucene's defaults. But this
doesn't mean we can simply
>>>>>>>>>>> ignore the horrors of what happens on
merge. merging needs to scale so
>>>>>>>>>>> that indexing really scales.
>>>>>>>>>>>
>>>>>>>>>>> At least it shouldnt spike RAM on trivial
data amounts and cause OOM,
>>>>>>>>>>> and definitely it shouldnt burn hours and
hours of CPU in O(n^2)
>>>>>>>>>>> fashion when indexing.
>>>>>>>>>>>
>>>>>>>>>>>
---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail:
dev-unsubscr...@lucene.apache.org
>>>>>>>>>>> For additional commands, e-mail:
dev-h...@lucene.apache.org
>>>>>>>>>>>
>>>>>>>>>
---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail:
dev-unsubscr...@lucene.apache.org
>>>>>>>>> For additional commands, e-mail:
dev-h...@lucene.apache.org
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Adrien
>>>>>>>
>>>>>>>
---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail:
dev-unsubscr...@lucene.apache.org
>>>>>>> For additional commands, e-mail:
dev-h...@lucene.apache.org
>>>>>>>
>>>>>>
---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail:
dev-unsubscr...@lucene.apache.org
>>>>>> For additional commands, e-mail:
dev-h...@lucene.apache.org
>>>>>>
>>>>
>>>> --
>>>> http://www.needhamsoftware.com (work)
>>>> http://www.the111shift.com (play)
>>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail:
dev-h...@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
