Smalyshev added a comment.
Moving to nodes didn't work either - when defining indexes for every property,
it creates over 20K lucene files and tries to open all of them and gets out of
filehandles, even though I've put a high limit on file descriptors (10)
somehow it's not enough. Looks lik
Liuxinyu970226 added a subscriber: Liuxinyu970226.
TASK DETAIL
https://phabricator.wikimedia.org/T87524
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
Liuxinyu970226 added a subscriber: Liuxinyu970226.
TASK DETAIL
https://phabricator.wikimedia.org/T62369
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
Smalyshev added a comment.
Main problem with Neo4j so far is indexing - edge indexes are either manual
addition (http://neo4j.com/docs/2.2.0-M03/indexing-add.html) or auto-indexing
(http://neo4j.com/docs/2.2.0-M03/auto-indexing.html) which requires
pre-defining all properties in configuration o
He7d3r added a project: JavaScript.
TASK DETAIL
https://phabricator.wikimedia.org/T85541
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: He7d3r
Cc
Smalyshev added a comment.
@Neunhoef In current data model, each edge carries a primary value, a boolean
flag and a small set (usually well under 10, in most cases 1-3 or none)
secondary values, each of which need to be indexed. It also can keep a set of
auxiliary values for each of those, whic
Manybubbles added a comment.
In https://phabricator.wikimedia.org/T88549#1021733, @Neunhoef wrote:
> I cannot really answer your question, in particular since it will depend on
> whether you have only "thousands of hash indexes" or even "thousands of
> skiplist indexes", as the above timings su
Neunhoef added a comment.
@Smalyshev: Please also note: in my tests today the 4 indexes for 3M documents
needed 672 MB of data, which is a reasonable amount of 226 bytes per document.
That should be about 3GB (optimistically!) for your 16M documents (forgetting
about edges!) If you really have
bd808 moved this task to Done on the MediaWiki-Core-Team workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T88550
WORKBOARD
https://phabricator.wikimedia.org/project/board/37/
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAI
Neunhoef added a comment.
> Do you think sparse indexes are going to give us enough performance thousands
> of these indexes with similar startup times to what we see now? Is that
> something we can hack around by using fewer indexes and searching them in
> more interesting ways? Say we just
Smalyshev added a comment.
The thing which worries me the most is the non-persistent indexes. Note that
the real data size would be not 3M nodes but about 20M nodes //and// over 100M
edges, which all need to be indexed. Or, if we convert edges to nodes for
indexing, that'd be about 150M nodes a
Neunhoef added a comment.
> Assuming we're OK with just planning for large server deployments: Does the
> memory requirement scale linearly with the size of the data? How does that
> play with sharding and replication? How large are the largest ArangoDB
> clusters?
The memory requirements f
Neunhoef added a comment.
> OK. Maybe its just a function of not using a super nice machine for testing.
> We really do want the system to scale down to work with less ram and cheap,
> big spinning disks so folks can run it on their laptop. That'll encourage
> experimentation.
Well, Arango
JanZerebecki assigned this task to thiemowmde.
JanZerebecki set Security to none.
TASK DETAIL
https://phabricator.wikimedia.org/T88437
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/s
JanZerebecki assigned this task to thiemowmde.
JanZerebecki set Security to none.
TASK DETAIL
https://phabricator.wikimedia.org/T88438
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/s
JanZerebecki assigned this task to hoo.
TASK DETAIL
https://phabricator.wikimedia.org/T76213
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hoo,
JanZerebecki moved this task to Done on the § Wikidata-Sprint-2015-02-03
workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T86694
WORKBOARD
https://phabricator.wikimedia.org/project/board/1034/
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe
Manybubbles moved this task to In Dev/Progress on the MediaWiki-Core-Team
workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T88721
WORKBOARD
https://phabricator.wikimedia.org/project/board/37/
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe o
Manybubbles reassigned this task from Manybubbles to Smalyshev.
TASK DETAIL
https://phabricator.wikimedia.org/T88821
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emai
Manybubbles moved this task to Needs Review/Feedback on the MediaWiki-Core-Team
workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T88549
WORKBOARD
https://phabricator.wikimedia.org/project/board/37/
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubsc
Manybubbles moved this task to In Dev/Progress on the MediaWiki-Core-Team
workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T88821
WORKBOARD
https://phabricator.wikimedia.org/project/board/37/
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe o
Manybubbles triaged this task as "High" priority.
Manybubbles added a project: wikidata-query-service.
Manybubbles set Security to none.
TASK DETAIL
https://phabricator.wikimedia.org/T88821
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
Manybubbles added a comment.
In https://phabricator.wikimedia.org/T88571#1017609, @daniel wrote:
> Neo4j river plugin for Elastic:
> https://github.com/sksamuel/elasticsearch-river-neo4j
Rivers are essentially a deprecated concept - stay away if you like sanity.
TASK DETAIL
https://phabric
Manybubbles added a comment.
Short answer is that Neo4j is indeed a contender and a pretty good one at that.
TASK DETAIL
https://phabricator.wikimedia.org/T88571
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https:
Manybubbles moved this task to Needs Review/Feedback on the MediaWiki-Core-Team
workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T88571
WORKBOARD
https://phabricator.wikimedia.org/project/board/37/
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubsc
hoo added a subscriber: hoo.
hoo closed this task as "Resolved".
hoo added a comment.
Patch approved... now we "only" need to remember to bump that constant whenever
we do breaking changes.
TASK DETAIL
https://phabricator.wikimedia.org/T86694
REPLY HANDLER ACTIONS
Reply to comment or attac
gerritbot added a comment.
Change 188277 merged by jenkins-bot:
Add version constant for EntityContent for parser options
https://gerrit.wikimedia.org/r/188277
TASK DETAIL
https://phabricator.wikimedia.org/T86694
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !u
JanZerebecki moved this task to Review on the § Wikidata-Sprint-2015-02-03
workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T76213
WORKBOARD
https://phabricator.wikimedia.org/project/board/1034/
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscrib
JanZerebecki added a project: § Wikidata-Sprint-2015-02-03.
TASK DETAIL
https://phabricator.wikimedia.org/T76213
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpre
Lydia_Pintscher triaged this task as "High" priority.
Lydia_Pintscher added subscribers: Snaterlicious, adrianheine, thiemowmde.
TASK DETAIL
https://phabricator.wikimedia.org/T88804
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PR
JanZerebecki edited the task description.
JanZerebecki set Security to none.
TASK DETAIL
https://phabricator.wikimedia.org/T88804
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settin
JanZerebecki created this task.
JanZerebecki added a subscriber: JanZerebecki.
JanZerebecki added projects: Wikidata, MediaWiki-extensions-WikibaseRepository.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
Clicking on "more" in the autocomplete in a new claim value opens t
gerritbot added a subscriber: gerritbot.
gerritbot added a project: Patch-For-Review.
gerritbot added a comment.
Change 189027 had a related patch set uploaded (by Hoo man):
Add mw.wikibase.renderSnak(s) for rendering arbitrary Snaks
https://gerrit.wikimedia.org/r/189027
https://phabricator.wiki
thiemowmde added a project: Patch-For-Review.
thiemowmde added a comment.
https://github.com/DataValues/Time/pull/35
TASK DETAIL
https://phabricator.wikimedia.org/T88438
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
Snaterlicious added a comment.
https://gerrit.wikimedia.org/r/#/c/189018/
Along with the refactoring, the Property DOM is nor re-rendered on page load
anymore. This definitely improves performance. `SnakTypeSelector` and
`Variation` are sill re-rendered though. That may be avoided as well (as th
Manybubbles added a comment.
In https://phabricator.wikimedia.org/T88549#1018178, @Fceller wrote:
> Hi, I'm the CTO of ArangoDB, so my comments are most certainly biased. I
> still would like to tell you about our opinions on the raised issues, namely
> full-text indexes and blueprint.
Thanks
Manybubbles added a comment.
In https://phabricator.wikimedia.org/T88549#1020635, @Neunhoef wrote:
> Disclaimer: Sorry, I forgot to introduce myself: My name is Max and I also
> work for ArangoDB.
Thanks!
> The 3M documents need around 11 GB of main memory. If you have less, then you
> see a
Snipre added a subscriber: Snipre.
TASK DETAIL
https://phabricator.wikimedia.org/T77981
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Snipre
Cc:
Tobi_WMDE_SW added a subscriber: zeljkofilipin.
TASK DETAIL
https://phabricator.wikimedia.org/T88541
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
T
Lydia_Pintscher moved this task to consider for next sprint on the Wikidata
workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T88795
WORKBOARD
https://phabricator.wikimedia.org/project/board/71/
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe
Lydia_Pintscher moved this task to needs discussion or investigation on the
Wikidata workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T88735
WORKBOARD
https://phabricator.wikimedia.org/project/board/71/
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !un
Lydia_Pintscher moved this task to consider for next sprint on the Wikidata
workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T88796
WORKBOARD
https://phabricator.wikimedia.org/project/board/71/
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe
Neunhoef added a comment.
Disclaimer: Sorry, I forgot to introduce myself: My name is Max and I also work
for ArangoDB.
Analysis
The 3M documents need around 11 GB of main memory. If you have less, then you
see a lot of swapping, because the insert operation in the indexes will
essentially do
Lydia_Pintscher moved this task to Done on the § Wikidata-Sprint-2015-02-03
workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T88695
WORKBOARD
https://phabricator.wikimedia.org/project/board/1034/
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscri
Lydia_Pintscher reassigned this task from Lydia_Pintscher to Snaterlicious.
Lydia_Pintscher set Security to none.
TASK DETAIL
https://phabricator.wikimedia.org/T88695
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
htt
Lydia_Pintscher closed blocking task T88695: Don't italicize text when entering
aliases as "Resolved".
TASK DETAIL
https://phabricator.wikimedia.org/T75654
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabr
Lydia_Pintscher closed this task as "Resolved".
Lydia_Pintscher claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T88695
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/
Lydia_Pintscher added a project: § Wikidata-Sprint-2015-02-03.
TASK DETAIL
https://phabricator.wikimedia.org/T88695
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/email
Lydia_Pintscher added subscribers: hoo, daniel, Lydia_Pintscher.
Lydia_Pintscher added a comment.
We'll not be anywhere near to supporting structured data on Commons in time for
the hackathon. What could be hacked on is improving remote editing as a whole
independent of Commons and structured da
Lydia_Pintscher added subscribers: daniel, Lydia_Pintscher.
TASK DETAIL
https://phabricator.wikimedia.org/T88735
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpre
Neunhoef added a comment.
Further experiments:
I dropped each of these indexes and recreated it. This isolates the rebuilding
of the indexes from the loading (and scanning) of the collection files. Here
are the timing results for the recreation (dropping is instantaneous):
- unique hash index
Lydia_Pintscher added a comment.
If the second gives uns performance improvement then that one. Otherwise the
first one please.
TASK DETAIL
https://phabricator.wikimedia.org/T88425
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL
Neunhoef added a subscriber: Neunhoef.
Neunhoef added a comment.
This is a report about an actual experiment.
I downloaded 20150126.json.gz to an AWS r3.2xlarge instance (61GB RAM, 80 GB
SSD, 8 vCPUs) and then used ArangoDB V 2.4.1 do do the import of the first
2961954 documents in the file. I
Snaterlicious added a subscriber: Lydia_Pintscher.
Snaterlicious added a comment.
While suppressing JS redrawing on load is not that hard (in relative
terms--still, it is the `snakview`), additional implications show up: When
stopping edit mode ("save"/"cancel"), the by far easiest action to upd
JanZerebecki added a comment.
This also has suggestions: https://www.mediawiki.org/wiki/Talk:Wikibase/Indexing
TASK DETAIL
https://phabricator.wikimedia.org/T86786
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
http
Tobi_WMDE_SW created this task.
Tobi_WMDE_SW added subscribers: WMDE-Fisch, Tobi_WMDE_SW, Aklapper.
Tobi_WMDE_SW added projects: Wikidata, MediaWiki-extensions-WikibaseClient.
TASK DESCRIPTION
TASK DETAIL
https://phabricator.wikimedia.org/T88796
REPLY HANDLER ACTIONS
Reply to comment or att
Tobi_WMDE_SW created this task.
Tobi_WMDE_SW added subscribers: WMDE-Fisch, Tobi_WMDE_SW, Aklapper.
Tobi_WMDE_SW added a project: Wikidata.
TASK DESCRIPTION
TASK DETAIL
https://phabricator.wikimedia.org/T88795
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsub
Nemo_bis added a subscriber: Nikerabbit.
Nemo_bis set Security to none.
TASK DETAIL
https://phabricator.wikimedia.org/T64695
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/pa
Nemo_bis added a subscriber: Nemo_bis.
TASK DETAIL
https://phabricator.wikimedia.org/T4
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Nemo_b
Snaterlicious added a comment.
https://gerrit.wikimedia.org/r/#/c/188990/
TASK DETAIL
https://phabricator.wikimedia.org/T88695
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/setting
daniel added subscribers: Aklapper, daniel.
daniel merged a task: T86186: Use subscription service for dispatching changes
to client.
TASK DETAIL
https://phabricator.wikimedia.org/T66590
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EM
daniel closed this task as a duplicate of T66590: Usage Tracking: dispatcher
needs to figure out which clients to send a change to.
TASK DETAIL
https://phabricator.wikimedia.org/T86186
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAI
daniel closed blocking task T86185: Track the subscriptions as "Resolved".
TASK DETAIL
https://phabricator.wikimedia.org/T73052
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings
daniel closed this task as "Resolved".
daniel claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T86185
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/ema
Lydia_Pintscher added a subscriber: Lydia_Pintscher.
Lydia_Pintscher added a comment.
What @Tgr said. Categories as they are are associated with pages and not
topics. They are also completely different across languages/cultures. So I'd
say no to moving them to Wikidata. That being said there is
65 matches
Mail list logo