[Wikidata-bugs] [Maniphest] [Commented On] T164773: Error replicating wikidata blazegraph setup
Smalyshev added a comment. Is this still an issue?TASK DETAILhttps://phabricator.wikimedia.org/T164773EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Hjfocs, Gehel, Smalyshev, Aklapper, Yurik, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T164773: Error replicating wikidata blazegraph setup
Yurik added a comment. @Smalyshev, I force-killed it, and it dumped this (I couldn't copy all of it - just some parts): at com.bigdata.rdf.sail.webapp.RESTServlet.doGet(RESTServlet.java:240) at com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doGet(MultiTenancyServlet.java:271) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667) at org.wikidata.query.rdf.blazegraph.throttling.ThrottlingFilter.doFilter(ThrottlingFilter.java:244) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at ch.qos.logback.classic.helpers.MDCInsertingServletFilter.doFilter(MDCInsertingServletFilter.java:51) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at org.wikidata.query.rdf.blazegraph.filters.ClientIPFilter.doFilter(ClientIPFilter.java:42) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:497) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539) at java.lang.Thread.run(Thread.java:748) "qtp931919113-27686" #27686 prio=5 os_prio=0 tid=0x7fbb5c00c000 nid=0x2c11 waiting on condition [0x7fba671f3000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0007febc0168> (a java.util.concurrent.FutureTask) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429) at java.util.concurrent.FutureTask.get(FutureTask.java:191) at com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:286) at com.bigdata.rdf.sail.webapp.QueryServlet.doSparqlQuery(QueryServlet.java:653) at com.bigdata.rdf.sail.webapp.QueryServlet.doGet(QueryServlet.java:288) at com.bigdata.rdf.sail.webapp.RESTServlet.doGet(RESTServlet.java:240) at com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doGet(MultiTenancyServlet.java:271) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667) at org.wikidata.query.rdf.blazegraph.throttling.ThrottlingFilter.doFilter(ThrottlingFilter.java:244) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at ch.qos.logback.classic.helpers.MDCInsertingServletFilter.doFilter(MDCInsertingServletFilter.java:51) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at org.wikidata.query.rdf.blazegraph.filters.ClientIPFilter.doFilter(ClientIPFilter.java:42) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at
[Wikidata-bugs] [Maniphest] [Commented On] T164773: Error replicating wikidata blazegraph setup
Smalyshev added a comment. Not sure whether it has some relation, hard to diagnose from this. Did you see the GC logs? What the status page for Blazegraph showed?TASK DETAILhttps://phabricator.wikimedia.org/T164773EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Gehel, Smalyshev, Aklapper, Yurik, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T164773: Error replicating wikidata blazegraph setup
Yurik added a comment. @Smalyshev & @Gehel I'm not sure if this is the same error or different. Today, the service froze in a peculiar way: all queries would time out (both from clients and update ones), and Blazegraph wouldn't quit with Ctrl+C. HTOP shows a single blazegraph process using 100% of a single CPU, but about once in a few minutes, almost all CPUs would jump to a 100% for about 5-10 seconds, and then go back to 0 except for a single process. The last errors in the log, might be unrelated: at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:159) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.rwstore.sector.AllocationContext.alloc(AllocationContext.java:359) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.rwstore.PSOutputStream.save(PSOutputStream.java:335) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.rwstore.PSOutputStream.getAddr(PSOutputStream.java:416) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.bop.solutions.SolutionSetStream.put(SolutionSetStream.java:297) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.bop.engine.LocalNativeChunkMessage.(LocalNativeChunkMessage.java:213) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.bop.engine.LocalNativeChunkMessage.(LocalNativeChunkMessage.java:147) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.bop.engine.StandaloneChunkHandler.handleChunk(StandaloneChunkHandler.java:90) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.outputChunk(ChunkedRunningQuery.java:1699) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.addReorderAllowed(ChunkedRunningQuery.java:1628) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1569) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1453) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:59) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:14) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.overflow(AbstractUnsynchronizedArrayBuffer.java:287) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add2(AbstractUnsynchronizedArrayBuffer.java:215) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add(AbstractUnsynchronizedArrayBuffer.java:173) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.handleJoin2(PipelineJoin.java:1868) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] ... 17 common frames omitted 07:53:53.567 [com.bigdata.journal.Journal.executorService23435] ERROR com.bigdata.util.concurrent.Haltable IP: UA: - com.bigdata.bop.join.PipelineJoin$JoinTask{ joinOp=com.bigdata.bop.join.PipelineJoin[2]()[ BOp.bopId=2, JoinAnnotations.constraints=null, AST2BOpBase.simpleJoin=true, BOp.evaluationContext=ANY, AccessPathJoinAnnotations.predicate=com.bigdata.rdf.spo.SPOPredicate[1](osmId=null, Vocab(56)[https://wiki.openstreetmap.org/wiki/Key:place], place=null)[ IPredicate.relationName=[wdq.spo], IPredicate.timestamp=1505627633562, BOp.bopId=1, AST2BOpBase.estimatedCardinality=4549666, AST2BOpBase.originalIndex=POS, IPredicate.flags=[KEYS,VALS,READONLY,PARALLEL]]]} : isFirstCause=true : com.bigdata.rwstore.sector.MemoryManagerClosedException com.bigdata.rwstore.sector.MemoryManagerClosedException: null at com.bigdata.rwstore.sector.MemoryManager.assertOpen(MemoryManager.java:110) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.rwstore.sector.MemoryManager.allocate(MemoryManager.java:671) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:195) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:169) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:159) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.rwstore.sector.AllocationContext.alloc(AllocationContext.java:359) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.rwstore.PSOutputStream.save(PSOutputStream.java:335) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.rwstore.PSOutputStream.getAddr(PSOutputStream.java:416) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.bop.solutions.SolutionSetStream.put(SolutionSetStream.java:297) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at com.bigdata.bop.engine.LocalNativeChunkMessage.(LocalNativeChunkMessage.java:213) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na] at
[Wikidata-bugs] [Maniphest] [Commented On] T164773: Error replicating wikidata blazegraph setup
Yurik added a comment. I just finished full re-import with -Xmx=16GB , and it worked fine. I am not sure what has caused the original issue. For future reference, here are the import stats (importing both Wikidata and OSM data from the same dir - 33GB total, 683 files of about the same size. Total time was about 36 hours. First 329 files were OSM, second 354 - wikidata (note how the graph starts jumping, possibly due to per-file statement distribution in WD. OSM data is much more uniform in terms of statement per osm object) F9290936: image.png raw datastatements time rate sets 107,179,984 640 167,220 6 189,557,036 1,260 150,378 11 255,343,483 1,884 135,523 15 323,230,658 2,533 127,587 19 389,362,380 3,231 120,504 23 454,181,776 3,971 114,351 27 519,770,290 4,654 111,667 31 585,537,711 5,358 109,275 35 669,713,329 6,025 111,138 40 753,727,112 6,704 112,415 45 835,803,507 7,485 111,656 50 901,307,143 8,218 109,668 54 966,655,528 8,963 107,845 58 1,015,648,545 9,570 106,123 61 1,080,889,536 10,310 104,829 65 1,147,921,605 11,098 103,427 69 1,218,376,248 11,800 103,245 73 1,288,894,078 12,464 103,401 77 1,340,627,982 13,252 101,158 80 1,374,208,625 13,948 98,521 82 1,408,174,205 14,568 96,659 84 1,459,277,475 15,389 94,824 87 1,510,100,090 16,207 93,174 90 1,560,517,757 16,986 91,866 93 1,627,435,035 17,713 91,877 97 1,693,942,260 18,401 92,053 101 1,760,534,757 19,151 91,927 105 1,827,353,450 19,934 91,669 109 1,877,349,669 20,571 91,260 112 1,927,029,732 21,190 90,937 115 1,976,839,801 21,803 90,665 118 2,043,621,669 22,602 90,417 122 2,093,614,307 23,222 90,153 125 2,143,530,955 23,884 89,744 128 2,193,300,587 24,504 89,506 131 2,259,768,850 25,281 89,382 135 2,309,402,067 25,896 89,178 138 2,359,045,834 26,524 88,937 141 2,408,813,042 27,193 88,582 144 2,458,534,137 27,866 88,224 147 2,508,173,248 28,587 87,736 150 2,557,695,348 29,236 87,481 153 2,607,093,365 29,879 87,254 156 2,672,980,080 30,659 87,181 160 2,722,486,059 31,293 86,997 163 2,771,962,432 31,913 86,859 166 2,837,660,998 32,616 87,001 170 2,887,208,486 33,270 86,779 173 2,936,799,169 33,925 86,567 176 2,986,273,795 34,593 86,325 179 3,035,728,568 35,243 86,136 182 3,085,184,315 35,859 86,035 185 3,151,468,039 36,608 86,086 189 3,201,011,063 37,213 86,017 192 3,267,535,396 37,935 86,134 196 3,334,686,516 38,576 86,443 200 3,401,886,904 39,240 86,692 204 3,468,645,120 39,954 86,814 208 3,534,648,676 40,730 86,781 212 3,600,523,802 41,492 86,775 216 3,666,138,573 42,282 86,706 220 3,731,829,153 43,077 86,629 224 3,797,535,576 43,829 86,643 228 3,863,523,948 44,582 86,659 232 3,928,985,444 45,258 86,812 236 3,994,209,028 45,921 86,979 240 4,059,507,043 46,604 87,104 244 4,125,187,426 47,359 87,104 248 4,190,652,309 48,108 87,108 252 4,256,123,933 48,782 87,246 256 4,321,507,795 49,466 87,362 260 4,386,987,648 50,120 87,528 264 4,452,442,948 50,739 87,750 268 4,517,769,637 51,363 87,956 272 4,583,567,763 52,006 88,133 276 4,649,357,008 52,705 88,213 280 4,715,013,078 53,387 88,316 284 4,780,538,748 54,036 88,468 288 4,845,958,188 54,701 88,588 292 4,912,126,601 55,324 88,787 296 4,977,535,397 55,951 88,961 300 5,042,533,501 56,599 89,091 304 5,107,487,478 57,217 89,264 308 5,172,445,286 57,824 89,450 312 5,253,524,269 58,545 89,734 317 5,305,784,367 59,194 89,633 320 5,343,676,632 60,030 89,016 322 5,381,274,598 60,966 88,265 324 5,418,420,058 61,807 87,665 326 5,468,584,289 62,692 87,228 329 5,518,098,321 63,442 86,977 333 5,564,009,469 64,131 86,759 339 5,619,110,296 64,763 86,763 345 5,673,559,079 65,380 86,777 354 5,715,132,031 65,986 86,610 360 5,756,082,773 66,616 86,406 368 5,812,130,630 67,248 86,428 376 5,858,211,979 67,888 86,291 382 5,917,602,276 68,563 86,308 390 5,972,627,425 69,255 86,241 396 6,033,689,471 70,387 85,720 400 6,061,126,737 71,089 85,260 402 6,090,631,401 71,841 84,778 405 6,120,734,012 72,539 84,377 409 6,153,355,223 73,182 84,082 413 6,207,067,286 73,873 84,023 420 6,253,755,885 74,611 83,817 427 6,284,713,321 75,280 83,484 432 6,319,282,922 76,000 83,147 439 6,355,867,878 76,609 82,964 444 6,399,714,387 77,288 82,803 450 6,436,640,839 78,036 82,482 455 6,481,234,379 78,650 82,405 461 6,526,759,664 79,391 82,209 466 6,579,995,114 80,501 81,737 470 6,599,378,396 81,211 81,261 471 6,625,131,210 82,089 80,705 473 6,645,098,333 82,772 80,281 475 6,669,624,733 83,502 79,873 478 6,699,812,271 84,314 79,462 482 6,738,122,743 84,990 79,281 486 6,774,046,102 85,662 79,078 492 6,808,866,336 86,267 78,927 497 6,833,736,316 86,915 78,625 501 6,860,334,945 87,592 78,320 506 6,887,971,112 88,315 77,992 511 6,919,244,750 88,952 77,786 515 6,953,331,447 89,571 77,628 520 6,978,517,812 90,206 77,361 523 7,007,112,877 90,843 77,134 527 7,044,596,712 91,468 77,016 532 7,080,015,049 92,212 76,779 536 7,124,088,113 93,041 76,569 540 7,145,384,434 94,007 76,008 541 7,163,582,483 94,892 75,491 542 7,177,490,061 95,537 75,127 543 7,198,819,202 96,483 74,611 545 7,217,160,866 97,276 74,191 547 7,239,559,213 98,115 73,786 550
[Wikidata-bugs] [Maniphest] [Commented On] T164773: Error replicating wikidata blazegraph setup
Yurik added a comment. I'm re-running the import with the default settings (mmx=16gb), almost 700 gz files (127GB). This time, all of the OSM data uses integer storage instead of the original strings ("osmnode:123" treats 123 as a prefix + an int). Will know the results in 2 days.TASK DETAILhttps://phabricator.wikimedia.org/T164773EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: YurikCc: Gehel, Smalyshev, Aklapper, Yurik, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T164773: Error replicating wikidata blazegraph setup
Smalyshev added a comment. MemoryManagerClosedException by itself is not a big issue as far as I can see, it may be query timeout or something like that. That shouldn't happen on data loading though (that one should not time out).TASK DETAILhttps://phabricator.wikimedia.org/T164773EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Gehel, Smalyshev, Aklapper, Yurik, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T164773: Error replicating wikidata blazegraph setup
Smalyshev added a comment. Didn't see it happen on my side, is it reproducible?TASK DETAILhttps://phabricator.wikimedia.org/T164773EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Smalyshev, Aklapper, Yurik, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, debt, Gehel, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs