Hello all, I've been trying to download the DBpedia mappings today (with a fresh clone of the DBpedia extraction repo) and the process seemed to never finish. I found out that some of the languages' mapping xmls because huge (e.g. Mapping_ja.xml was ~100MB) and kept on growing until I killed the process. (see dump below) It seems some mappings just keep repeating inside the xml. The problematic languages are pl, ko, ja, bg, ar, sr and el.
Has anyone else seen this? Any clue as to how to solve it or bypass it? (it's either a bug in the mappings api, or in the reader, I guess) Thanks, Omri $ ../run download-mappings [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building DBpedia Core Libraries 4.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] >>> scala-maven-plugin:3.1.6:run (default-cli) @ core >>> [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (default) @ core --- [INFO] [INFO] --- maven-resources-plugin:2.3:resources (default-resources) @ core --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory dbpedia_git/core/src/main/resources [INFO] [INFO] --- scala-maven-plugin:3.1.6:compile (process-resources) @ core --- [WARNING] Expected all dependencies to require Scala version: 2.10.3 [WARNING] org.dbpedia.extraction:core:4.0-SNAPSHOT requires scala version: 2.10.3 [WARNING] org.scalatest:scalatest_2.10:1.9.1 requires scala version: 2.10.3 [WARNING] org.scala-lang:scala-actors:2.10.3 requires scala version: 2.10.3 [WARNING] net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.3 [WARNING] net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.0 [WARNING] Multiple versions of scala libraries detected! [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ core --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- scala-maven-plugin:3.1.6:compile (compile) @ core --- [WARNING] Expected all dependencies to require Scala version: 2.10.3 [WARNING] org.dbpedia.extraction:core:4.0-SNAPSHOT requires scala version: 2.10.3 [WARNING] org.scalatest:scalatest_2.10:1.9.1 requires scala version: 2.10.3 [WARNING] org.scala-lang:scala-actors:2.10.3 requires scala version: 2.10.3 [WARNING] net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.3 [WARNING] net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.0 [WARNING] Multiple versions of scala libraries detected! [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.3:testResources (default-testResources) @ core --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 25 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ core --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- scala-maven-plugin:3.1.6:testCompile (test-compile) @ core --- [WARNING] Expected all dependencies to require Scala version: 2.10.3 [WARNING] org.dbpedia.extraction:core:4.0-SNAPSHOT requires scala version: 2.10.3 [WARNING] org.scalatest:scalatest_2.10:1.9.1 requires scala version: 2.10.3 [WARNING] org.scala-lang:scala-actors:2.10.3 requires scala version: 2.10.3 [WARNING] net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.3 [WARNING] net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.0 [WARNING] Multiple versions of scala libraries detected! [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] <<< scala-maven-plugin:3.1.6:run (default-cli) @ core <<< [INFO] [INFO] --- scala-maven-plugin:3.1.6:run (default-cli) @ core --- [WARNING] Expected all dependencies to require Scala version: 2.10.3 [WARNING] org.dbpedia.extraction:core:4.0-SNAPSHOT requires scala version: 2.10.3 [WARNING] org.scalatest:scalatest_2.10:1.9.1 requires scala version: 2.10.3 [WARNING] org.scala-lang:scala-actors:2.10.3 requires scala version: 2.10.3 [WARNING] net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.3 [WARNING] net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.0 [WARNING] Multiple versions of scala libraries detected! [INFO] launcher 'download-mappings' selected => org.dbpedia.extraction.util.MappingsDownloader downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_ru.xml downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_de.xml downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_el.xml downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_et.xml downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_es.xml downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_nl.xml downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_hi.xml downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_eo.xml downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_cy.xml downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_zh.xml downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_cs.xml downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_ur.xml downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_ga.xml downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_bg.xml downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_ar.xml downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_en.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_zh.xml in 1.9705708 seconds downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_sk.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_ur.xml in 2.207869 seconds downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_bn.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_et.xml in 2.258747 seconds downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_sr.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_eo.xml in 2.9767556 seconds downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_eu.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_ru.xml in 4.1540837 seconds downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_id.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_sk.xml in 2.4665806 seconds downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_sl.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_bn.xml in 2.6779795 seconds downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_it.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_ga.xml in 5.731542 seconds downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_hi.xml in 5.953644 seconds downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_hr.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_cy.xml in 6.2881546 seconds downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_eu.xml in 3.4312966 seconds downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_ca.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_hr.xml in 2.6589563 seconds downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_ja.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_es.xml in 9.406335 seconds downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_pt.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_cs.xml in 9.530285 seconds downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_id.xml in 5.628132 seconds downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_pl.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_it.xml in 5.09654 seconds downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_ca.xml in 4.712224 seconds downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_ko.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_sl.xml in 12.70928 seconds downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_tr.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_de.xml in 17.712517 seconds downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_pt.xml in 12.278304 seconds downloading mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_hu.xml downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_hu.xml in 5.856224 seconds downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_tr.xml in 10.662249 seconds downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_nl.xml in 28.139694 seconds downloaded mappings from http://mappings.dbpedia.org/api.php to ../mappings/Mapping_en.xml in 39.06333 seconds *<here some of the files keep growing infinitely and the process doesn't stop>* ^Z <http://everything.me/> Omri Oren Algorithm Engineer, EverythingMe <http://everything.me/> [email protected] <[email protected]>
------------------------------------------------------------------------------
_______________________________________________ Dbpedia-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dbpedia-developers
