Hello all,

I've been trying to download the DBpedia mappings today (with a fresh clone
of the DBpedia extraction repo) and the process seemed to never finish.
I found out that some of the languages' mapping xmls because huge (e.g.
Mapping_ja.xml was ~100MB) and kept on growing until I killed the process.
(see dump below)
It seems some mappings just keep repeating inside the xml.
The problematic languages are pl, ko, ja, bg, ar, sr and el.

Has anyone else seen this?
Any clue as to how to solve it or bypass it? (it's either a bug in the
mappings api, or in the reader, I guess)

Thanks,
Omri





$ ../run download-mappings
[INFO] Scanning for projects...
[INFO]

[INFO]
------------------------------------------------------------------------
[INFO] Building DBpedia Core Libraries 4.0-SNAPSHOT
[INFO]
------------------------------------------------------------------------
[INFO]
[INFO] >>> scala-maven-plugin:3.1.6:run (default-cli) @ core >>>
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (default) @ core ---
[INFO]
[INFO] --- maven-resources-plugin:2.3:resources (default-resources) @ core
---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
dbpedia_git/core/src/main/resources
[INFO]
[INFO] --- scala-maven-plugin:3.1.6:compile (process-resources) @ core ---
[WARNING]  Expected all dependencies to require Scala version: 2.10.3
[WARNING]  org.dbpedia.extraction:core:4.0-SNAPSHOT requires scala version:
2.10.3
[WARNING]  org.scalatest:scalatest_2.10:1.9.1 requires scala version: 2.10.3
[WARNING]  org.scala-lang:scala-actors:2.10.3 requires scala version: 2.10.3
[WARNING]  net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.3
[WARNING]  net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.0
[WARNING] Multiple versions of scala libraries detected!
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ core ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- scala-maven-plugin:3.1.6:compile (compile) @ core ---
[WARNING]  Expected all dependencies to require Scala version: 2.10.3
[WARNING]  org.dbpedia.extraction:core:4.0-SNAPSHOT requires scala version:
2.10.3
[WARNING]  org.scalatest:scalatest_2.10:1.9.1 requires scala version: 2.10.3
[WARNING]  org.scala-lang:scala-actors:2.10.3 requires scala version: 2.10.3
[WARNING]  net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.3
[WARNING]  net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.0
[WARNING] Multiple versions of scala libraries detected!
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.3:testResources (default-testResources)
@ core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 25 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
core ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- scala-maven-plugin:3.1.6:testCompile (test-compile) @ core ---
[WARNING]  Expected all dependencies to require Scala version: 2.10.3
[WARNING]  org.dbpedia.extraction:core:4.0-SNAPSHOT requires scala version:
2.10.3
[WARNING]  org.scalatest:scalatest_2.10:1.9.1 requires scala version: 2.10.3
[WARNING]  org.scala-lang:scala-actors:2.10.3 requires scala version: 2.10.3
[WARNING]  net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.3
[WARNING]  net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.0
[WARNING] Multiple versions of scala libraries detected!
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] <<< scala-maven-plugin:3.1.6:run (default-cli) @ core <<<
[INFO]
[INFO] --- scala-maven-plugin:3.1.6:run (default-cli) @ core ---
[WARNING]  Expected all dependencies to require Scala version: 2.10.3
[WARNING]  org.dbpedia.extraction:core:4.0-SNAPSHOT requires scala version:
2.10.3
[WARNING]  org.scalatest:scalatest_2.10:1.9.1 requires scala version: 2.10.3
[WARNING]  org.scala-lang:scala-actors:2.10.3 requires scala version: 2.10.3
[WARNING]  net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.3
[WARNING]  net.liftweb:lift-json_2.10:2.5.1 requires scala version: 2.10.0
[WARNING] Multiple versions of scala libraries detected!
[INFO] launcher 'download-mappings' selected =>
org.dbpedia.extraction.util.MappingsDownloader
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_ru.xml
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_de.xml
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_el.xml
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_et.xml
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_es.xml
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_nl.xml
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_hi.xml
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_eo.xml
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_cy.xml
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_zh.xml
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_cs.xml
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_ur.xml
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_ga.xml
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_bg.xml
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_ar.xml
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_en.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_zh.xml in 1.9705708 seconds
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_sk.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_ur.xml in 2.207869 seconds
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_bn.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_et.xml in 2.258747 seconds
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_sr.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_eo.xml in 2.9767556 seconds
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_eu.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_ru.xml in 4.1540837 seconds
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_id.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_sk.xml in 2.4665806 seconds
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_sl.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_bn.xml in 2.6779795 seconds
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_it.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_ga.xml in 5.731542 seconds
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_hi.xml in 5.953644 seconds
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_hr.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_cy.xml in 6.2881546 seconds
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_eu.xml in 3.4312966 seconds
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_ca.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_hr.xml in 2.6589563 seconds
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_ja.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_es.xml in 9.406335 seconds
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_pt.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_cs.xml in 9.530285 seconds
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_id.xml in 5.628132 seconds
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_pl.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_it.xml in 5.09654 seconds
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_ca.xml in 4.712224 seconds
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_ko.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_sl.xml in 12.70928 seconds
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_tr.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_de.xml in 17.712517 seconds
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_pt.xml in 12.278304 seconds
downloading mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_hu.xml
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_hu.xml in 5.856224 seconds
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_tr.xml in 10.662249 seconds
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_nl.xml in 28.139694 seconds
downloaded mappings from http://mappings.dbpedia.org/api.php to
../mappings/Mapping_en.xml in 39.06333 seconds
*<here some of the files keep growing infinitely and the process doesn't
stop>*
^Z



 <http://everything.me/>
Omri Oren
Algorithm Engineer, EverythingMe <http://everything.me/>
 [email protected] <[email protected]>
------------------------------------------------------------------------------
_______________________________________________
Dbpedia-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dbpedia-developers

Reply via email to