[
https://issues.apache.org/jira/browse/SPARK-24784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-24784.
----------------------------------
Resolution: Incomplete
> Retraining (each document as separate file) creates OOME
> --------------------------------------------------------
>
> Key: SPARK-24784
> URL: https://issues.apache.org/jira/browse/SPARK-24784
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.3.1
> Environment: Spark 2.3.1
> JDK8
> SBT 1.1.6
> Ubuntu 18.04
> Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
> Reporter: Jatin Puri
> Priority: Minor
> Labels: bulk-closed
>
> This looks like a bug. The reproducible can be found at:
> [https://github.com/purijatin/spark-retrain-bug]
> Information also available at:
> [http://apache-spark-developers-list.1001551.n3.nabble.com/Retraining-with-each-document-as-separate-file-creates-OOME-td24335.html]
>
> *Issue:*
> The program takes input a set of individual files (json) and each file
> contains a single document with the same json schema as others.
> The spark program computes the tf-idf of the documents (Tokenizer ->
> Stopword remover -> stemming -> tf -> tfidf).
> Once the training is complete, it unpersists and restarts the operation again.
>
>
> When run with `-Xmx1500M` memory, it fails with OOME after about 10
> iterations.
>
> *Temporary Fix:*
> When all the input documents are merged to a single file, then the issue is
> no longer found.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
