[ https://issues.apache.org/jira/browse/SPARK-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264207#comment-15264207 ]
Saikat Kanjilal commented on SPARK-14302: ----------------------------------------- Ok I finished my initial assessment of the ml directory, here are the things that I see duplicated: 1) The initialization of the SparkContext 2) The initialization of SqlContext 3) Creation of data frames Xusen should we move the functionality of the above into a common python file and have that be referenced by all the other files. I can create a Common.py and remove all the above code from all the files and have that be referenced if needed. I wil next begin my assesment of the mllib directory > Python examples code merge and clean up > --------------------------------------- > > Key: SPARK-14302 > URL: https://issues.apache.org/jira/browse/SPARK-14302 > Project: Spark > Issue Type: Sub-task > Components: Examples > Reporter: Xusen Yin > Priority: Minor > Labels: starter > > Duplicated code that I found in python/examples/mllib and python/examples/ml: > * python/ml > ** None > * Unsure duplications, double check > ** dataframe_example.py > ** kmeans_example.py > ** simple_params_example.py > ** simple_text_classification_pipeline.py > * python/mllib > ** gaussian_mixture_model.py > ** kmeans.py > ** logistic_regression.py > * Unsure duplications, double check > ** correlations.py > ** random_rdd_generation.py > ** sampled_rdds.py > ** word2vec.py > When merging and cleaning those code, be sure not disturb the previous > example on and off blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org