Excellent & Thanks Davies. Yep, now runs fine and takes 1/2 the time ! This was exactly why I had put in the elapsed time calculations. And thanks for the new pyspark.sql.functions.
+1 from my side for 1.5.0 RC3. Cheers <k/> On Fri, Sep 4, 2015 at 9:57 PM, Davies Liu <dav...@databricks.com> wrote: > Could you update the notebook to use builtin SQL function month and year, > instead of Python UDF? (they are introduced in 1.5). > > Once remove those two udfs, it runs successfully, also much faster. > > On Fri, Sep 4, 2015 at 2:22 PM, Krishna Sankar <ksanka...@gmail.com> > wrote: > > Yin, > > It is the > > https://github.com/xsankar/global-bd-conf/blob/master/004-Orders.ipynb. > > Cheers > > <k/> > > > > On Fri, Sep 4, 2015 at 9:58 AM, Yin Huai <yh...@databricks.com> wrote: > >> > >> Hi Krishna, > >> > >> Can you share your code to reproduce the memory allocation issue? > >> > >> Thanks, > >> > >> Yin > >> > >> On Fri, Sep 4, 2015 at 8:00 AM, Krishna Sankar <ksanka...@gmail.com> > >> wrote: > >>> > >>> Thanks Tom. Interestingly it happened between RC2 and RC3. > >>> Now my vote is +1/2 unless the memory error is known and has a > >>> workaround. > >>> > >>> Cheers > >>> <k/> > >>> > >>> > >>> On Fri, Sep 4, 2015 at 7:30 AM, Tom Graves <tgraves...@yahoo.com> > wrote: > >>>> > >>>> The upper/lower case thing is known. > >>>> https://issues.apache.org/jira/browse/SPARK-9550 > >>>> I assume it was decided to be ok and its going to be in the release > >>>> notes but Reynold or Josh can probably speak to it more. > >>>> > >>>> Tom > >>>> > >>>> > >>>> > >>>> On Thursday, September 3, 2015 10:21 PM, Krishna Sankar > >>>> <ksanka...@gmail.com> wrote: > >>>> > >>>> > >>>> +? > >>>> > >>>> 1. Compiled OSX 10.10 (Yosemite) OK Total time: 26:09 min > >>>> mvn clean package -Pyarn -Phadoop-2.6 -DskipTests > >>>> 2. Tested pyspark, mllib > >>>> 2.1. statistics (min,max,mean,Pearson,Spearman) OK > >>>> 2.2. Linear/Ridge/Laso Regression OK > >>>> 2.3. Decision Tree, Naive Bayes OK > >>>> 2.4. KMeans OK > >>>> Center And Scale OK > >>>> 2.5. RDD operations OK > >>>> State of the Union Texts - MapReduce, Filter,sortByKey (word > >>>> count) > >>>> 2.6. Recommendation (Movielens medium dataset ~1 M ratings) OK > >>>> Model evaluation/optimization (rank, numIter, lambda) with > >>>> itertools OK > >>>> 3. Scala - MLlib > >>>> 3.1. statistics (min,max,mean,Pearson,Spearman) OK > >>>> 3.2. LinearRegressionWithSGD OK > >>>> 3.3. Decision Tree OK > >>>> 3.4. KMeans OK > >>>> 3.5. Recommendation (Movielens medium dataset ~1 M ratings) OK > >>>> 3.6. saveAsParquetFile OK > >>>> 3.7. Read and verify the 4.3 save(above) - sqlContext.parquetFile, > >>>> registerTempTable, sql OK > >>>> 3.8. result = sqlContext.sql("SELECT > >>>> OrderDetails.OrderID,ShipCountry,UnitPrice,Qty,Discount FROM Orders > INNER > >>>> JOIN OrderDetails ON Orders.OrderID = OrderDetails.OrderID") OK > >>>> 4.0. Spark SQL from Python OK > >>>> 4.1. result = sqlContext.sql("SELECT * from people WHERE State = > 'WA'") > >>>> OK > >>>> 5.0. Packages > >>>> 5.1. com.databricks.spark.csv - read/write OK > >>>> (--packages com.databricks:spark-csv_2.11:1.2.0-s_2.11 didn’t work. > But > >>>> com.databricks:spark-csv_2.11:1.2.0 worked) > >>>> 6.0. DataFrames > >>>> 6.1. cast,dtypes OK > >>>> 6.2. groupBy,avg,crosstab,corr,isNull,na.drop OK > >>>> 6.3. All joins,sql,set operations,udf OK > >>>> > >>>> Two Problems: > >>>> > >>>> 1. The synthetic column names are lowercase ( i.e. now > >>>> ‘sum(OrderPrice)’; previously ‘SUM(OrderPrice)’, now ‘avg(Total)’; > >>>> previously 'AVG(Total)'). So programs that depend on the case of the > >>>> synthetic column names would fail. > >>>> 2. orders_3.groupBy("Year","Month").sum('Total').show() > >>>> fails with the error ‘java.io.IOException: Unable to acquire > 4194304 > >>>> bytes of memory’ > >>>> orders_3.groupBy("CustomerID","Year").sum('Total').show() - fails > >>>> with the same error > >>>> Is this a known bug ? > >>>> Cheers > >>>> <k/> > >>>> P.S: Sorry for the spam, forgot Reply All > >>>> > >>>> On Tue, Sep 1, 2015 at 1:41 PM, Reynold Xin <r...@databricks.com> > wrote: > >>>> > >>>> Please vote on releasing the following candidate as Apache Spark > version > >>>> 1.5.0. The vote is open until Friday, Sep 4, 2015 at 21:00 UTC and > passes if > >>>> a majority of at least 3 +1 PMC votes are cast. > >>>> > >>>> [ ] +1 Release this package as Apache Spark 1.5.0 > >>>> [ ] -1 Do not release this package because ... > >>>> > >>>> To learn more about Apache Spark, please see http://spark.apache.org/ > >>>> > >>>> > >>>> The tag to be voted on is v1.5.0-rc3: > >>>> > >>>> > https://github.com/apache/spark/commit/908e37bcc10132bb2aa7f80ae694a9df6e40f31a > >>>> > >>>> The release files, including signatures, digests, etc. can be found > at: > >>>> > http://people.apache.org/~pwendell/spark-releases/spark-1.5.0-rc3-bin/ > >>>> > >>>> Release artifacts are signed with the following key: > >>>> https://people.apache.org/keys/committer/pwendell.asc > >>>> > >>>> The staging repository for this release (published as 1.5.0-rc3) can > be > >>>> found at: > >>>> > https://repository.apache.org/content/repositories/orgapachespark-1143/ > >>>> > >>>> The staging repository for this release (published as 1.5.0) can be > >>>> found at: > >>>> > https://repository.apache.org/content/repositories/orgapachespark-1142/ > >>>> > >>>> The documentation corresponding to this release can be found at: > >>>> > http://people.apache.org/~pwendell/spark-releases/spark-1.5.0-rc3-docs/ > >>>> > >>>> > >>>> ======================================= > >>>> How can I help test this release? > >>>> ======================================= > >>>> If you are a Spark user, you can help us test this release by taking > an > >>>> existing Spark workload and running on this release candidate, then > >>>> reporting any regressions. > >>>> > >>>> > >>>> ================================================ > >>>> What justifies a -1 vote for this release? > >>>> ================================================ > >>>> This vote is happening towards the end of the 1.5 QA period, so -1 > votes > >>>> should only occur for significant regressions from 1.4. Bugs already > present > >>>> in 1.4, minor regressions, or bugs related to new features will not > block > >>>> this release. > >>>> > >>>> > >>>> =============================================================== > >>>> What should happen to JIRA tickets still targeting 1.5.0? > >>>> =============================================================== > >>>> 1. It is OK for documentation patches to target 1.5.0 and still go > into > >>>> branch-1.5, since documentations will be packaged separately from the > >>>> release. > >>>> 2. New features for non-alpha-modules should target 1.6+. > >>>> 3. Non-blocker bug fixes should target 1.5.1 or 1.6.0, or drop the > >>>> target version. > >>>> > >>>> > >>>> ================================================== > >>>> Major changes to help you focus your testing > >>>> ================================================== > >>>> > >>>> As of today, Spark 1.5 contains more than 1000 commits from 220+ > >>>> contributors. I've curated a list of important changes for 1.5. For > the > >>>> complete list, please refer to Apache JIRA changelog. > >>>> > >>>> RDD/DataFrame/SQL APIs > >>>> > >>>> - New UDAF interface > >>>> - DataFrame hints for broadcast join > >>>> - expr function for turning a SQL expression into DataFrame column > >>>> - Improved support for NaN values > >>>> - StructType now supports ordering > >>>> - TimestampType precision is reduced to 1us > >>>> - 100 new built-in expressions, including date/time, string, math > >>>> - memory and local disk only checkpointing > >>>> > >>>> DataFrame/SQL Backend Execution > >>>> > >>>> - Code generation on by default > >>>> - Improved join, aggregation, shuffle, sorting with cache friendly > >>>> algorithms and external algorithms > >>>> - Improved window function performance > >>>> - Better metrics instrumentation and reporting for DF/SQL execution > >>>> plans > >>>> > >>>> Data Sources, Hive, Hadoop, Mesos and Cluster Management > >>>> > >>>> - Dynamic allocation support in all resource managers (Mesos, YARN, > >>>> Standalone) > >>>> - Improved Mesos support (framework authentication, roles, dynamic > >>>> allocation, constraints) > >>>> - Improved YARN support (dynamic allocation with preferred locations) > >>>> - Improved Hive support (metastore partition pruning, metastore > >>>> connectivity to 0.13 to 1.2, internal Hive upgrade to 1.2) > >>>> - Support persisting data in Hive compatible format in metastore > >>>> - Support data partitioning for JSON data sources > >>>> - Parquet improvements (upgrade to 1.7, predicate pushdown, faster > >>>> metadata discovery and schema merging, support reading non-standard > legacy > >>>> Parquet files generated by other libraries) > >>>> - Faster and more robust dynamic partition insert > >>>> - DataSourceRegister interface for external data sources to specify > >>>> short names > >>>> > >>>> SparkR > >>>> > >>>> - YARN cluster mode in R > >>>> - GLMs with R formula, binomial/Gaussian families, and elastic-net > >>>> regularization > >>>> - Improved error messages > >>>> - Aliases to make DataFrame functions more R-like > >>>> > >>>> Streaming > >>>> > >>>> - Backpressure for handling bursty input streams. > >>>> - Improved Python support for streaming sources (Kafka offsets, > Kinesis, > >>>> MQTT, Flume) > >>>> - Improved Python streaming machine learning algorithms (K-Means, > linear > >>>> regression, logistic regression) > >>>> - Native reliable Kinesis stream support > >>>> - Input metadata like Kafka offsets made visible in the batch details > UI > >>>> - Better load balancing and scheduling of receivers across cluster > >>>> - Include streaming storage in web UI > >>>> > >>>> Machine Learning and Advanced Analytics > >>>> > >>>> - Feature transformers: CountVectorizer, Discrete Cosine > transformation, > >>>> MinMaxScaler, NGram, PCA, RFormula, StopWordsRemover, and > VectorSlicer. > >>>> - Estimators under pipeline APIs: naive Bayes, k-means, and isotonic > >>>> regression. > >>>> - Algorithms: multilayer perceptron classifier, PrefixSpan for > >>>> sequential pattern mining, association rule generation, 1-sample > >>>> Kolmogorov-Smirnov test. > >>>> - Improvements to existing algorithms: LDA, trees/ensembles, GMMs > >>>> - More efficient Pregel API implementation for GraphX > >>>> - Model summary for linear and logistic regression. > >>>> - Python API: distributed matrices, streaming k-means and linear > models, > >>>> LDA, power iteration clustering, etc. > >>>> - Tuning and evaluation: train-validation split and multiclass > >>>> classification evaluator. > >>>> - Documentation: document the release version of public API methods > >>>> > >>>> > >>>> > >>>> > >>>> > >>> > >> > > >
