Author: arnabp20
Date: Wed Oct 14 14:00:02 2020
New Revision: 41930
Log:
SystemDS 2.0 is the first major release under the new name. This release
contains a major refactoring, a few major features, a large number of
improvements and fixes, and some experimental features to better support the
end-to-end data science lifecycle. In addition to that, this release also
removes several features that are not up to the mark and outdated.
The major changes (compared to SystemML 1.2) include
• New mechanism for DML-bodied (script-level) builtin functions, and a wealth
of new built-in functions for data preprocessing including data cleaning,
augmentation and feature engineering techniques, new ML algorithms, and model
debugging.
• Several methods for data cleaning have been implemented including multiple
imputations with multivariate imputation by chained equations (MICE) and other
techniques, SMOTE, an oversampling technique for class imbalance, forward and
backward NA filling, cleaning using schema and length information, support for
outlier detection using standard deviation and inter-quartile range, and
functional dependency discovery.
• A complete framework for lineage tracing and reuse including support for
loop deduplication, full and partial reuse, compiler assisted reuse, several
new rewrites to facilitate reuse.
• New federated runtime backend including support for federated matrices and
frames, federated builtins (transform-encode, decode etc.).
• Refactor compression package and add functionalities including quantization
for lossy compression, binary cell operations, left matrix multiplication.
[experimental]
• New python bindings with supports for several builtins, matrix operations,
federated tensors and lineage traces.
• Cuda implementation of cumulative aggregate operators (cumsum, cumprod etc.)
• New model debugging technique with slice finder.
• New tensor data model (basic tensors of different value types, data tensors
with schema) [experimental]
• Cloud deployment scripts for AWS and scripts to set up and start federated
operations.
• Performance improvements with parallel sort, gpu cum agg, append cbind etc.
• Various compiler and runtime improvements including new and improved
rewrites, reduced Spark context creation, new eval framework, list operations,
updated native kernel libraries to name a few.
• New data reader/writer for json frames and support for sql as a data source.
• Miscellaneous improvements: improved documentation, better testing,
run/release scripts, improved packaging, Docker container for systemds, support
for lambda expressions, bug fixes.
• Removed MapReduce compiler and runtime backend, pydml parser, Java-UDF
framework, script-level debugger.
• Deprecated ./scripts/algorithms, as those algorithms will be part of
SystemDS builtins.
Added:
release/systemds/2.0.0/
release/systemds/2.0.0/systemds-2.0.0-bin.tgz (with props)
release/systemds/2.0.0/systemds-2.0.0-bin.tgz.asc
release/systemds/2.0.0/systemds-2.0.0-bin.tgz.sha512
release/systemds/2.0.0/systemds-2.0.0-bin.zip (with props)
release/systemds/2.0.0/systemds-2.0.0-bin.zip.asc
release/systemds/2.0.0/systemds-2.0.0-bin.zip.sha512
release/systemds/2.0.0/systemds-2.0.0-src.tgz (with props)
release/systemds/2.0.0/systemds-2.0.0-src.tgz.asc
release/systemds/2.0.0/systemds-2.0.0-src.tgz.sha512
release/systemds/2.0.0/systemds-2.0.0-src.zip (with props)
release/systemds/2.0.0/systemds-2.0.0-src.zip.asc
release/systemds/2.0.0/systemds-2.0.0-src.zip.sha512
Modified:
release/systemds/KEYS
Added: release/systemds/2.0.0/systemds-2.0.0-bin.tgz
==============================================================================
Binary file - no diff available.
Propchange: release/systemds/2.0.0/systemds-2.0.0-bin.tgz
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: release/systemds/2.0.0/systemds-2.0.0-bin.tgz.asc
==============================================================================
--- release/systemds/2.0.0/systemds-2.0.0-bin.tgz.asc (added)
+++ release/systemds/2.0.0/systemds-2.0.0-bin.tgz.asc Wed Oct 14 14:00:02 2020
@@ -0,0 +1,16 @@
+-----BEGIN PGP SIGNATURE-----
+
+iQIzBAABCAAdFiEEQ1mvX2j9HbS19F/lovLBSoEVKNEFAl97PRMACgkQovLBSoEV
+KNGhhhAAvygOjwjnKVkYh7wzuNPYaPlW3HM4NYSIFfpkQ2UU3pDCUyAO73jgFD4z
+95WjHgrId76EK6QYAYmLqMAIR/9Cud1vtRfabQcUTeHovTw93I7skV4ANQJqmaat
+zG4sBGP0n2Agy22P9b4j6v1CJLadia5CWrDKEM7Nm3MjEELvZi7OuvnOAE8Q35Rq
+6bVtiUp2XuFn8uB3hsyLDDDfnm9K8Zk2Q1+g0O09ilVqeidbCUYGJSWXfpcOoY+R
+ARO4P1eKT5fxze01FaV7f3vbsCb6uoJuIJFfwXLQXxLLjOT6bnsC1I1eEgDlnF6M
+G7DiR7rusqFmPwgk95SqCTjQuDy4yo5YzbrpbqdlrqsruwkTkxqSj0hlazJSWrps
+bmdfQ+WfTbHAHVXnwefcKyiXTz77MSb3FCMxRoLRmnEXDnkOJFa7MkbP1a6TFtJI
+n0clViTKE0esWSiLUixoWsrm8sL5UqBf3LqYy8O0tg1CjaRPJEP/yty1a9UaPXkU
+nQtIx/2RcYOnBbWA2ZzHJhSvG0BrBl2CaKYwpyC/nSxr9ezrde8KLhLxZoV34DaP
+fJssc0F8cEqCzyLMNw5PB7Ig88J0DhungWAw4vqJxDN/F8F+AxpkAPQkyZ1y6B+f
+OwJkr4ofmxcf2a18v5b9MCD3VR8QYylB73Kis/GPZ1p+7E7k1fg=
+=LW+p
+-----END PGP SIGNATURE-----
Added: release/systemds/2.0.0/systemds-2.0.0-bin.tgz.sha512
==============================================================================
--- release/systemds/2.0.0/systemds-2.0.0-bin.tgz.sha512 (added)
+++ release/systemds/2.0.0/systemds-2.0.0-bin.tgz.sha512 Wed Oct 14 14:00:02
2020
@@ -0,0 +1 @@
+efddc477beda4ede144f1758e13721ea936f1eef3d107f8d7359474da5a6a66f729fc017ff0b981c879616f7c8d2cf4f43b965a4678de87825861938425468f9
systemds-2.0.0-bin.tgz
Added: release/systemds/2.0.0/systemds-2.0.0-bin.zip
==============================================================================
Binary file - no diff available.
Propchange: release/systemds/2.0.0/systemds-2.0.0-bin.zip
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: release/systemds/2.0.0/systemds-2.0.0-bin.zip.asc
==============================================================================
--- release/systemds/2.0.0/systemds-2.0.0-bin.zip.asc (added)
+++ release/systemds/2.0.0/systemds-2.0.0-bin.zip.asc Wed Oct 14 14:00:02 2020
@@ -0,0 +1,16 @@
+-----BEGIN PGP SIGNATURE-----
+
+iQIzBAABCAAdFiEEQ1mvX2j9HbS19F/lovLBSoEVKNEFAl97PRIACgkQovLBSoEV
+KNEFeg//URgZwGHETFYDJinCf75ho+zBJ959O519ChDRhS4Z3jRK30Z5lJk2oQ5r
+yAN3AzoGpONkBDA2/eFr933U9kzy/xVMWF2QtLs/VhRGuyimp36wW2sl/M8BsO7q
+GgKkZmi2dvmpgxhbnhaJkkLT7t3aV1rrw/K02/zDxTYRbQocBnqkHqHRFRlLFYTc
+5drk9SMdutKlbbGsajJmjf8gdiglNscz//QHL6hRUM0k+oNBWodaz9Tj5+bDr0n5
+IIRUV3xU9nIRjr5QCvYzzmYxJjwevuhrJbUdPL/mLp7lZGlDuXktY1QJRZHgmBwF
+/LgNbMsY8A2VLgUoItm+jfEF2QH0MAmkV5LJJW7i0oIzdAPwJ+fenXyx5kvaEbZy
+U4Hib9gPF5cwTOG+ArOFCPeWdy7t5NsLlEqCtnEFfcBM/Iqy2UYpXod0Z5jaSTEM
+O5SYBJ/8/EMk6o6JWwcVt7VgEQI4B6wYR8FU2giFv86AsiYkTtYJorHKDBG7nj82
+Z15uiTj988G8H9P3gAR9f0eCjULdL0OJ5i/bA77w7URPqIJLODP9UBES5rE2HEC1
+1n0ivmkW9A77sG7NZEE29fsURu1KocVJDJ742wwga9k6fbwWHHcWH4Yr6gfNfIpP
+gXOj2PFuJO3P/cBghkJ8c6PeICOHFNFmft8MVWFHTLTNRxM3Njc=
+=fs8b
+-----END PGP SIGNATURE-----
Added: release/systemds/2.0.0/systemds-2.0.0-bin.zip.sha512
==============================================================================
--- release/systemds/2.0.0/systemds-2.0.0-bin.zip.sha512 (added)
+++ release/systemds/2.0.0/systemds-2.0.0-bin.zip.sha512 Wed Oct 14 14:00:02
2020
@@ -0,0 +1 @@
+5bbda2b5217c266642fa2f4e3ef31da5cbc46dfd9797e8ce0fa6837182b562e057be8d9a0d616d46e54b84e9e2679e6bfda9a5152e40ab65bfc29b6eb71f58d9
systemds-2.0.0-bin.zip
Added: release/systemds/2.0.0/systemds-2.0.0-src.tgz
==============================================================================
Binary file - no diff available.
Propchange: release/systemds/2.0.0/systemds-2.0.0-src.tgz
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: release/systemds/2.0.0/systemds-2.0.0-src.tgz.asc
==============================================================================
--- release/systemds/2.0.0/systemds-2.0.0-src.tgz.asc (added)
+++ release/systemds/2.0.0/systemds-2.0.0-src.tgz.asc Wed Oct 14 14:00:02 2020
@@ -0,0 +1,16 @@
+-----BEGIN PGP SIGNATURE-----
+
+iQIzBAABCAAdFiEEQ1mvX2j9HbS19F/lovLBSoEVKNEFAl97PRMACgkQovLBSoEV
+KNFQGxAAgKQS6E7mfy4Haeknllr4CxjTadIYAak/ghKIDaO3WOawLZPHMFha1Q79
+V9Ghg/ASiOJbt2rIk7yYNLIDU/s95vRUfvpruXuHiRkPzig9Dr1dlY8YgH/kFd9G
+1FXyU65fVYlr+hraG1IcUFi8ojibx9ypTdml+blkPfKJG35XOAWwFsN1A6TCLeEe
+g6JhlDlNRDx1uC2fjf+hNGLm65YFGQ83Z+WMMZNPeXS+z9gYj6HeeLxUZYSetX/3
+H63lRb68BoRTp468SzhgAqMt+XwIrJAP5nLPowzXRNlwY+vc++16zQCIjAoG4o/Z
+rAigwi/39gBa2Ip4bFyhV0U3ZoHD3SFHXVc3t4A2mY6LuTPYK4syjxUJ9QoVUrqE
+CfpI5CcW0kOVFdywSuq3VOkC5uD7pTiSSOaMUzEx10Qelnp97qcqmTCrJ6tak9xr
+3vLbArGppubgl8namTus04DKD+EbTOOKVcJ4mpZKYNCtHbUXSDWNGcDWVMLTqTaJ
+xjhUtM0Ojmkz0hXTISQgy75c8n0TCtd/YJvhyVxeDUiqa+hZTTacXiKGrJZfkX/6
+E/czJKk4ilEsDB7qCMNk/I0R2+yk7Re9M6u6BaIeF/f0969x/jrjl3MzaWA2T7WB
+/J6eKEJHky9O4fYMvpD7DH6CxHkhyVE+gpmBXbfrPq+CzKH1XRY=
+=/NNJ
+-----END PGP SIGNATURE-----
Added: release/systemds/2.0.0/systemds-2.0.0-src.tgz.sha512
==============================================================================
--- release/systemds/2.0.0/systemds-2.0.0-src.tgz.sha512 (added)
+++ release/systemds/2.0.0/systemds-2.0.0-src.tgz.sha512 Wed Oct 14 14:00:02
2020
@@ -0,0 +1 @@
+a5161d68bfe16569d2fe931e04328b00a47c81a7e61411263eebb01b8113291f2a69266bcb14f5eeb273095e1124d65106def5a7f3dee00ed49b2618b2358b07
systemds-2.0.0-src.tgz
Added: release/systemds/2.0.0/systemds-2.0.0-src.zip
==============================================================================
Binary file - no diff available.
Propchange: release/systemds/2.0.0/systemds-2.0.0-src.zip
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: release/systemds/2.0.0/systemds-2.0.0-src.zip.asc
==============================================================================
--- release/systemds/2.0.0/systemds-2.0.0-src.zip.asc (added)
+++ release/systemds/2.0.0/systemds-2.0.0-src.zip.asc Wed Oct 14 14:00:02 2020
@@ -0,0 +1,16 @@
+-----BEGIN PGP SIGNATURE-----
+
+iQIzBAABCAAdFiEEQ1mvX2j9HbS19F/lovLBSoEVKNEFAl97PRIACgkQovLBSoEV
+KNELxA//YD2TRZ30851I9P1OwqEOCxBq4vmbWds9bLfi5Ry6GgJEFjfImxFJRhQE
+W/2r7XbvES/xRTstx94JEYMaWl0nBOJkbkhYVxyH7l9qsuUgCMxYCZn+mxRrMdBb
+waK9nV8We4OXCdy2lnnXEdSmXU9lgKzvfLiJf912sQaAvi6ZPksPISs8EdMrA065
+HhOv47YA2MCC1PFf6ryuydV9YDwhmAQPiPXSHITHvjr4drBBC1dcDhRywRWgIuxQ
+idd5JTvIaxqsMKAyqkkqPMO5SIBlwkDleQo3gYJzl2vy9pzMCvYwbnuLJnPN4OSE
+WlNIfN1PUID/VbPcKkQispmFfJ9g2dYvqGJ/Oq1tfNxbKNf7fctEKF8cQhL9PWIc
+aKJ1sC3bete4yEc3Wi7sV6KAv+21Drno/JQ99s28DkMiGuQih7aH+WlXw3eRU7WX
+pjgoaObu6VWiUH9Jp7+yyQzMjKiKdUkOQdZhqLgKw0L8iB2SoR5Is7lTFIcxdyqA
+VeFx6BXQchHlDldkonTDq/gQsb4CMAWI0HLYCFTP1cHO0NoTJcXI7jlNOtXilJGQ
+r2oDkvDY2bMzj+Hi7lyM2zMPNbPzx+TYEsmFn35i05eMlI6hHtm3C5yUij4tqNBd
+m34Il3TtLSTYWBGUt8R6oHDIFg+2dRUkENy9PUY6KJmDM4l9WKE=
+=3YZv
+-----END PGP SIGNATURE-----
Added: release/systemds/2.0.0/systemds-2.0.0-src.zip.sha512
==============================================================================
--- release/systemds/2.0.0/systemds-2.0.0-src.zip.sha512 (added)
+++ release/systemds/2.0.0/systemds-2.0.0-src.zip.sha512 Wed Oct 14 14:00:02
2020
@@ -0,0 +1 @@
+d622a435175eeeeff91707544f874724cbe875da3b41abfe9ddb986665a884869d87740134602d3a11fcbc0885ed387f5becc9e61f374687fced2f28a504c036
systemds-2.0.0-src.zip
Modified: release/systemds/KEYS
==============================================================================
--- release/systemds/KEYS (original)
+++ release/systemds/KEYS Wed Oct 14 14:00:02 2020
@@ -1,4 +1,4 @@
-This file contains the GPG keys of Apache SystemML developers.
+This file contains the GPG keys of Apache SystemDS developers.
Users: gpg --import KEYS
@@ -24,9 +24,9 @@ Developers:
********************************* PLEASE NOTE
**********************************
Releases will be signed using one of these keys in this file. This file will
- be available with the distributed Apache SystemML releases at:
+ be available with the distributed Apache SystemDS releases at:
- https://dist.apache.org/repos/dist/release/incubator/systemml/KEYS
+ https://dist.apache.org/repos/dist/release/systemds/KEYS
********************************************************************************
@@ -314,3 +314,61 @@ x3ZWR4EAmduC/pVZoohJBBgRAgAJBQJGhczNAhsM
12SHPDD1EhclKffJ68Y55NGMAJkBNGqPHX3+fJ+QIDhBgfm4YjvhEQ==
=Zgm1
-----END PGP PUBLIC KEY BLOCK-----
+pub rsa4096 2020-08-31 [SC]
+ 4359AF5F68FD1DB4B5F45FE5A2F2C14A811528D1
+uid [ultimate] Arnab Phani <[email protected]>
+sig 3 A2F2C14A811528D1 2020-08-31 Arnab Phani <[email protected]>
+sub rsa4096 2020-08-31 [E]
+sig A2F2C14A811528D1 2020-08-31 Arnab Phani <[email protected]>
+
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQINBF9NXSwBEADHiooKh4xWeDVllUfaWlW5HjLsFf7/bWGpQxuFrhM3LqXB6vO7
+XiK5TfZxmfCN3XxvmIE3FIeuyC9WLI9LPhdbvA7P+SHNRa2511iy58yvqMAZ+XFU
+9BsnfpuUvtkwSTaLMtxXx3Ota9k10yb/nKKBWmQfxPWyRhfhxMgOGgkzP+nQ5x0Q
+mtN0CMreb+fyM2WYVnv41c5t0Mq80IF5tUbBVTPZTjz1QJXey9cnm0tmdZ4ETex4
+BeVe3FM4rb8/cgm7pcPW+MaMC3D0ZDZG44XBCJk/9oDRV5QPhqLZd+/7tco4e+r2
+2iXll/rgk/WWP9Mx+tZNq2MVa0Pc+jDKPOfRhMntIkgKhhuVEEARIgUJTS2/7Rxe
+QNOXsyojp5kkCxMF7h8gAEibM4OnXXL9tn/icyUJLm2HNl0M4AmC0SxtCFtCpFgx
+aRgfxJdCBnNOSbeZ98lFPp49NirDuH9lBdOJEMzSSBG7Gc5Cb5iSWdc/jYAUYT2X
+a6x4sE2CbZloNF+mSoKVo6unAiEcnP4d4KthYKR3p0vOYOu7Mwj6f+aHu/afr01D
+tcKQSc5qH0TSR2RgHrEiSpgtw3hTZWGV9tsoF+oinTmXBG1k3N0pC1yyl9Rbj287
+3Bwe29CrnDqBtBdlhAtx7XfXEUtZD8fBprWAERZJNWWVvMqrG5lR5ffN6wARAQAB
+tCJBcm5hYiBQaGFuaSA8cGhhbmlhcm5hYkBnbWFpbC5jb20+iQJOBBMBCAA4FiEE
+Q1mvX2j9HbS19F/lovLBSoEVKNEFAl9NXSwCGwMFCwkIBwIGFQoJCAsCBBYCAwEC
+HgECF4AACgkQovLBSoEVKNFozA//TjSnTV5WfB4KrMc3oM2OZcyzBlrG3PCAH5KL
+CvuZan5Rx3iRDtwH7VNxIfrDNXoLN0kn1YizYagII+sXKcohmNoc0nsYbFh4YijG
+RqvigQpXM8V/UDd38LQnpQKaJhZ127duB/ZjV8BQwJvQUbNWUDOqs/hZCy2W+5mC
+Ezej90PYDBumFIPk5HZYnE8tZe6XoRAYlQVuUSSAtSJPZvGrSmwAzFlU/0Zmifi5
+9dNzJkM7Wx37ySbrtXYlemnoAvtSmG0Rv7qUKH2eXs/G/Yha8srhwIw0cbIlGdSP
+BgBJ/qKj+FohWh8A6sNzC3IAyncGcdpaDEXfXGmOjl54vF6pA9y8OaMN06kaHVSD
+KAVEjc6wsFHcQKw0Bgy/73takyyNVR7NcyZ8F7urmkKKnH3tAtjhUSO9l3slYjRl
+Nl0vKbarAVX3iYRwo1D8qBMU7GDfvDRYlKSrfAnCoWiUSGUaxUhTMe9hTWNvmYCl
+CgO0U5aPaoUIqu9sgJBEz0EEDuQMG9QDJyE1dl3JBbfSsa3EJsZ7au7yy/Wf6eJF
+SUTiB0VZPRQnlIOK8xYmgfZHBJxqqAee3zXKWuiDKDeciLeAC6n2wGzAJTaG5ZR6
+VUTaN3gEx3YXcvNvgPC28CvAMiKJBNiMDIXd3BXxPqd1oHzHutL2NI5IPaqbRnB4
+QBrLBrO5Ag0EX01dLAEQAM3PDWWOhiB0yivFVMSR6I5qOuGiLGWPU10GqDaXFPRY
+718zBdQbSiXPtet06pPx8T3EDLeX4brXekWvayn3PctVashbkhL7BH/kCPqsMYG6
+gt3T4XtGuUVfn3Hsr8/i+SvJV7r/+oQ8pi04WdcLNLElEpYfcT9//8kShiOmeOIJ
+kHwdSocY7DS2cgBOk+Cvcb5GYNroO+PjkB8v7WSnfo59F+BD6GR55s/O82VZnuXj
+3cXl4J3p6IZWR0O7BtWb1zMDICzHN7URVSs80TH07KExRJwu6snUmOmzhtgo3R4L
+NqYIM1+16vIEfP55N39labgwIUDFzHu0rDKmNg5XcZ7DHtXuZCqx+lwP9l1csTIg
+cBbA5HIUM0ODcODICc7eauFNfVawRmDqscL3ThCu8/trEA5DwFrOPM3K7MFYIQKI
+CfLfuM6McP2swZwTJEjpscO4bhfyGje2O7Npxa8Xhp2xhkeRqb7kdqqgv9cB/dR4
+t49zDrr1pKumggfyQ3g8dRxKBqlKRdu/vvrWv0I+YBGsJN4Y6Xs6h2WcGiITJg/4
+h5ELi/WYsVcVu+rSBFAFHDA2mNUEMFNp2UifmWc7TbzQCnSkh/ZFzxL5qr3Dx8nV
+R2hnEaVYN5KvEJP9Iegs/2BderdaSAZHyK15Ff6ow5hyVpzY4Pg3MwTeS3MbwUZ9
+ABEBAAGJAjYEGAEIACAWIQRDWa9faP0dtLX0X+Wi8sFKgRUo0QUCX01dLAIbDAAK
+CRCi8sFKgRUo0f2hD/0Xb32eMrRP1pX0pfcmhN39RHwyIsb89Fo9VqSPiRqOFbKA
+XNHkVUTH76Kf79GCN0DPq0uckV+W3KqR6oI/7e/YCcV8cybTiCRywYT+BlavGzjK
+HcysZqLC8aa+TDjHKO7zSKkiOG5+IeTf4pZETls1x1LMzWDxTPxFCZdiHystfz9E
+OQrrtIH/SlNlv/Q4e2sJMEVAWlLwr/UqS+eStOOqgv0zYOwcJVkDKenYfKrfTvYP
+EKbJFbC7DGnx5NhQKKWFJhmvyDnlM+l6BmsUeHdFiBC3+ygiLrXCyWHnSJNs+77c
+UyaWhM+LF4FSXTrej/SXyKHrfSSRyNfqTItQU4vZWCfrrMc95ZgnuTtF1x8rr/IG
+FL7hOBy1NWt5XIw3gDBVXVTqRMxeuluarxu685+7zlhyP0N56Qru/JlxDDwtpX1X
+FaNxOucAE7AohSDvi2+g8tHnRhV94nDzyOekKwa9uZHqrmRSrH6hcG92clxU8GFQ
+sWyyfpmlK2hXp/2+ks4jO64QwgpF7/wUhy7V5kXxZJjRavllhv86kZ/iCg7BY0qE
+ZWqNLIXvhY5rhR4+KuuKKEapLVCQelBGZdEH4+uySJrst0toQUbc85y9ToyAlK87
+l0DGQgF+WAtjNcMNYDHjrorlrbOYozas/3B2WZ95BoiDyuSvH3U8Lm8zs1WmNg==
+=/IHI
+-----END PGP PUBLIC KEY BLOCK-----
