This is an automated email from the ASF dual-hosted git repository.
mboehm7 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/master by this push:
new ae97466 [SYSTEMDS-263] Move Python ONNX API and tests to staging
ae97466 is described below
commit ae97466b734efe90503ee96677cbd4155887eac1
Author: Matthias Boehm <[email protected]>
AuthorDate: Sun Nov 8 19:04:06 2020 +0100
[SYSTEMDS-263] Move Python ONNX API and tests to staging
Recent changes of the onnx dependency are apparently
backwards-incompatible. As this API is an experimental feature, we now
move the related code and tests back to staging. Once we come to, we
need to think about fixing the version for test stability.
---
.github/workflows/python.yml | 4 +-
dev/Tasks-obsolete.txt | 371 ---------------------
.../staging/onnx}/onnx_systemds/README.md | 0
.../staging/onnx}/onnx_systemds/__init__.py | 0
.../staging/onnx}/onnx_systemds/convert.py | 0
.../staging/onnx}/onnx_systemds/onnx_helper.py | 0
.../staging/onnx}/onnx_systemds/operator_gen.py | 0
.../staging/onnx}/onnx_systemds/render.py | 0
.../templates/graph_function.dml.jinja | 0
.../onnx_systemds/templates/graph_header.dml.jinja | 0
.../onnx}/onnx_systemds/templates/main.dml.jinja | 0
.../templates/matrix_initialize.dml.jinja | 0
.../onnx_systemds/templates/model_header.dml.jinja | 0
.../templates/module_import.dml.jinja | 0
.../operators/2input_1output_operator.dml.jinja | 0
.../templates/operators/function_call.dml.jinja | 0
.../templates/operators/if_operator.dml.jinja | 0
.../templates/operators/neg.dml.jinja | 0
.../onnx}/onnx_systemds/templates/util.dml.jinja | 0
.../staging/onnx}/onnx_systemds/util.py | 0
.../onnx/onnx_systemds_docs}/onnx_systemds.rst | 0
.../onnx_systemds_docs}/onnx_systemds_design.rst | 0
.../staging/onnx/onnx_systemds_tests}/__init__.py | 0
.../dml_wrapper/simple_conv_layer_2_wrapper.dml | 0
.../dml_wrapper/simple_conv_layer_wrapper.dml | 0
.../dml_wrapper/simple_dropout_layer_wrapper.dml | 0
.../dml_wrapper/simple_if_graph_wrapper.dml | 0
.../dml_wrapper/simple_mat_add_mul_sub_wrapper.dml | 0
.../dml_wrapper/simple_mat_add_wrapper.dml | 0
.../dml_wrapper/simple_mat_initialized_wrapper.dml | 0
.../dml_wrapper/simple_maxpool_layer_wrapper.dml | 0
.../simple_relu_tanh_sigmoid_softmax_wrapper.dml | 0
.../simple_conv_layer_2_reference.out | 0
.../simple_conv_layer_reference.out | 0
.../output_reference/simple_if_graph_reference.out | 0
.../simple_mat_add_mul_sub_reference.out | 0
.../output_reference/simple_mat_add_reference.out | 0
.../simple_mat_initialized_reference.out | 0
.../simple_maxpool_layer_reference.out | 0
.../simple_relu_tanh_sigmoid_softmax_reference.out | 0
.../test_models/model_generate.py | 0
.../onnx/onnx_systemds_tests}/test_simple.py | 0
.../staging/onnx/onnx_systemds_tests}/util.py | 0
src/main/python/docs/requires-docs.txt | 2 -
.../python/docs/source/getting_started/install.rst | 2 +-
src/main/python/setup.py | 2 -
src/main/python/systemds/__init__.py | 3 +-
src/test/java/org/apache/sysds/test/TestUtils.java | 2 -
48 files changed, 3 insertions(+), 383 deletions(-)
diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml
index 264b9c9..9947135 100644
--- a/.github/workflows/python.yml
+++ b/.github/workflows/python.yml
@@ -85,7 +85,7 @@ jobs:
run: sudo apt-get install protobuf-compiler libprotoc-dev
- name: Install pip Dependencies
- run: pip install numpy py4j wheel scipy sklearn jinja2 onnx requests
+ run: pip install numpy py4j wheel scipy sklearn requests
- name: Build Python Package
run: |
@@ -98,13 +98,11 @@ jobs:
export PATH=$SYSTEMDS_ROOT/bin:$PATH
export SYSDS_QUIET=1
cd src/main/python
- python tests/onnx_systemds/test_models/model_generate.py
python -m unittest discover -s tests -p 'test_*.py'
echo "Exit Status: " $?
- name: Run all python tests no environment
run: |
cd src/main/python
- python tests/onnx_systemds/test_models/model_generate.py
python -m unittest discover -s tests -p 'test_*.py'
echo "Exit Status: " $?
diff --git a/dev/Tasks-obsolete.txt b/dev/Tasks-obsolete.txt
deleted file mode 100644
index 9f157fb..0000000
--- a/dev/Tasks-obsolete.txt
+++ /dev/null
@@ -1,371 +0,0 @@
-<!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% end comment %}
--->
-
----
-GENERAL NOTES:
- * Assign IDs for epics by default with a key range of 10
- * Keep epic tasks in the key range, otherwise create new epic
----
-
-SYSTEMDS-10 Compiler Rework / Misc
- * 11 Support DML-bodied builtin functions OK
- * 12 Remove unnecessary HOP/LOP indirections OK
- * 13 Refactoring test cases into component/integration OK
- * 14 Complete removal of external functions from all scripts
- * 15 Travis integration w/ subset of tests OK
(removed for Github Actions)
- * 16 Remove instruction patching
- * 17 Refactoring of program block hierarchy OK
- * 18 Improve API for new dml-bodied builtin functions OK
- * 19 Cleanup pass cpvar+rmvar to mvvar instructions OK
-
-SYSTEMDS-20 New Data Model
- * 21 Finalize dense tensor blocks OK
- * 22 Sparse double/float tensor blocks
- * 23 Sparse int/bool tensor blocks
- * 24 Initial tensor dml integration OK
- * 25 Initial data tensor implementation
- * 26 Non-zero default value for sparse (row/col)
- * 27 Tensor buffer pool integration
- * 28 Tensor local readers and writers (textcell, binary) OK
- * 29 Tensor local readers and writers (linearize csv)
-
-SYSTEMDS-30 Builtin and Packaging
- * 31 Shell script for local runs OK
- * 32 Shell script for spark runs OK
- * 33 Cleanup hadoop dependency for local runs
- * 34 Wrapper blocks for sequence files
- * 35 Replace unnecessary dependencies w/ custom
- * 36 Shell script for AWS execution OK
- * 37 Cleanup mvn package, compile, test OK
- * 38 Runscript rename, improve, fix cygwin OK
-
-SYSTEMDS-40 Preprocessing builtins
- * 41 Add new winsorize builtin function OK
- * 42 Add new outlier builtin function OK
- * 43 Add new scale builtin function OK
- * 44 SotA normalization primitives
- * 45 SotA outlier detection primitives
- * 46 Generalization of quantiles
- * 47 Add new normalize builtin function OK
- * 48 Extended transform encode/apply: feature hashing OK
-
-SYSTEMDS-50 I/O Formats
- * 51 Support for homogeneous JSON (local/distributed)
- * 52 Support for libsvm files (local/distributed)
- * 53 New sql data source (local, distributed)
- * 54 Support for is.na, is.nan, is.infinite OK
- * 55 Cleanup file formats, binary cell, input/output infos OK
- * 56 Consolidate single/multi-threaded readers
- * 57 Support for hdf5 files (local/distributed)
-
-SYSTEMDS-60 Update SystemML improvements
- * 61 Take over cumulative aggregate improvements OK
- * 62 Take over sparsity estimation improvements OK
- * 63 Take over fixes for transform w/ binning
-
-SYSTEMDS-70 Lineage Tracing and Reuse OK
- * 71 Basic collection of lineage traces OK
- * 72 Serialization and deserialization of lineage traces OK
- * 73 Deduplication of lineage traces OK
- (loops, nested loops, branchless loops) OK
- * 74 Performance features lineage tracing OK
- * 75 Reuse cache based on lineage traces OK
- * 76 Generate runtime plan from lineage trace OK
- * 77 New builtin function for obtaining lineage OK
- * 78 Extended lineage tracing (parfor, funs, codegen) OK
- * 79 Reuse cache configuration options (cmd line options) OK
-
-SYSTEMDS-80 Improved distributed operations
- * 81 Avoid unnecessary replication on rmm
- * 82 Avoid unnecessary replication on binary
-
-SYSTEMDS-100 Various Fixes
- * 101 Fix spark quantiles w/ multiple queries OK
- * 102 Fix parser issue after refactoring OK
- * 103 Fix handling of builtin functions w/ matching udfs OK
- * 104 Fix failing tests due to incorrect parfor parameters OK
- * 105 Fix all application/function tests (various issues) OK
- * 106 Fix correctness of as.integer for negative numbers OK
- * 107 Fix correctness IPA check dimension-preserving OK
- * 108 Fix codegen optimizer (early costing abort) OK
- * 109 Fix update-in-place w/ udf function calls OK
-
-SYSTEMDS-110 New Builtin Functions
- * 111 Time builtin function for script-level measurements OK
- * 112 Image data augmentation builtin functions OK
- * 113 Builtin functions for linear regression algorithms OK
- * 114 Builtin function for stepwise regression OK
- * 115 Builtin function for model debugging (slice finder) OK
- * 116 Builtin function for kmeans OK
- * 117 Builtin function for lm cross validation OK
- * 118 Builtin function for hyperparameter grid search
- * 119 Builtin functions for l2svm and msvm OK
-
-SYSTEMDS-120 Performance Features
- * 121 Avoid spark context creation on parfor result merge OK
- * 122 Reduce thread contention on parfor left indexing OK
- * 123 Avoid unnecessary spark context creation on explain OK
- * 124 Improved cbind performance (small rhs) OK
- * 125 Parallel Sort (previously 191) OK
-
-SYSTEMDS-130 IPA and Size Propagation
- * 131 New IPA pass for function call forwarding OK
- * 132 Size propagation via data characteristics
- * 133 Hop Size Propagation for Tensors
- * 134 Increased repretition and fixpoint detection OK
- * 135 Fix IPA scalar propagation w/ named arguments OK
-
-SYSTEMDS-140 Distributed Tensor Operations
- * 141 Infrastructure for spark tensor operations
- * 142 Basis tensor operations (datagen, aggregate, elementwise)
-
-SYSTEMDS-150 Releases
- * 151 Fixing native BLAS instructions (MKL,OpenBLAS) OK
- * 152 Run script documentation OK
- * 153 convert all occurrences SystemML/sysml to SystemDS/sysds OK
- * 154 fix license headers in all touched files OK
- * 155 import release scripts from SystemML repo OK
- * 156 initial codestyle eclipse template OK
- * 157 Shell script simplification, cleanups and fixes for 0.2 OK
-
-SYSTEMDS-160 Tensor Compiler/Runtime
- * 161 Local readers and writers for tensors (all formats)
- * 162 Spark binary tensor instructions
-
-SYSTEMDS-170 Lineage full and partial reuse
- * 171 Initial version of partial rewrites OK
- * 172 Parfor integration (blocked waiting for results) OK
- * 173 Improved cost estimates OK
- * 174 Reuse rewrite for rbind/cbind-tsmm/ba+* OK
- * 175 Refactoring of lineage rewrite code OK
- * 176 Reuse rewrite for cbind/rbind-elementwise */+
- * 177 Reuse rewrite for aggregate OK
- * 178 Compiler assisted reuse (eg. CV, lmCG)
- * 179 Lineage handling for lists OK
-
-SYSTEMDS-180 New Builtin Functions II OK
- * 181 Builtin function for naive bayes OK
- * 182 Builtin function for typeof (frame schema detection) OK
- * 183 Builtin function detectSchema OK
- * 184 Builtin function GNMF OK
- * 185 Builtin function PNMF OK
- * 186 Builtin function ALS-DS OK
- * 187 Builtin function ALS-CG OK
- * 188 Builtin function ALS OK
- * 189 Builtin function mice (chained equation imputation) OK
-
-SYSTEMDS-190 New Builtin Functions III
- * 191 Builtin function multi logreg OK
- * 192 Extended cmdline interface -b for calling builtin functions
- * 193 New outlierBySds builtin function (delete row/cell) OK
- * 194 New outlierByIQR builtin function (delete row/cell) OK
- * 195 Builtin function mice for nominal features OK
- * 196 Builtin function intersect (set intersection) OK
- * 197 Builtin function for functional dependency discovery OK
- * 198 Extended slice finding (classification) OK
- * 199 Builtin function Multinominal Logistic Regression Predict OK
- * 200 Builtin function Gaussian Mixture Model OK
-
-SYSTEMDS-200 Various Fixes
- * 201 Fix spark append instruction zero columns OK
- * 202 Fix rewrite split DAG multiple removeEmpty (pending op) OK
- * 203 Fix codegen outer compilation (MV operations) OK
- * 204 Fix rewrite simplify sequences of binary comparisons OK
- * 205 Fix scoping of builtin dml-bodied functions (vs user-defined)
- * 206 Fix codegen outer template compilation (tsmm) OK
- * 207 Fix builtin function call hoisting from expressions OK
- * 208 Fix bufferpool leak (live var analysis and createvar) OK
- * 209 Fix performance sparse M-CV elementwise multiply OK
-
-SYSTEMDS-210 Extended lists Operations
- * 211 Cbind and Rbind over lists of matrices OK
- * 212 Creation of empty lists OK
- * 213 Add entries to lists OK
- * 214 Removal of entries from lists OK
- * 215 List expansion during dynamic recompilation OK
- * 216 New rewrite for tsmm/mm over list of folds OK
-
-SYSTEMDS-220 Federated Tensors and Instructions
- * 221 Initial infrastructure federated operations OK
- * 222 Federated matrix-vector multiplication OK
- * 223 Federated unary aggregates OK
- * 224 Federated frames OK
- * 225 Federated elementwise operations OK
- * 226 Federated rbind and cbind OK
- * 227 Federated worker setup and infrastructure OK
- * 228 Federated matrix-matrix multiplication OK
- * 229 Federated transform functionality
-
-SYSTEMDS-230 Lineage Integration
- * 231 Use lineage in buffer pool
- * 232 Lineage of code generated operators OK
- * 233 Lineage cache and reuse of function results OK
- * 234 Lineage tracing for spark instructions OK
- * 235 Lineage tracing for remote-spark parfor OK
- * 236 Extended multi-level lineage cache for statement blocks OK
- * 237 Unmark loop dependent operations OK
- * 238 Robustness parfor lineage merge (flaky tests) OK
- * 239 Make lineage accessible through python bindings OK
-
-SYSTEMDS-240 GPU Backend Improvements
- * 241 Dense GPU cumulative aggregates OK
-
-SYSTEMDS-250 Extended Slice Finding
- * 251 Alternative slice enumeration approach OK
- * 252 Initial data slicing implementation Python OK
- * 253 Distributed slicing algorithms (task/data parallel) OK
- * 254 Consolidation and fixes distributed slice finding OK
-
-SYSTEMDS-260 Misc Tools
- * 261 Stable marriage algorithm OK
- * 262 Data augmentation tool for data cleaning OK
- * 263 ONNX graph importer (Python API, docs, tests) OK
- * 264 ONNX graph exporter
- * 265 Entity resolution pipelines and primitives OK
-
-SYSTEMDS-270 Compressed Matrix Blocks
- * 271 Reintroduce compressed matrix blocks from SystemML OK
- * 272 Simplify and speedup compression tests OK
- * 273 Refactor compressed Matrix Block to simplify responsibilities OK
- * 273a Redesign allocation of ColGroups in ColGroupFactory
- * 274 Make the DDC Compression dictionary share correctly OK
- * 275 Include compressionSettings in DMLConfiguration
- * 276 Allow Uncompressed Columns to be in sparse formats OK
- * 277 Sampling based estimators fix
- * 278 Compression-CoCode algorithm optimization
- * 278a Return ColGroups estimated compression ratio to Factory OK
- * 279 Add missing standard lossless compression techniques
- * 279a ColGroup FOR (Frame of reference) encoding
- * 279b ColGroup DEL (Delta) encoding
- * MINOR Reduce memory usage for compression statistics. OK
- * MINOR Make ContainsAZero() method in UncompressedBitMap
-
-SYSTEMDS-280 New Rewrites
- * 281 New rewrites sum(rmEmpty), sum(table) OK
- * 282 New rewrite binary add chain to nary plus OK
-
-SYSTEMDS-290 Misc Compiler Features
- * 291 Eval function calls of dml-bodied builtin functions OK
- * 292 Eval function calls of native builtin functions (genate fun)
- * 293 Compiler rewrites for eval to normal function calls
- * 294 DML-bodied builtin functions with frames as main input
-
-SYSTEMDS-300 Github Workflow/Actions
- * 301 Initial Setup of GitHub Actions for automatic testing OK
- * 302 Workflow for Application tests OK
- * 303 Workflow for Function tests OK
- * 304 Workflow make documentation OK
- * 305 Workflow run docker containers for speedup OK
-
-SYSTEMDS-310 Python Bindings
- * 311 Initial Python Binding for federated execution OK
- * 312 Python 3.6 compatibility OK
- * 313 Python Documentation upload via Github Actions OK
- * 314 Python SystemDS context manager OK
- * 314a Change py4j interaction using py4j for termination not stdin
- * 315 Python Federated Matrices Tests OK
- * 316 Extended Python API (rand, lm, mm) OK
- * 317 Python Rerun with different inputs without recompilation
- * 318 Python Multiple outputs
- * 319 Python Caching of intermediates
-
-SYSTEMDS-320 Merge SystemDS into Apache SystemML OK
- * 321 Merge histories of SystemDS and SystemML OK
- * 322 Change global package names OK
- * 323 Fix licenses and notice file OK
-
-SYSTEMDS-330 Lineage Tracing, Reuse and Integration
- * 331 Cache and reuse scalar outputs (instruction and multi-level) OK
- * 332 Parfor integration with multi-level reuse OK
- * 333 Improve cache eviction with actual compute time OK
- * 334 Cache scalars only with atleast one matrix inputs
- * 335 Weighted eviction policy (function(size,computetime,LRU time)) OK
- * 336 Better use of cache status to handle multithreading
- * 337 Adjust disk I/O speed by recording actual time taken OK
- * 338 Extended lineage tracing (rmEmpty, lists), partial rewrites OK
- * 339 Lineage tracing robustness (indexed updates, algorithms)
-
-SYSTEMDS-340 Compiler Assisted Lineage Caching and Reuse
- * 341 Finalize unmarking of loop dependent operations
- * 342 Mark functions as last-use to enable early eviction
- * 343 Identify equal last level HOPs to ensure SB-level reuse
- * 344 Unmark functions/SBs containing non-determinism for caching OK
- * 345 Compiler assisted cache configuration
-
-SYSTEMDS-350 Data Cleaning Framework
- * 351 New builtin function for error correction by schema OK
- * 352 New builtin function for error correction by length Ok
-
-SYSTEMDS-360 Privacy/Data Exchange Constraints
- * 361 Initial privacy meta data (compiler/runtime) OK
- * 362 Runtime privacy propagation
- * 363 Compile-time privacy propagation
- * 364 Error handling violated privacy constraints OK
- * 365 Extended privacy/data exchange constraints OK
-
-SYSTEMDS-370 Lossy Compression Blocks
- * 371 ColGroup Quantization OK
(Naive Q8)
- * 372 ColGroup Base Data change (from Double to Naive Q8) OK
- * 373 Fix error in new compression OK
-
-SYSTEMDS-380 Memory Footprint
- * 381 Matrix Block Memory footprint update
-
-SYSTEMDS-390 New Builtin Functions IV
- * 391 New GLM builtin function (from algorithms) OK
- * 392 Builtin function for missing value imputation via FDs OK
- * 393 Builtin to find Connected Components of a graph OK
- * 394 Builtin for one-hot encoding of matrix (not frame), see table OK
- * 395 SVM rework and utils (confusionMatrix, msvmPredict) OK
- * 396 Builtin for counting number of distinct values OK
- * 397 Algorithm for neural collaborative filtering (NCF) OK
-
-SYSTEMDS-400 Spark Backend Improvements
- * 401 Fix output block indexes of rdiag (diagM2V) OK
-
-SYSTEMDS-410 Lineage Tracing, Reuse and Integration II
- * 411 Improved handling of multi-level cache duplicates OK
- * 412 Robust lineage tracing (non-recursive, parfor) OK
- * 413 Cache and reuse MultiReturnBuiltin instructions OK
- * 414 New rewrite for PCA --> lmDS pipeline OK
- * 415 MLContext lineage support (tracing and reuse correctness) OK
- * 416 Lineage deduplication while, nested if, loop sequences OK
- * 417 New rewrite for partial reuse in StepLM OK
- * 418 Performance lineage tracing and reuse probing small data OK
- * 419 Performance and robustness partial rewrites
-
-SYSTEMDS-420 Compiler Improvements
- * 421 Fix invalid IPA scalar propagation into functions OK
-
-SYSTEMDS-500 Documentation Webpage Reintroduction
- * 501 Make Documentation webpage framework OK
-
-Others:
- * Break append instruction to cbind and rbind
-
-SYSTEMDS-510 IO formats
- * 511 Add protobuf support to write and read FrameBlocks to HDFS OK
-
-SYSTEMDS-520 Lineage Tracing, Reuse and Integration III
- * 521 New lineage option exposing cache policies OK
-
-SYSTEMDS-610 Cleaning Pipelines
- * 611 Initial Brute force execution OK
-
-
diff --git a/src/main/python/systemds/onnx_systemds/README.md
b/scripts/staging/onnx/onnx_systemds/README.md
similarity index 100%
rename from src/main/python/systemds/onnx_systemds/README.md
rename to scripts/staging/onnx/onnx_systemds/README.md
diff --git a/src/main/python/systemds/onnx_systemds/__init__.py
b/scripts/staging/onnx/onnx_systemds/__init__.py
similarity index 100%
rename from src/main/python/systemds/onnx_systemds/__init__.py
rename to scripts/staging/onnx/onnx_systemds/__init__.py
diff --git a/src/main/python/systemds/onnx_systemds/convert.py
b/scripts/staging/onnx/onnx_systemds/convert.py
similarity index 100%
rename from src/main/python/systemds/onnx_systemds/convert.py
rename to scripts/staging/onnx/onnx_systemds/convert.py
diff --git a/src/main/python/systemds/onnx_systemds/onnx_helper.py
b/scripts/staging/onnx/onnx_systemds/onnx_helper.py
similarity index 100%
rename from src/main/python/systemds/onnx_systemds/onnx_helper.py
rename to scripts/staging/onnx/onnx_systemds/onnx_helper.py
diff --git a/src/main/python/systemds/onnx_systemds/operator_gen.py
b/scripts/staging/onnx/onnx_systemds/operator_gen.py
similarity index 100%
rename from src/main/python/systemds/onnx_systemds/operator_gen.py
rename to scripts/staging/onnx/onnx_systemds/operator_gen.py
diff --git a/src/main/python/systemds/onnx_systemds/render.py
b/scripts/staging/onnx/onnx_systemds/render.py
similarity index 100%
rename from src/main/python/systemds/onnx_systemds/render.py
rename to scripts/staging/onnx/onnx_systemds/render.py
diff --git
a/src/main/python/systemds/onnx_systemds/templates/graph_function.dml.jinja
b/scripts/staging/onnx/onnx_systemds/templates/graph_function.dml.jinja
similarity index 100%
rename from
src/main/python/systemds/onnx_systemds/templates/graph_function.dml.jinja
rename to scripts/staging/onnx/onnx_systemds/templates/graph_function.dml.jinja
diff --git
a/src/main/python/systemds/onnx_systemds/templates/graph_header.dml.jinja
b/scripts/staging/onnx/onnx_systemds/templates/graph_header.dml.jinja
similarity index 100%
rename from
src/main/python/systemds/onnx_systemds/templates/graph_header.dml.jinja
rename to scripts/staging/onnx/onnx_systemds/templates/graph_header.dml.jinja
diff --git a/src/main/python/systemds/onnx_systemds/templates/main.dml.jinja
b/scripts/staging/onnx/onnx_systemds/templates/main.dml.jinja
similarity index 100%
rename from src/main/python/systemds/onnx_systemds/templates/main.dml.jinja
rename to scripts/staging/onnx/onnx_systemds/templates/main.dml.jinja
diff --git
a/src/main/python/systemds/onnx_systemds/templates/matrix_initialize.dml.jinja
b/scripts/staging/onnx/onnx_systemds/templates/matrix_initialize.dml.jinja
similarity index 100%
rename from
src/main/python/systemds/onnx_systemds/templates/matrix_initialize.dml.jinja
rename to
scripts/staging/onnx/onnx_systemds/templates/matrix_initialize.dml.jinja
diff --git
a/src/main/python/systemds/onnx_systemds/templates/model_header.dml.jinja
b/scripts/staging/onnx/onnx_systemds/templates/model_header.dml.jinja
similarity index 100%
rename from
src/main/python/systemds/onnx_systemds/templates/model_header.dml.jinja
rename to scripts/staging/onnx/onnx_systemds/templates/model_header.dml.jinja
diff --git
a/src/main/python/systemds/onnx_systemds/templates/module_import.dml.jinja
b/scripts/staging/onnx/onnx_systemds/templates/module_import.dml.jinja
similarity index 100%
rename from
src/main/python/systemds/onnx_systemds/templates/module_import.dml.jinja
rename to scripts/staging/onnx/onnx_systemds/templates/module_import.dml.jinja
diff --git
a/src/main/python/systemds/onnx_systemds/templates/operators/2input_1output_operator.dml.jinja
b/scripts/staging/onnx/onnx_systemds/templates/operators/2input_1output_operator.dml.jinja
similarity index 100%
rename from
src/main/python/systemds/onnx_systemds/templates/operators/2input_1output_operator.dml.jinja
rename to
scripts/staging/onnx/onnx_systemds/templates/operators/2input_1output_operator.dml.jinja
diff --git
a/src/main/python/systemds/onnx_systemds/templates/operators/function_call.dml.jinja
b/scripts/staging/onnx/onnx_systemds/templates/operators/function_call.dml.jinja
similarity index 100%
rename from
src/main/python/systemds/onnx_systemds/templates/operators/function_call.dml.jinja
rename to
scripts/staging/onnx/onnx_systemds/templates/operators/function_call.dml.jinja
diff --git
a/src/main/python/systemds/onnx_systemds/templates/operators/if_operator.dml.jinja
b/scripts/staging/onnx/onnx_systemds/templates/operators/if_operator.dml.jinja
similarity index 100%
rename from
src/main/python/systemds/onnx_systemds/templates/operators/if_operator.dml.jinja
rename to
scripts/staging/onnx/onnx_systemds/templates/operators/if_operator.dml.jinja
diff --git
a/src/main/python/systemds/onnx_systemds/templates/operators/neg.dml.jinja
b/scripts/staging/onnx/onnx_systemds/templates/operators/neg.dml.jinja
similarity index 100%
rename from
src/main/python/systemds/onnx_systemds/templates/operators/neg.dml.jinja
rename to scripts/staging/onnx/onnx_systemds/templates/operators/neg.dml.jinja
diff --git a/src/main/python/systemds/onnx_systemds/templates/util.dml.jinja
b/scripts/staging/onnx/onnx_systemds/templates/util.dml.jinja
similarity index 100%
rename from src/main/python/systemds/onnx_systemds/templates/util.dml.jinja
rename to scripts/staging/onnx/onnx_systemds/templates/util.dml.jinja
diff --git a/src/main/python/systemds/onnx_systemds/util.py
b/scripts/staging/onnx/onnx_systemds/util.py
similarity index 100%
rename from src/main/python/systemds/onnx_systemds/util.py
rename to scripts/staging/onnx/onnx_systemds/util.py
diff --git a/src/main/python/docs/source/onnx_systemds/onnx_systemds.rst
b/scripts/staging/onnx/onnx_systemds_docs/onnx_systemds.rst
similarity index 100%
rename from src/main/python/docs/source/onnx_systemds/onnx_systemds.rst
rename to scripts/staging/onnx/onnx_systemds_docs/onnx_systemds.rst
diff --git a/src/main/python/docs/source/onnx_systemds/onnx_systemds_design.rst
b/scripts/staging/onnx/onnx_systemds_docs/onnx_systemds_design.rst
similarity index 100%
rename from src/main/python/docs/source/onnx_systemds/onnx_systemds_design.rst
rename to scripts/staging/onnx/onnx_systemds_docs/onnx_systemds_design.rst
diff --git a/src/main/python/tests/onnx_systemds/__init__.py
b/scripts/staging/onnx/onnx_systemds_tests/__init__.py
similarity index 100%
rename from src/main/python/tests/onnx_systemds/__init__.py
rename to scripts/staging/onnx/onnx_systemds_tests/__init__.py
diff --git
a/src/main/python/tests/onnx_systemds/dml_wrapper/simple_conv_layer_2_wrapper.dml
b/scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_conv_layer_2_wrapper.dml
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/dml_wrapper/simple_conv_layer_2_wrapper.dml
rename to
scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_conv_layer_2_wrapper.dml
diff --git
a/src/main/python/tests/onnx_systemds/dml_wrapper/simple_conv_layer_wrapper.dml
b/scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_conv_layer_wrapper.dml
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/dml_wrapper/simple_conv_layer_wrapper.dml
rename to
scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_conv_layer_wrapper.dml
diff --git
a/src/main/python/tests/onnx_systemds/dml_wrapper/simple_dropout_layer_wrapper.dml
b/scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_dropout_layer_wrapper.dml
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/dml_wrapper/simple_dropout_layer_wrapper.dml
rename to
scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_dropout_layer_wrapper.dml
diff --git
a/src/main/python/tests/onnx_systemds/dml_wrapper/simple_if_graph_wrapper.dml
b/scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_if_graph_wrapper.dml
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/dml_wrapper/simple_if_graph_wrapper.dml
rename to
scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_if_graph_wrapper.dml
diff --git
a/src/main/python/tests/onnx_systemds/dml_wrapper/simple_mat_add_mul_sub_wrapper.dml
b/scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_mat_add_mul_sub_wrapper.dml
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/dml_wrapper/simple_mat_add_mul_sub_wrapper.dml
rename to
scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_mat_add_mul_sub_wrapper.dml
diff --git
a/src/main/python/tests/onnx_systemds/dml_wrapper/simple_mat_add_wrapper.dml
b/scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_mat_add_wrapper.dml
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/dml_wrapper/simple_mat_add_wrapper.dml
rename to
scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_mat_add_wrapper.dml
diff --git
a/src/main/python/tests/onnx_systemds/dml_wrapper/simple_mat_initialized_wrapper.dml
b/scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_mat_initialized_wrapper.dml
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/dml_wrapper/simple_mat_initialized_wrapper.dml
rename to
scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_mat_initialized_wrapper.dml
diff --git
a/src/main/python/tests/onnx_systemds/dml_wrapper/simple_maxpool_layer_wrapper.dml
b/scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_maxpool_layer_wrapper.dml
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/dml_wrapper/simple_maxpool_layer_wrapper.dml
rename to
scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_maxpool_layer_wrapper.dml
diff --git
a/src/main/python/tests/onnx_systemds/dml_wrapper/simple_relu_tanh_sigmoid_softmax_wrapper.dml
b/scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_relu_tanh_sigmoid_softmax_wrapper.dml
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/dml_wrapper/simple_relu_tanh_sigmoid_softmax_wrapper.dml
rename to
scripts/staging/onnx/onnx_systemds_tests/dml_wrapper/simple_relu_tanh_sigmoid_softmax_wrapper.dml
diff --git
a/src/main/python/tests/onnx_systemds/output_reference/simple_conv_layer_2_reference.out
b/scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_conv_layer_2_reference.out
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/output_reference/simple_conv_layer_2_reference.out
rename to
scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_conv_layer_2_reference.out
diff --git
a/src/main/python/tests/onnx_systemds/output_reference/simple_conv_layer_reference.out
b/scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_conv_layer_reference.out
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/output_reference/simple_conv_layer_reference.out
rename to
scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_conv_layer_reference.out
diff --git
a/src/main/python/tests/onnx_systemds/output_reference/simple_if_graph_reference.out
b/scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_if_graph_reference.out
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/output_reference/simple_if_graph_reference.out
rename to
scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_if_graph_reference.out
diff --git
a/src/main/python/tests/onnx_systemds/output_reference/simple_mat_add_mul_sub_reference.out
b/scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_mat_add_mul_sub_reference.out
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/output_reference/simple_mat_add_mul_sub_reference.out
rename to
scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_mat_add_mul_sub_reference.out
diff --git
a/src/main/python/tests/onnx_systemds/output_reference/simple_mat_add_reference.out
b/scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_mat_add_reference.out
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/output_reference/simple_mat_add_reference.out
rename to
scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_mat_add_reference.out
diff --git
a/src/main/python/tests/onnx_systemds/output_reference/simple_mat_initialized_reference.out
b/scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_mat_initialized_reference.out
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/output_reference/simple_mat_initialized_reference.out
rename to
scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_mat_initialized_reference.out
diff --git
a/src/main/python/tests/onnx_systemds/output_reference/simple_maxpool_layer_reference.out
b/scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_maxpool_layer_reference.out
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/output_reference/simple_maxpool_layer_reference.out
rename to
scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_maxpool_layer_reference.out
diff --git
a/src/main/python/tests/onnx_systemds/output_reference/simple_relu_tanh_sigmoid_softmax_reference.out
b/scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_relu_tanh_sigmoid_softmax_reference.out
similarity index 100%
rename from
src/main/python/tests/onnx_systemds/output_reference/simple_relu_tanh_sigmoid_softmax_reference.out
rename to
scripts/staging/onnx/onnx_systemds_tests/output_reference/simple_relu_tanh_sigmoid_softmax_reference.out
diff --git a/src/main/python/tests/onnx_systemds/test_models/model_generate.py
b/scripts/staging/onnx/onnx_systemds_tests/test_models/model_generate.py
similarity index 100%
rename from src/main/python/tests/onnx_systemds/test_models/model_generate.py
rename to scripts/staging/onnx/onnx_systemds_tests/test_models/model_generate.py
diff --git a/src/main/python/tests/onnx_systemds/test_simple.py
b/scripts/staging/onnx/onnx_systemds_tests/test_simple.py
similarity index 100%
rename from src/main/python/tests/onnx_systemds/test_simple.py
rename to scripts/staging/onnx/onnx_systemds_tests/test_simple.py
diff --git a/src/main/python/tests/onnx_systemds/util.py
b/scripts/staging/onnx/onnx_systemds_tests/util.py
similarity index 100%
rename from src/main/python/tests/onnx_systemds/util.py
rename to scripts/staging/onnx/onnx_systemds_tests/util.py
diff --git a/src/main/python/docs/requires-docs.txt
b/src/main/python/docs/requires-docs.txt
index 478ef87..9305d93 100644
--- a/src/main/python/docs/requires-docs.txt
+++ b/src/main/python/docs/requires-docs.txt
@@ -24,6 +24,4 @@ sphinx_rtd_theme
numpy
py4j
scipy
-jinja2
-onnx
requests
\ No newline at end of file
diff --git a/src/main/python/docs/source/getting_started/install.rst
b/src/main/python/docs/source/getting_started/install.rst
index 596bc8f..8505185 100644
--- a/src/main/python/docs/source/getting_started/install.rst
+++ b/src/main/python/docs/source/getting_started/install.rst
@@ -62,7 +62,7 @@ Once installed you please verify your version numbers.
Additionally you have to install a few python packages.
Note depending on your installation you might need to use pip3 instead of pip::
- pip install numpy py4j wheel jinja2 onnx requests
+ pip install numpy py4j wheel requests
Then to build the system you do the following
diff --git a/src/main/python/setup.py b/src/main/python/setup.py
index 9340440..5939101 100755
--- a/src/main/python/setup.py
+++ b/src/main/python/setup.py
@@ -38,8 +38,6 @@ ARTIFACT_VERSION_SHORT = ARTIFACT_VERSION.split("-")[0]
REQUIRED_PACKAGES = [
'numpy >= 1.8.2',
'py4j >= 0.10.9',
- 'jinja2 >= 2.11.2',
- 'onnx >= 1.7.0',
'requests >= 2.24.0'
]
diff --git a/src/main/python/systemds/__init__.py
b/src/main/python/systemds/__init__.py
index 064f78c..0aecd2e 100644
--- a/src/main/python/systemds/__init__.py
+++ b/src/main/python/systemds/__init__.py
@@ -22,7 +22,6 @@
from systemds import context
from systemds import matrix
from systemds import operator
-from systemds import onnx_systemds
from systemds import examples
-__all__ = [context, matrix, operator, onnx_systemds, examples]
+__all__ = [context, matrix, operator, examples]
diff --git a/src/test/java/org/apache/sysds/test/TestUtils.java
b/src/test/java/org/apache/sysds/test/TestUtils.java
index 7b7893b..b0e5d64 100644
--- a/src/test/java/org/apache/sysds/test/TestUtils.java
+++ b/src/test/java/org/apache/sysds/test/TestUtils.java
@@ -2079,7 +2079,6 @@ public class TestUtils
* @param matrix
* matrix
*/
- @SuppressWarnings("deprecation")
public static void writeBinaryTestMatrixCells(String file, double[][]
matrix) {
try {
SequenceFile.Writer writer = null;
@@ -2127,7 +2126,6 @@ public class TestUtils
* @param sparseFormat
* sparse format
*/
- @SuppressWarnings("deprecation")
public static void writeBinaryTestMatrixBlocks(String file, double[][]
matrix, int rowsInBlock, int colsInBlock,
boolean sparseFormat) {
SequenceFile.Writer writer = null;
