I was finally able to hit an OOM error.
I initially was running everything from the top.
When I run mvn test from the viennacl directory...
/mahout/viennacl$ mvn test
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model
for org.apache.mahout:mahout-native-viennacl_2.10:jar:0.13.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @
org.apache.mahout:mahout-native-viennacl_${scala.compat.version}:[unknown-version],
/home/trevor/gits/mahout/viennacl/pom.xml, line 31, column 15
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but
found duplicate declaration of plugin org.codehaus.mojo:exec-maven-plugin @
org.apache.mahout:mahout-native-viennacl_${scala.compat.version}:[unknown-version],
/home/trevor/gits/mahout/viennacl/pom.xml, line 181, column 15
[WARNING]
[WARNING] It is highly recommended to fix these problems because they
threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support
building such malformed projects.
[WARNING]
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] Building Mahout Native VienniaCL OpenCL Bindings 0.13.0-SNAPSHOT
[INFO]
------------------------------------------------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:1.4:enforce (enforce-versions) @
mahout-native-viennacl_2.10 ---
[INFO]
[INFO] --- scala-maven-plugin:3.2.0:add-source (add-scala-sources) @
mahout-native-viennacl_2.10 ---
[INFO] Add Source directory:
/home/trevor/gits/mahout/viennacl/src/main/scala
[INFO] Add Test Source directory:
/home/trevor/gits/mahout/viennacl/src/test/scala
[INFO]
[INFO] --- maven-dependency-plugin:2.3:properties (default) @
mahout-native-viennacl_2.10 ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
mahout-native-viennacl_2.10 ---
[INFO]
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @
mahout-native-viennacl_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/home/trevor/gits/mahout/viennacl/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile) @
mahout-native-viennacl_2.10 ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @
mahout-native-viennacl_2.10 ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- exec-maven-plugin:1.2.1:exec (javacpp) @
mahout-native-viennacl_2.10 ---
Warning: Could not load platform properties for class
org.apache.mahout.viennacl.opencl.GPUMMul
Warning: Could not load platform properties for class
org.apache.mahout.viennacl.opencl.GPUMMul$
Generating
/home/trevor/gits/mahout/viennacl/target/classes/org/apache/mahout/viennacl/opencl/javacpp/jniViennaCL.cpp
Compiling
/home/trevor/gits/mahout/viennacl/target/classes/org/apache/mahout/viennacl/opencl/javacpp/linux-x86_64/libjniViennaCL.so
g++ -I/usr/include/viennacl -I/usr/lib/jvm/java-7-openjdk-amd64/include
-I/usr/lib/jvm/java-7-openjdk-amd64/include/linux
/home/trevor/gits/mahout/viennacl/target/classes/org/apache/mahout/viennacl/opencl/javacpp/jniViennaCL.cpp
-msse3 -ffast-math -fopenmp -fpermissive -Wl,-rpath,$ORIGIN/
-Wl,-z,noexecstack -Wl,-Bsymbolic -march=native -m64 -Wall -Ofast -fPIC
-shared -s -o libjniViennaCL.so -lOpenCL
Deleting
/home/trevor/gits/mahout/viennacl/target/classes/org/apache/mahout/viennacl/opencl/javacpp/jniViennaCL.cpp
[INFO]
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources)
@ mahout-native-viennacl_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/home/trevor/gits/mahout/viennacl/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.0:testCompile (scala-test-compile) @
mahout-native-viennacl_2.10 ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.3:testCompile (default-testCompile) @
mahout-native-viennacl_2.10 ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @
mahout-native-viennacl_2.10 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- scalatest-maven-plugin:1.0:test (test) @
mahout-native-viennacl_2.10 ---
Discovery starting.
Discovery completed in 201 milliseconds.
Run starting. Expected test count is: 7
ViennaCLSuiteVCL:
- row-major viennacl::matrix
+ OCL matrix memory domain after assgn=2
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul solver
jvmRWRW
gpuRWCW
log4j:WARN No appenders could be found for logger
(org.apache.mahout.viennacl.opencl.GPUMMul$).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul solver
jvmRWRW
gpuRWCW
- dense vcl mmul with fast_copy
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul solver
jvmRWRW
gpuRWCW
- mmul microbenchmark
+ Mahout multiplication time: 2630 ms.
+ ViennaCL/OpenCL multiplication time: 2072 ms.
+ ViennaCL/cpu/OpenMP multiplication time: 813 ms.
- trans
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul solver
gpuSparseRWRW
ViennaCL: FATAL ERROR: Kernel start failed for 'spgemm_stage3'.
ViennaCL: Smaller work sizes could not solve the problem.
ViennaCL: FATAL ERROR: CL_MEM_OBJECT_ALLOCATION_FAILURE
ViennaCL could not allocate memory on the device. Most likely the device
simply ran out of memory.
If you think that this is a bug in ViennaCL, please report it at
[email protected] and supply at least the following
information:
* Operating System
* Which OpenCL implementation (AMD, NVIDIA, etc.)
* ViennaCL version
Many thanks in advance!falling back to JVM MMUL
ViennaCL: FATAL ERROR: Kernel start failed for 'spgemm_stage3'.
ViennaCL: Smaller work sizes could not solve the problem.
- sparse mmul microbenchmark *** FAILED ***
java.lang.RuntimeException: ViennaCL: FATAL ERROR:
CL_MEM_OBJECT_ALLOCATION_FAILURE
ViennaCL could not allocate memory on the device. Most likely the device
simply ran out of memory.
If you think that this is a bug in ViennaCL, please report it at
[email protected] and supply at least the following
information:
* Operating System
* Which OpenCL implementation (AMD, NVIDIA, etc.)
* ViennaCL version
Many thanks in advance!
at
org.apache.mahout.viennacl.opencl.javacpp.CompressedMatrix.allocate(Native
Method)
at
org.apache.mahout.viennacl.opencl.javacpp.CompressedMatrix.<init>(CompressedMatrix.scala:61)
at
org.apache.mahout.viennacl.opencl.ViennaCLSuiteVCL$$anonfun$5.apply$mcV$sp(ViennaCLSuiteVCL.scala:250)
at
org.apache.mahout.viennacl.opencl.ViennaCLSuiteVCL$$anonfun$5.apply(ViennaCLSuiteVCL.scala:218)
at
org.apache.mahout.viennacl.opencl.ViennaCLSuiteVCL$$anonfun$5.apply(ViennaCLSuiteVCL.scala:218)
at
org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
...
+ Mahout Sparse multiplication time: 13303 ms.
- VCL Dense Matrix %*% Dense vector
+ Mahout dense matrix %*% dense vector multiplication time: 0 ms.
+ ViennaCL/cpu/OpenMP dense matrix %*% dense vector multiplication time:
4 ms.
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul solver
gpuRWCW
- Sparse %*% Dense mmul microbenchmark
+ Mahout multiplication time: 821 ms.
+ ViennaCL/OpenCL multiplication time: 754 ms.
+ ViennaCL/cpu/OpenMP multiplication time: 1473 ms.
Run completed in 26 seconds, 944 milliseconds.
Total number of tests run: 7
Suites: completed 2, aborted 0
Tests: succeeded 6, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 45.600 s
[INFO] Finished at: 2016-10-28T17:58:45-05:00
[INFO] Final Memory: 20M/309M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.scalatest:scalatest-maven-plugin:1.0:test (test) on project
mahout-native-viennacl_2.10: There are test failures -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Trevor Grant
Data Scientist
https://github.com/rawkintrevo
http://stackexchange.com/users/3002022/rawkintrevo
http://trevorgrant.org
*"Fortunate is he, who is able to know the causes of things." -Virgil*
On Fri, Oct 28, 2016 at 5:38 PM, Andrew Musselman <
[email protected]> wrote:
> >
> > My Desktop
>
> >
> Branch: master
>
> > Build Command: mvn clean install -Phadoop2
>
> > Built and tested successfully
>
> >
> Branch: andrewpalumbo/mahout/viennacl-opmmul-a
>
> > Bulid Command: mvn clean install -Pviennacl -Phadoop2
>
> Builds successfully-
>
> > Tests still riddled with
>
> >
> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
>
> > [WARN] Unable to create class GPUMMul: attempting OpenMP version
>
> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
>
> > [INFO] Unable to create class OMPMMul: falling back to java version
>
> >
> Also OOM
> ViennaCLSuiteOMP:
> - row-major viennacl::matrix
> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> [WARN] Unable to create class GPUMMul: attempting OpenMP version
> [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> [INFO] Unable to create class OMPMMul: falling back to java version
> - mmul microbenchmark
> + Mahout multiplication time: 8875 ms.
> + ViennaCL/cpu/OpenMP multiplication time: 599 ms.
> - trans
> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> [WARN] Unable to create class GPUMMul: attempting OpenMP version
> [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> [INFO] Unable to create class OMPMMul: falling back to java version
> *** RUN ABORTED ***
> java.lang.OutOfMemoryError: Java heap space
> at
> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.rehash(
> Int2DoubleOpenHashMap.java:1059)
> at
> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.insert(
> Int2DoubleOpenHashMap.java:295)
> at
> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.put(
> Int2DoubleOpenHashMap.java:301)
> at
> org.apache.mahout.math.RandomAccessSparseVector.setQuick(
> RandomAccessSparseVector.java:130)
> at
> org.apache.mahout.math.SparseRowMatrix.setQuick(SparseRowMatrix.java:105)
> at org.apache.mahout.math.AbstractMatrix.assign(AbstractMatrix.java:256)
> at
> org.apache.mahout.math.scalabindings.MatrixOps.$
> colon$eq(MatrixOps.scala:192)
> at
> org.apache.mahout.math.scalabindings.MatrixOps.cloned(MatrixOps.scala:260)
> at
> org.apache.mahout.math.scalabindings.MatrixOps.$minus(MatrixOps.scala:66)
> at
> org.apache.mahout.viennacl.openmp.ViennaCLSuiteOMP$$
> anonfun$4.apply$mcV$sp(ViennaCLSuiteOMP.scala:152)
>
> >
>
> Machine Info:
>
> > ============================================================
> ====================
>
> >
> os major/minor:
>
> > $ lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description: Ubuntu 16.04.1 LTS
> Release: 16.04
> Codename: xenial
>
> >
>
> chip arch:
>
> > $ uname -a
> Linux Bob 4.4.0-34-generic #53-Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016
> x86_64 x86_64 x86_64 GNU/Linux
>
> >
> $ cat /proc/cpuinfo
>
> > ...
>
> > model name : Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
>
> > ...
>
> > ^^ 6 cores
>
> >
>
> GPU:
>
> > $ sudo nvidia-smi
>
> >
> +-----------------------------------------------------------
> ------------------+
> | NVIDIA-SMI 367.44 Driver Version: 367.44
> |
> |-------------------------------+----------------------+----
> ------------------+
> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr.
> ECC |
> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute
> M. |
> |===============================+======================+====
> ==================|
> | 0 GeForce GTX 750 Ti Off | 0000:01:00.0 On |
> N/A |
> | 29% 25C P8 1W / 38W | 307MiB / 1998MiB | 0%
> Default |
> +-------------------------------+----------------------+----
> ------------------+
>
>
> +-----------------------------------------------------------
> ------------------+
> | Processes: GPU
> Memory |
> | GPU PID Type Process name Usage
> |
> |===========================================================
> ==================|
> | 0 1537 G ...C-EnableWebRtcEcdsa,WebRTC-H264WithOpenH2
> 87MiB |
> | 0 1990 G /usr/lib/xorg/Xorg
> 101MiB |
> | 0 2319 G /usr/bin/gnome-shell
> 104MiB |
> | 0 4360 G /usr/lib/xorg/Xorg
> 12MiB |
> +-----------------------------------------------------------
> ------------------+
>
> >
>
>
>
> clinfo output:
>
> > $ clinfo
> Number of platforms 1
> Platform Name NVIDIA CUDA
> Platform Vendor NVIDIA Corporation
> Platform Version OpenCL 1.2 CUDA 8.0.0
> Platform Profile FULL_PROFILE
> Platform Extensions
> cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics
> cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics
> cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing
> cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll
> cl_nv_copy_opts
> Platform Extensions function suffix NV
>
> Platform Name NVIDIA CUDA
> Number of devices 1
> Device Name GeForce GTX 750 Ti
> Device Vendor NVIDIA Corporation
> Device Vendor ID 0x10de
> Device Version OpenCL 1.2 CUDA
> Driver Version 367.44
> Device OpenCL C Version OpenCL C 1.2
> Device Type GPU
> Device Profile FULL_PROFILE
> Device Topology (NV) PCI-E, 01:00.0
> Max compute units 5
> Max clock frequency 1150MHz
> Compute Capability (NV) 5.0
> Device Partition (core)
> Max number of sub-devices 1
> Supported partition types None
> Max work item dimensions 3
> Max work item sizes 1024x1024x64
> Max work group size 1024
> Preferred work group size multiple 32
> Warp size (NV) 32
> Preferred / native vector sizes
> char 1 / 1
> short 1 / 1
> int 1 / 1
> long 1 / 1
> half 0 / 0 (n/a)
> float 1 / 1
> double 1 / 1
> (cl_khr_fp64)
> Half-precision Floating-point support (n/a)
> Single-precision Floating-point support (core)
> Denormals Yes
> Infinity and NANs Yes
> Round to nearest Yes
> Round to zero Yes
> Round to infinity Yes
> IEEE754-2008 fused multiply-add Yes
> Support is emulated in software No
> Correctly-rounded divide and sqrt operations Yes
> Double-precision Floating-point support (cl_khr_fp64)
> Denormals Yes
> Infinity and NANs Yes
> Round to nearest Yes
> Round to zero Yes
> Round to infinity Yes
> IEEE754-2008 fused multiply-add Yes
> Support is emulated in software No
> Correctly-rounded divide and sqrt operations No
> Address bits 64, Little-Endian
> Global memory size 2095841280 (1.952GiB)
> Error Correction support No
> Max memory allocation 523960320 (499.7MiB)
> Unified memory for Host and Device No
> Integrated memory (NV) No
> Minimum alignment for any data type 128 bytes
> Alignment of base address 4096 bits (512 bytes)
> Global Memory cache type Read/Write
> Global Memory cache size 81920
> Global Memory cache line 128 bytes
> Image support Yes
> Max number of samplers per kernel 32
> Max size for 1D images from buffer 134217728 pixels
> Max 1D or 2D image array size 2048 images
> Max 2D image size 16384x16384 pixels
> Max 3D image size 4096x4096x4096 pixels
> Max number of read image args 256
> Max number of write image args 16
> Local memory type Local
> Local memory size 49152 (48KiB)
> Registers per block (NV) 65536
> Max constant buffer size 65536 (64KiB)
> Max number of constant args 9
> Max size of kernel argument 4352 (4.25KiB)
> Queue properties
> Out-of-order execution Yes
> Profiling Yes
> Prefer user sync for interop No
> Profiling timer resolution 1000ns
> Execution capabilities
> Run OpenCL kernels Yes
> Run native kernels No
> Kernel execution timeout (NV) Yes
> Concurrent copy and kernel execution (NV) Yes
> Number of async copy engines 1
> printf() buffer size 1048576 (1024KiB)
> Built-in kernels
> Device Available Yes
> Compiler Available Yes
> Linker Available Yes
> Device Extensions
> cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics
> cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics
> cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing
> cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll
> cl_nv_copy_opts
>
> NULL platform behavior
> clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) NVIDIA CUDA
> clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) Success [NV]
> clCreateContext(NULL, ...) [default] Success [NV]
> clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found in
> platform
> clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) No platform
> clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No devices
> found in platform
> clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) No devices found in
> platform
> clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) No platform
>
> ICD loader properties
> ICD loader Name OpenCL ICD Loader
> ICD loader Vendor OCL Icd free software
> ICD loader Version 2.2.8
> ICD loader Profile OpenCL 1.2
> NOTE: your OpenCL library declares to support OpenCL 1.2,
> but it seems to support up to OpenCL 2.1 too.
>
> >
>
> Java major/minor and vendor:
>
> > $ java -version
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-3ubuntu1~16.04.1-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
>
> >
>
> vcl version,
>
> > (Ignoring what is in /usr/include/viennacl/version.hpp)
>
> > 1.7.1
>
>
> gcc version:
>
> > $ gcc --version
> gcc (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609
>
> >
>
> On Wed, Oct 26, 2016 at 11:31 PM, Andrew Palumbo <[email protected]>
> wrote:
>
> >
> >
> >
> > Could everybody please post any errors when stack trace that they are
> > getting on master or the vcl pr branch along with full system info?
> >
> > Ie. Branch, os major/minor, chip arch, GPU, clinfo output, Java
> > major/minor and vendor, vcl version, gcc version, and anything else that
> > may be useful so that we may compare?
> >
> > Thx
> >
> > Andy
> >
> >
> > Sent from my Galaxy Tab A
> >
>
