Re: [julia-users] Segmentation fault during Julia compilation
Hi All, I am working on building/porting Julia on ppc64le architecture. I am using Ubuntu 14.10 on top of ppc64le hardware, while compiling Julia code (master branch) I was getting segmentation fault, I am able to resolve this segmentation fault by turning on ‘MEMDEBUG’ flag from ‘src/options.h’ file. I decided to work more this issue and try to find out root cause of segmentation fault, so I started studying/understanding memory management of Julia. I have couple of questions in my mind regarding memory management of Julia and I want to discuss those here. 1. While defining ‘REGION_PG_COUNT’ macro, 4096 value is used, I want to know what is the significance of 4096? If 4096 is indicates page-size then this code is valid or work fine on amd64/x86_64 architecture where page size 4k and it may behave abnormally in case of PPC64 where page size is 64k, basically here I want to discuss the impact of large page size on Julia code and what all other things I need to take into consideration while porting Julia on PPC64le. 2. Past few days I was working on understanding memory management scheme of Julia and I find it bit of difficult and time consuming process though I have some success. I want to know is there any official / unofficial document around which will help me understand it. Any suggestions/pointers on above mention points are much appreciated. -Mahesh On Tue, Aug 18, 2015 at 8:11 PM, Jameson Nash wrote: > It is a considerable performance impact to run with MEMDEBUG, but > otherwise has no side-effects. It is not necessary to run with this flag in > production (and probably not helpful either, since you wouldn't have a > debugger attached). > > > On Tue, Aug 18, 2015 at 9:53 AM Mahesh Waidande < > mahesh.waidande1...@gmail.com> wrote: > >> Hi Jamseson, >> >> Thanks for explaining memory allocations on PPC and providing pointers on >> resolving segmentation fault, pointers are really helpful and I am working >> on those. I am able to compile Julia master branch after turning ‘MEMDEBUG‘ >> flag on from options.h file, compilation went smooth and I am able to see >> the Julia prompt. Although I will continue to work on finding root cause of >> segmentation fault, occur at a time of Julia initialization. >> >> >> I think when we turn on the ‘MEMDEBUG‘ flag it will reduce a performance >> of Julia bit as with MEMDEBUG no memory pools are used and all allocation >> is treated as big. >> >> >> Apart from performance issue, I have few questions in my mind and I would >> like to discuss those, >> 1. Apart from performance hit, is there any other functionality has >> impacted due to turning on ‘MEMDEBUG’ flag OR what are side effects of >> turning ‘MEMDEBUG’ flag on? >> 2. Should I use these settings (turning MEMDEBUG flag on) in production >> environment or in release mode? >> >> >> >> -Mahesh >> >> On Fri, Aug 14, 2015 at 10:03 PM, Jameson Nash wrote: >> >>> It's a JIT copy of a julia function named "new". The last time this >>> error popped up, it was due to an error in the free_page function logic to >>> compute whether it was safe to free the current page (since PPC using large >>> pages). One place to check then is to ensure the invalid pointer hadn't >>> accidentally being deleted by an madvise(DONTNEED) for an unrelated page >>> free operations. >>> >>> Beyond that, I would suggest trying with the `MEMDEBUG` turned on in >>> options.h (which will also disable the `free_page` function). >>> >>> Also, when you have gdb running, there are many more useful things to >>> print than just the backtrace. For starters, I would suggest looking at >>> `disassembly` and `info registers`. Also, go `up` on the stack trace and >>> look at `jl_(f->linfo)`, `jl_(jl_uncompress_ast(f->linfo, f->linfo->ast))`, >>> and `jl_(args[0])` / `jl_(args[1])` >>> >>> >>> On Fri, Aug 14, 2015 at 9:07 AM Mahesh Waidande < >>> mahesh.waidande1...@gmail.com> wrote: >>> >>>> Hi All, >>>> >>>> I am working on building/porting Julia on ppc64le architecture. I am >>>> using Ubuntu 14.10 on top of ppc64le hardware, while compiling Julia >>>> code(master branch) I am getting segmentation fault. I tried to debug >>>> segmentation fault with tools like gdb/vgdb , valgrind , electric-fence >>>> etc. but I not able to find a root cause of it. I need some >>>> help/pointers/suggestions on how I resolve it. >>>> >>>> Here are some details which will help you to diagnose a probl
Re: [julia-users] Segmentation fault during Julia compilation
Hi Jamseson, Thanks for explaining memory allocations on PPC and providing pointers on resolving segmentation fault, pointers are really helpful and I am working on those. I am able to compile Julia master branch after turning ‘MEMDEBUG‘ flag on from options.h file, compilation went smooth and I am able to see the Julia prompt. Although I will continue to work on finding root cause of segmentation fault, occur at a time of Julia initialization. I think when we turn on the ‘MEMDEBUG‘ flag it will reduce a performance of Julia bit as with MEMDEBUG no memory pools are used and all allocation is treated as big. Apart from performance issue, I have few questions in my mind and I would like to discuss those, 1. Apart from performance hit, is there any other functionality has impacted due to turning on ‘MEMDEBUG’ flag OR what are side effects of turning ‘MEMDEBUG’ flag on? 2. Should I use these settings (turning MEMDEBUG flag on) in production environment or in release mode? -Mahesh On Fri, Aug 14, 2015 at 10:03 PM, Jameson Nash wrote: > It's a JIT copy of a julia function named "new". The last time this error > popped up, it was due to an error in the free_page function logic to > compute whether it was safe to free the current page (since PPC using large > pages). One place to check then is to ensure the invalid pointer hadn't > accidentally being deleted by an madvise(DONTNEED) for an unrelated page > free operations. > > Beyond that, I would suggest trying with the `MEMDEBUG` turned on in > options.h (which will also disable the `free_page` function). > > Also, when you have gdb running, there are many more useful things to > print than just the backtrace. For starters, I would suggest looking at > `disassembly` and `info registers`. Also, go `up` on the stack trace and > look at `jl_(f->linfo)`, `jl_(jl_uncompress_ast(f->linfo, f->linfo->ast))`, > and `jl_(args[0])` / `jl_(args[1])` > > > On Fri, Aug 14, 2015 at 9:07 AM Mahesh Waidande < > mahesh.waidande1...@gmail.com> wrote: > >> Hi All, >> >> I am working on building/porting Julia on ppc64le architecture. I am >> using Ubuntu 14.10 on top of ppc64le hardware, while compiling Julia >> code(master branch) I am getting segmentation fault. I tried to debug >> segmentation fault with tools like gdb/vgdb , valgrind , electric-fence >> etc. but I not able to find a root cause of it. I need some >> help/pointers/suggestions on how I resolve it. >> >> Here are some details which will help you to diagnose a problem, >> >> 1. Machine details : >> $ uname -a >> Linux pts00433-vm1 3.16.0-30-generic #40-Ubuntu SMP Mon Jan 12 22:07:11 >> UTC 2015 ppc64le ppc64le ppc64le GNU/Linux >> $ >> >> 2. Snapshot of ‘make debug’ log >> make[1]: Leaving directory '/home/test/Mahesh/julia/julia/base' >> make[1]: Entering directory '/home/test/Mahesh/julia/julia' >> cd base && /home/test/Mahesh/julia/julia/usr/bin/julia-debug -C native >> --output-ji /home/test/Mahesh/julia/julia/usr/lib/julia/inference0.ji -f >> coreimg.jl >> >> Electric Fence 2.2 Copyright (C) 1987-1999 Bruce Perens < >> br...@perens.com> >> Segmentation fault >> Makefile:175: recipe for target >> '/home/test/Mahesh/julia/julia/usr/lib/julia/inference0.ji' failed >> make[1]: *** [/home/test/Mahesh/julia/julia/usr/lib/julia/inference0.ji] >> Error 139 >> make[1]: Leaving directory '/home/test/Mahesh/julia/julia' >> Makefile:64: recipe for target 'julia-inference' failed >> make: *** [julia-inference] Error 2 >> >> 3. gdb stack trace >> test@pts00433-vm1:~/Mahesh/julia/julia/base$ gdb --args >> /home/test/Mahesh/julia/julia/usr/bin/julia-debug -C native --output-ji >> /home/test/Mahesh/julia/julia/usr/lib/julia/inference0.ji -f coreimg.jl >> GNU gdb (Ubuntu 7.8-1ubuntu4) 7.8.0.20141001-cvs >> Copyright (C) 2014 Free Software Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later < >> http://gnu.org/licenses/gpl.html> >> This is free software: you are free to change and redistribute it. >> There is NO WARRANTY, to the extent permitted by law. Type "show copying" >> and "show warranty" for details. >> This GDB was configured as "powerpc64le-linux-gnu". >> Type "show configuration" for configuration details. >> For bug reporting instructions, please see: >> <http://www.gnu.org/software/gdb/bugs/>. >> Find the GDB manual and other documentation resources online at: >> <http://www.gnu.org/software/gdb/documentation/>. >> For help, type "help". >> Type
Re: [julia-users] Julia binary goes in infinite wait on Ubuntu 14.10 (ppc64le)
Hi Jameson Nash, Thanks for reply, pointers provided by you on debugging are really helpful. I am going to try it out and figuring out what more changes I need to make to build Julia on ppc64le architecture. Apologies for late reply as I was a bit occupied. -Mahesh On Wed, Jul 15, 2015 at 11:19 AM, Jameson Nash wrote: > I had at one point fairly recently (back in February, I think) shown that > it should be possible to run julia on the ppc64le hardware. However, ports > are time consuming and difficult to maintain (I don't own a PPC computer > for one thing). Julia Computing could help with a port if interested. > > For community support, there is a dedicated tag in the issue tracker for > helping organize anyone working on that platform ( > https://github.com/JuliaLang/julia/issues?q=is%3Aopen+is%3Aissue+label%3APower) > share issues. > > For debugging tips, I would continue with the general gdb work you were > doing and try to determine why it hasn't blocked in the libuv `uv_run` > function. I recommend using julia-debug so that you can see locals, and > walk the stack, using `print jl_(f->linfo)`, and `print jl_(args[0])`, etc > to figure out where exactly in the program initialization it got stuck. > > > On Fri, Jul 3, 2015 at 10:05 AM Mahesh Waidande < > mahesh.waidande1...@gmail.com> wrote: > >> Hi All, >> >> I am working on building/porting Julia on ppc64le architecture. I am >> using Ubuntu 14.10 on top of ppc64le hardware >> and I am able to build/compile Julia master branch on it. When I execute >> julia binary, it goes into indefinite wait and >> I am not able to view julia prompt .Any pointer on this strange behavior >> would be great help. >> >> Below is some background information which will help to understand/ >> resolve a problem. >> >> 1. Machine details : >> >> test@pts00433-vm1:~/Mahesh/julia/julia$ uname -a >> Linux pts00433-vm1 3.16.0-30-generic #40-Ubuntu SMP Mon Jan 12 22:07:11 >> UTC 2015 ppc64le ppc64le ppc64le GNU/Linux >> test@pts00433-vm1:~/Mahesh/julia/julia$ >> >> >> 2. Content of Make.user file >> >> test@pts00433-vm1:~/Mahesh/julia/julia$ cat Make.user >> include /home/test/Mahesh/julia/julia/Make.powerpc >> >> # __PPC64__ defined but __ppc64__ is not! >> CFLAGS += -D__ppc64__ >> CXXFLAGS += -D__ppc64__ >> LDFLAGS = -llzma >> VERBOSE = 1 >> test@pts00433-vm1:~/Mahesh/julia/julia$ >> >> 3. Content of Make.powerpc file >> >> test@pts00433-vm1:~/Mahesh/julia/julia$ cat Make.powerpc >> # This Make.ppc script should help you get started building Julia on a >> PowerPC machine >> # $ echo "include $(JULIAHOME)/Make.powerpc" >> Make.user >> >> # most of our dependencies these contain m4 scripts that are too old >> (specifically libtool.m4) to know about ppc64le >> # so for now, we must use the system ones >> USE_SYSTEM_FFTW=1 >> USE_SYSTEM_PCRE=0 >> USE_SYSTEM_ARPACK=1 >> USE_SYSTEM_GMP=1 >> USE_SYSTEM_MPFR=1 >> >> # openblas doesn't know about ld version 2 assembly PROLOGUE syntax >> USE_SYSTEM_BLAS=1 >> >> # it's not optimized, and we trust the system libm >> USE_SYSTEM_LIBM=1 >> >> # this one doesn't seem to be able to compile for assembly code >> USE_SYSTEM_LIBUNWIND=1 >> >> # this one takes awhile to build, but it's generally worthwhile -- your >> choice >> USE_SYSTEM_LLVM?=0 >> ifeq ($(USE_SYSTEM_LLVM),1) >> LLVM_CONFIG=llvm-config-3.5 >> else >> LLVM_ASSERTIONS=1 >> LLVM_VER=3.6.0 >> LLVM_ASSERTIONS=1 >> endif >> >> # On a newly allocated machine, you'll need the following dependencies: >> # build essentials: >> # $ sudo apt-get install git g++ gfortran make cmake libedit-dev >> zlib1g-dev libssl-dev >> # system lib replacements: >> # $ sudo apt-get install libfftw3-dev libunwind8-dev libllvm3.5 libblas3 >> libarpack2-dev libedit-dev libpcre3-dev libgmp-dev libmpfr-dev >> # autogen essentials: >> # $ sudo apt-get install libtool autoconf >> test@pts00433-vm1:~/Mahesh/julia/julia$ >> >> >> 4. While compiling below code snippet I was getting segmentation fault. >> For resolving segmentation fault I made changes >> in ‘options.h’ file, I turn on ‘MEMDEBUG’ flag in ‘options.h’ file. >> >> julia/usr/bin/julia -C native --output-o >> /home/test/Mahesh/julia/julia/usr/lib/julia/inference0.o -f coreimg.jl >> >> diff of ‘options.h’ file, >> >> test@pts00433-vm1:~/Mahesh/julia/julia$ git diff src/options.h >> diff --git a/
[julia-users] Julia binary goes in infinite wait on Ubuntu 14.10 (ppc64le)
Hi All, I am working on building/porting Julia on ppc64le architecture. I am using Ubuntu 14.10 on top of ppc64le hardware and I am able to build/compile Julia master branch on it. When I execute julia binary, it goes into indefinite wait and I am not able to view julia prompt .Any pointer on this strange behavior would be great help. Below is some background information which will help to understand/ resolve a problem. 1. Machine details : test@pts00433-vm1:~/Mahesh/julia/julia$ uname -a Linux pts00433-vm1 3.16.0-30-generic #40-Ubuntu SMP Mon Jan 12 22:07:11 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux test@pts00433-vm1:~/Mahesh/julia/julia$ 2. Content of Make.user file test@pts00433-vm1:~/Mahesh/julia/julia$ cat Make.user include /home/test/Mahesh/julia/julia/Make.powerpc # __PPC64__ defined but __ppc64__ is not! CFLAGS += -D__ppc64__ CXXFLAGS += -D__ppc64__ LDFLAGS = -llzma VERBOSE = 1 test@pts00433-vm1:~/Mahesh/julia/julia$ 3. Content of Make.powerpc file test@pts00433-vm1:~/Mahesh/julia/julia$ cat Make.powerpc # This Make.ppc script should help you get started building Julia on a PowerPC machine # $ echo "include $(JULIAHOME)/Make.powerpc" >> Make.user # most of our dependencies these contain m4 scripts that are too old (specifically libtool.m4) to know about ppc64le # so for now, we must use the system ones USE_SYSTEM_FFTW=1 USE_SYSTEM_PCRE=0 USE_SYSTEM_ARPACK=1 USE_SYSTEM_GMP=1 USE_SYSTEM_MPFR=1 # openblas doesn't know about ld version 2 assembly PROLOGUE syntax USE_SYSTEM_BLAS=1 # it's not optimized, and we trust the system libm USE_SYSTEM_LIBM=1 # this one doesn't seem to be able to compile for assembly code USE_SYSTEM_LIBUNWIND=1 # this one takes awhile to build, but it's generally worthwhile -- your choice USE_SYSTEM_LLVM?=0 ifeq ($(USE_SYSTEM_LLVM),1) LLVM_CONFIG=llvm-config-3.5 else LLVM_ASSERTIONS=1 LLVM_VER=3.6.0 LLVM_ASSERTIONS=1 endif # On a newly allocated machine, you'll need the following dependencies: # build essentials: # $ sudo apt-get install git g++ gfortran make cmake libedit-dev zlib1g-dev libssl-dev # system lib replacements: # $ sudo apt-get install libfftw3-dev libunwind8-dev libllvm3.5 libblas3 libarpack2-dev libedit-dev libpcre3-dev libgmp-dev libmpfr-dev # autogen essentials: # $ sudo apt-get install libtool autoconf test@pts00433-vm1:~/Mahesh/julia/julia$ 4. While compiling below code snippet I was getting segmentation fault. For resolving segmentation fault I made changes in ‘options.h’ file, I turn on ‘MEMDEBUG’ flag in ‘options.h’ file. julia/usr/bin/julia -C native --output-o /home/test/Mahesh/julia/julia/usr/lib/julia/inference0.o -f coreimg.jl diff of ‘options.h’ file, test@pts00433-vm1:~/Mahesh/julia/julia$ git diff src/options.h diff --git a/src/options.h b/src/options.h index 23540ae..e2318cb 100644 --- a/src/options.h +++ b/src/options.h @@ -40,7 +40,7 @@ // with MEMDEBUG, every object is allocated explicitly with malloc, and // filled with 0xbb before being freed. this helps tools like valgrind // catch invalid accesses. -// #define MEMDEBUG +#define MEMDEBUG // GC_VERIFY force a full verification gc along with every quick gc to ensure no // reachable memory is freed test@pts00433-vm1:~/Mahesh/julia/julia$ 5. I attached gdb to Julia and try to find out the reason why I am not getting Julia command prompt. Below is stacktrace of gdb. test@pts00433-vm1:~/Mahesh/julia/julia$ gdb ./julia (gdb) r Starting program: /home/test/Mahesh/julia/julia/julia [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1". _ _ _ _(_)_ | A fresh approach to technical computing (_) | (_) (_)| Documentation: http://docs.julialang.org _ _ _| |_ __ _ | Type "help()" for help. | | | | | | |/ _` | | | | |_| | | | (_| | | Version 0.4.0-dev+5769 (2015-07-03 04:36 UTC) _/ |\__'_|_|_|\__'_| | Commit 2c1ac18* (0 days old master) |__/ | powerpc64le-linux-gnu ^C Program received signal SIGINT, Interrupt. 0x3fffb6a138fc in __pause_nocancel () at ../sysdeps/unix/syscall-template.S:81 81 ../sysdeps/unix/syscall-template.S: No such file or directory. (gdb) bt #0 0x3fffb6a138fc in __pause_nocancel () at ../sysdeps/unix/syscall-template.S:81 #1 0x3fffb4f5fb10 in julia_wait_18019 () at task.jl:302 #2 0x3fffb4f69bc4 in julia_wait_18197 (c=) at task.jl:225 #3 0x3fffb6c00330 in jl_apply (nargs=1, args=0x3fffef38, f=0x12c5e370) at julia.h:1260 #4 jl_trampoline (F=0x12c5e370, args=0x3fffef38, nargs=) at builtins.c:975 #5 0x3fffb6bf1b00 in jl_apply (nargs=1, args=0x3fffef38, f=) at julia.h:1260 #6 jl_apply_generic (F=0x12c59b00, args=0x3fffef38, nargs=) at gf.c:1632 #7 0x3fffb2c60088 in julia_anonymous_24015 () #8 0x3fffb6c00330 in jl_apply (nargs=0, args=0x0, f=0x12a77d30) at julia.h:1260 #9 jl_trampoline (F=0x12a77d30, args=0x0, nargs=) at builtins.c:975 #10 0x3fffb6b04
Re: [julia-users] link time error while running make utility on ppc64le
Hi isaiah, Thanks for quick reply, links provided by you are really helpful. I am going through it and figuring out what changes I need to make to build Julia on ppc64le architecture. Apologies for late reply as I was a bit occupied. -Mahesh On Mon, Jun 22, 2015 at 8:33 PM, Isaiah Norton wrote: > PPC is not supported at all on release-0.3, but there was some significant > work on PPC quite recently on master. Julia is not regularly tested on PPC > yet, so mileage may vary, but master will be a much better starting point. > See > > https://github.com/JuliaLang/julia/blob/master/Make.powerpc > and > https://groups.google.com/forum/#!topic/julia-dev/BYVCyUlNR8c > > On Mon, Jun 22, 2015 at 9:42 AM, Mahesh Waidande < > mahesh.waidande1...@gmail.com> wrote: > >> Hi Folks, >> >> >> I am building latest stable release of Julia (release-0.3) on ppc64le >> architecture. >> >> I am running Ubuntu 14.10 on ppc64le hardware. I am getting below link >> time errors while running make utility. >> >> >> Here are the error details, >> >> ppc64-test-altivec.o: In function `vec_stack': >> >> ppc64-test-altivec.c:(.text+0x13c): undefined reference to >> `_Uppc64_init_local' >> >> ppc64-test-altivec.c:(.text+0x194): undefined reference to >> `_Uppc64_get_reg' >> >> ppc64-test-altivec.c:(.text+0x204): undefined reference to >> `_Uppc64_get_reg' >> >> ppc64-test-altivec.c:(.text+0x274): undefined reference to >> `_Uppc64_get_fpreg' >> >> ppc64-test-altivec.c:(.text+0x2e0): undefined reference to >> `_Uppc64_regname' >> >> ppc64-test-altivec.c:(.text+0x3d4): undefined reference to >> `_Uppc64_get_proc_name' >> >> ppc64-test-altivec.c:(.text+0x470): undefined reference to `_Uppc64_step' >> >> ppc64-test-altivec.c:(.text+0x4a0): undefined reference to >> `_Uppc64_get_reg' >> >> collect2: error: ld returned 1 exit status >> >> Makefile:788: recipe for target 'ppc64-test-altivec' failed >> >> >> >> Machine/Hardware details (output of uname -a ) >> >> Linux vm1 3.16.0-23-generic #31-Ubuntu SMP Tue Oct 21 17:55:08 UTC 2014 >> ppc64le ppc64le ppc64le GNU/Linux >> >> >> >> I google around above error and try to apply some patches but no luck >> still I am getting same errors. Any pointer on above error will be great >> help. >> >> >> >> Thanks >> >> -Mahesh >> > >
[julia-users] link time error while running make utility on ppc64le
Hi Folks, I am building latest stable release of Julia (release-0.3) on ppc64le architecture. I am running Ubuntu 14.10 on ppc64le hardware. I am getting below link time errors while running make utility. Here are the error details, ppc64-test-altivec.o: In function `vec_stack': ppc64-test-altivec.c:(.text+0x13c): undefined reference to `_Uppc64_init_local' ppc64-test-altivec.c:(.text+0x194): undefined reference to `_Uppc64_get_reg' ppc64-test-altivec.c:(.text+0x204): undefined reference to `_Uppc64_get_reg' ppc64-test-altivec.c:(.text+0x274): undefined reference to `_Uppc64_get_fpreg' ppc64-test-altivec.c:(.text+0x2e0): undefined reference to `_Uppc64_regname' ppc64-test-altivec.c:(.text+0x3d4): undefined reference to `_Uppc64_get_proc_name' ppc64-test-altivec.c:(.text+0x470): undefined reference to `_Uppc64_step' ppc64-test-altivec.c:(.text+0x4a0): undefined reference to `_Uppc64_get_reg' collect2: error: ld returned 1 exit status Makefile:788: recipe for target 'ppc64-test-altivec' failed Machine/Hardware details (output of uname -a ) Linux vm1 3.16.0-23-generic #31-Ubuntu SMP Tue Oct 21 17:55:08 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux I google around above error and try to apply some patches but no luck still I am getting same errors. Any pointer on above error will be great help. Thanks -Mahesh
