[julia-users] Merging dataframes
Is there a way to avoid copying when doing vcat(df1::DataFrame, df2::DataFrame, …)? I’m trying to open hundreds of files with DataFrames, merge all of them and save a single ~150M row x 100 col DataFrame using HDF5 and JLD (to be opened later using mmap), and it seems to work marvelously, apart from the vcat. Does a no-copy option exist? I’m aware of DataStreams as a concept, but as I understand, they’re not fully fleshed out yet.
Re: [julia-users] Merging dataframes
Thanks! It seems to work quite well using append!(d1::DataArray, d2::DataArray) from DataArrays.jl trunk. If I had a time machine to extend the Sunday, I’d work on a proper version of vcat using that, but alas. I appreciate the amazing work that has gone into Julia, HDF5.jl and DataFrames.jl. :) On 26 Jan 2014, at 21:55, John Myles White wrote: > This is quite close to being possible, but we’re missing a few things. > > Daniel Jones recently added an append! method to DataArrays, which would let > you do this column-by-column. > > To help you out, we need to add an append! method to DataFrames as well. I’ve > wanted that badly myself lately. > > I will try to get to this today, but am already pretty overwhelmed with work > for the day. > > — John > > On Jan 26, 2014, at 11:02 AM, Joosep Pata wrote: > >> Is there a way to avoid copying when doing vcat(df1::DataFrame, >> df2::DataFrame, …)? I’m trying to open hundreds of files with DataFrames, >> merge all of them and save a single ~150M row x 100 col DataFrame using HDF5 >> and JLD (to be opened later using mmap), and it seems to work marvelously, >> apart from the vcat. >> Does a no-copy option exist? I’m aware of DataStreams as a concept, but as I >> understand, they’re not fully fleshed out yet. >
[julia-users] evaluate function on DataFrame row
I would like to do an explicit loop over a large DataFrame and evaluate a function which depends on a subset of the columns in an arbitrary way. What would be the fastest way to accomplish this? Presently, I’m doing something like ~~~ f(df::DataFrame, i::Integer) = df[i, :a] * df[i, :b] + df[i, :c] for i=1:nrow(df) x = f(df, i) end ~~~ which according to Profile creates a major bottleneck. Would it make sense to somehow pre-create an immutable type corresponding to a single row (my data are BitsKind), and run a compiled function on these row-objects with strong typing? Thanks in advance for any advice, Joosep
Re: [julia-users] evaluate function on DataFrame row
Thanks! I wasn’t aware of eachrow, this seems quite close to what I had in mind. I ran some simplistic timing checks [1], and the eachrow method is 2-3x faster. I also tried the type asserts, byt they didn’t seem to make a difference. I forgot to mention earlier that my data can also be NA, so it’s not that easy for the compiler. [1] http://nbviewer.ipython.org/urls/dl.dropbox.com/s/mj8g1s0ewmpd1b6/dataframe_iter_speed.ipynb?create=1 Cheers, Joosep On 01 Feb 2014, at 15:11, David van Leeuwen wrote: > Hi, > > There now is the eachrow iterator which might do what you want more > efficiently. > > df = DataFrame(a=1:2, b=2:3) > func(r::DataFrameRow) = r["a"] * r["b"] > for r in eachrow(df) >println(func(r)) > end > you can also use integer indices for the dataframerow r, r[1] * r[2] > > Cheers, > > ---david > > On Saturday, February 1, 2014 1:25:04 PM UTC+1, Joosep Pata wrote: > I would like to do an explicit loop over a large DataFrame and evaluate a > function which depends on a subset of the columns in an arbitrary way. What > would be the fastest way to accomplish this? Presently, I’m doing something > like > > ~~~ > f(df::DataFrame, i::Integer) = df[i, :a] * df[i, :b] + df[i, :c] > > for i=1:nrow(df) > x = f(df, i) > end > ~~~ > > which according to Profile creates a major bottleneck. > > Would it make sense to somehow pre-create an immutable type corresponding to > a single row (my data are BitsKind), and run a compiled function on these > row-objects with strong typing? > > Thanks in advance for any advice, > Joosep
Re: [julia-users] evaluate function on DataFrame row
Thanks everyone. The isna(DataArray, Integer) method will certainly be useful. On 02 Feb 2014, at 01:19, John Myles White wrote: > Hi Joosep, > > The current way to get the best performance for your example is going to look > something like the following code: > > a, b, c = df[:a], df[:b], df[:c] > T = eltype(a) # Assumes eltype(a) == eltype(b) == eltype(c) > for i in 1:size(df, 1) > if !(isna(a, i) || isna(b, i) || isna(c, i)) > x = a[i]::T * b[i]::T + c[i]::T > end > end > > This should resolve all type inference problems and get you the highest > performance possible given our current codebase. > > In general, there are three main bottlenecks when working with > DataArrays/DataFrames that we’re trying to find ways to work around: > > (1) Indexing into the full DataFrame on each iteration is slightly wasteful, > since you can cache the identity of the “:a” column before starting the > loop’s body. This is a fairly minor speedup, but it’s still worth doing. > > (2) Checking whether entries are NA is currently very wasteful because the > entries, absent type assertions, are of uncertain type. The new > isna(DataArray, Integer) method lets you check, per-entry, whether that > specific entry is NA. This is much faster because it’s completely well-typed > as only Bool’s ever result from this computation. > > (3) Indexing into a DataArray right now isn’t type-stable, because the > outcome might be a value of type T or it might be an NA of type NAType. > Adding the type assertions mentioned by Ivar will help a lot with this, once > you guarantee that there are no NA’s in the DataArray. > > - John > >> f(df::DataFrame, i::Integer) = df[i, :a] * df[i, :b] + df[i, :c] >> > >> > for i=1:nrow(df) >> > x = f(df, i) >> > end >> > ~~~ >> > >> > which according to Profile creates a major bottleneck. >> > >> > Would it make sense to somehow pre-create an immutable type corresponding >> > to a single row (my data are BitsKind), and run a compiled function on >> > these row-objects with strong typing? >> > >> > Thanks in advance for any advice, >> > Joosep >> > > — John > > > On Feb 1, 2014, at 6:28 AM, David van Leeuwen > wrote: > >> Hi, >> >> I saw you define a function f(::DataFrameRow) inside the timing loop. I >> wonder whether the Julia JIT re-compiles this local function each time, or >> whether it caches the compiled version. I don't really know. >> >> Apparently there is a performance penalty for anonymous functions, as in >> map(x->x*x, i:10), but I don't know if this extends to locally defined >> functions. >> >> Cheers, >> >> ---david >> >> On Saturday, February 1, 2014 3:08:18 PM UTC+1, Joosep Pata wrote: >> Thanks! >> >> I wasn’t aware of eachrow, this seems quite close to what I had in mind. I >> ran some simplistic timing checks [1], and the eachrow method is 2-3x >> faster. I also tried the type asserts, byt they didn’t seem to make a >> difference. I forgot to mention earlier that my data can also be NA, so it’s >> not that easy for the compiler. >> >> [1] >> http://nbviewer.ipython.org/urls/dl.dropbox.com/s/mj8g1s0ewmpd1b6/dataframe_iter_speed.ipynb?create=1 >> >> >> Cheers, >> Joosep >> >> On 01 Feb 2014, at 15:11, David van Leeuwen wrote: >> >> > Hi, >> > >> > There now is the eachrow iterator which might do what you want more >> > efficiently. >> > >> > df = DataFrame(a=1:2, b=2:3) >> > func(r::DataFrameRow) = r["a"] * r["b"] >> > for r in eachrow(df) >> >println(func(r)) >> > end >> > you can also use integer indices for the dataframerow r, r[1] * r[2] >> > >> > Cheers, >> > >> > ---david >> > >> > On Saturday, February 1, 2014 1:25:04 PM UTC+1, Joosep Pata wrote: >> > I would like to do an explicit loop over a large DataFrame and evaluate a >> > function which depends on a subset of the columns in an arbitrary way. >> > What would be the fastest way to accomplish this? Presently, I’m doing >> > something like >> > >> > ~~~ >> > f(df::DataFrame, i::Integer) = df[i, :a] * df[i, :b] + df[i, :c] >> > >> > for i=1:nrow(df) >> > x = f(df, i) >> > end >> > ~~~ >> > >> > which according to Profile creates a major bottleneck. >> > >> > Would it make sense to somehow pre-create an immutable type corresponding >> > to a single row (my data are BitsKind), and run a compiled function on >> > these row-objects with strong typing? >> > >> > Thanks in advance for any advice, >> > Joosep >> >
Re: [julia-users] Is there a literal way to make a Symbol separated with space? Ex. [ :test TEST ]
julia> symbol("test test")
:test test
On 07 Feb 2014, at 18:25, Ismael VC wrote:
> julia> a = :test
> :test
>
> julia> typeof(a)
> Symbol
>
> julia> b = :test test
> ERROR: syntax: extra token "test" after end of expression
>
> julia> b = :"test test" # Ruby style?
> "test test"
>
> julia> typeof(b) # Not a Symbol?
> ASCIIString (constructor with 1 method)
>
> julia> b = symbol(b) # How to write it without calling symbol() function.
> :test test
>
> julia> typeof(b)
> Symbol
>
>
> I just red that in Ruby you use:
>
> :"test test"
>
> in Smalltalk:
>
> #'test test'
>
> and in Lisp:
>
> |test test|
>
Re: [julia-users] Is there a literal way to make a Symbol separated with space? Ex. [ :test TEST ]
Oops, sorry, I was selectively blind.
Joosep
On 07 Feb 2014, at 19:09, Eric Davies wrote:
> Ismael mentioned that in his original post. He is looking for a way to do
> that but with a literal syntax (no intermediate string+functioncall).
>
> On Friday, 7 February 2014 12:06:56 UTC-6, Joosep Pata wrote:
> julia> symbol("test test")
> :test test
>
> On 07 Feb 2014, at 18:25, Ismael VC wrote:
>
> > julia> a = :test
> > :test
> >
> > julia> typeof(a)
> > Symbol
> >
> > julia> b = :test test
> > ERROR: syntax: extra token "test" after end of expression
> >
> > julia> b = :"test test" # Ruby style?
> > "test test"
> >
> > julia> typeof(b) # Not a Symbol?
> > ASCIIString (constructor with 1 method)
> >
> > julia> b = symbol(b) # How to write it without calling symbol() function.
> > :test test
> >
> > julia> typeof(b)
> > Symbol
> >
> >
> > I just red that in Ruby you use:
> >
> > :"test test"
> >
> > in Smalltalk:
> >
> > #'test test'
> >
> > and in Lisp:
> >
> > |test test|
> >
>
[julia-users] default argument values using Clang.jl
Has anyone tried extracting default argument values for a function from a C header using Clang.jl? In general, I find the package really great and easy to use (thanks Isaiah). Thanks, Joosep
[julia-users] macro for generating functions with optional arguments
Hi,
I’m trying to write a macro that would generate functions with optional
arguments. However, I can’t figure out why the following code behaves as it
does. I’d appreciate if someone told me what I’m doing wrong.
~~~
#1) def. value in quote, works
ex = :(x)
q = quote
function f1($ex=1)
x
end
end
macroexpand(q)|>println
eval(q)
f1()|>println
#2) def. value in expression, does not work
ex = :(x=1)
q = quote
function f2($ex)
x
end
end
println("does not work")
macroexpand(q)|>println
eval(q)
f2()|>println
~~~
1) does not allow me to conveniently construct a list of arguments with values,
whereas
2) gives me
> ERROR: syntax: "x=1" is not a valid function argument name
The macroexpand of either expression looks identical, except for spaces around
the “=“ sign for 2), which should not make a syntactical difference.
cheers, Joosep
Re: [julia-users] Re: What is a Number?
Similarly, I have wondered what methods should an AbstractDataFrame
implement. Do we have a mechanism of specifying the interface of an
abstract type?
24.02.2014 15:00 kirjutas kuupäeval "Johan Sigfrids" <
johan.sigfr...@gmail.com>:
> This is actually a really good question. I found myself wondering the same
> thing the other day.
>
> On Monday, February 24, 2014 1:54:59 PM UTC+2, andrew cooke wrote:
>>
>> Working on the finite field code I found myself asking "what is a
>> Number?".
>>
>> One answer is:
>>
>> julia> Base.subtypetree(Number)
>> (Number,{(Complex{Float16},{}),(Complex{Float32},{}),(Complex{Float64
>> },{}),(Complex{T<:Real},{}),(Real,{(FloatingPoint,{(BigFloat,{}),(Float16
>> ,{}),(Float32,{}),(Float64,{})}),(Integer,{(BigInt,{}),(Bool,{}),(Char
>> ,{}),(Signed,{(Int128,{}),(Int16,{}),(Int32,{}),(Int64,{}),(Int8,{})}),(
>> Unsigned,{(Uint128,{}),(Uint16,{}),(Uint32,{}),(Uint64,{}),(Uint8
>> ,{})})}),(MathConst{sym},{}),(Rational{T<:Integer},{})})})
>>
>> but that doesn't help so much. What I really wanted to know is - what
>> methods are assumed to exist for something that is a subtype of Number?
>>
>> And I don't know how to answer that.
>>
>> Maybe (I don't think so) Julia needs some kind of concept like abstract
>> methods, where you can name methods for Number that any subtype must
>> implement?
>>
>> Maybe there needs to be some kind of tool that introspects the code base
>> and says "90% of subtypes define real and abs"?
>>
>> Maybe this has already been discussed or is clearly not an issue?
>>
>> Andrew
>>
>
[julia-users] Process called via 'run' does not always terminate
Hi, I’m running various external processes via ‘run(cmd)’, but strangely, occasionally ‘run’ seems to hang despite the external program finishing. It does not seem to be entirely predictable, but it’s quite persistent, and I don’t have a good test case yet. I was wondering if someone had an idea what could cause this so that I could try to isolate it. Cheers, Joosep
Re: [julia-users] Process called via 'run' does not always terminate
Thanks for the reply. >> What system are you on? > julia> versioninfo() > Julia Version 0.3.0-prerelease+2177 > Commit 7d475bf* (2014-03-27 01:39 UTC) > Platform Info: > System: Darwin (x86_64-apple-darwin13.1.0) > CPU: Intel(R) Core(TM) i7-2677M CPU @ 1.80GHz > WORD_SIZE: 64 > BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY) > LAPACK: libopenblas > LIBM: libopenlibm I’ll try to reproduce it on an SLC6 box. Joosep On 28 Mar 2014, at 17:13, Stefan Karpinski wrote: > > > On Fri, Mar 28, 2014 at 8:11 AM, Joosep Pata wrote: > Hi, > > I’m running various external processes via ‘run(cmd)’, but strangely, > occasionally ‘run’ seems to hang despite the external program finishing. It > does not seem to be entirely predictable, but it’s quite persistent, and I > don’t have a good test case yet. > I was wondering if someone had an idea what could cause this so that I could > try to isolate it. > > Cheers, > Joosep >
[julia-users] duplicated inline plots in IJulia
Hi, Is anyone else seeing duplicated output from PyPlot in conjunction with IJulia in inline mode recently? This does not appear with Gadfly, nor with pyplot in IPython. Cheers, Joosep
[julia-users] precompile EACCES error
I can't seem to load DataFrames due to a precompilation issue ~~~ julia> using DataFrames INFO: Precompiling module DataFrames... ERROR: could not spawn `/Users/joosep/Documents/julia/usr/lib/julia -Cnative -J/Users/joosep/Documents/julia/usr/lib/julia/sys.dylib --output-ji /Users/joosep/.julia/lib/v0.4/DataFrames.ji --output-incremental=yes --startup-file=no --history-file=no --eval 'while !eof(STDIN) eval(Main, deserialize(STDIN)) end '`: permission denied (EACCES) in _jl_spawn at process.jl:226 ~~~ This is a clean build, but maybe I have missed something - has anyone else seen this? I'm using ~~~ - DataFrames0.6.9 julia> versioninfo() Julia Version 0.4.0-dev+6723 Commit b59b0dd* (2015-08-14 08:12 UTC) Platform Info: System: Darwin (x86_64-apple-darwin14.3.0) CPU: Intel(R) Core(TM) i5-4278U CPU @ 2.60GHz WORD_SIZE: 64 BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell) LAPACK: libopenblas LIBM: libopenlibm LLVM: libLLVM-3.3 ~~~
[julia-users] Re: precompile EACCES error
Turns out the issue was setting JULIA_HOME=/Users/joosep/Documents/julia/usr/lib/ which caused the precompiler to look for the julia binary in the wrong place. On Wednesday, 19 August 2015 13:27:12 UTC+2, Joosep Pata wrote: > > I can't seem to load DataFrames due to a precompilation issue > > ~~~ > julia> using DataFrames > INFO: Precompiling module DataFrames... > ERROR: could not spawn `/Users/joosep/Documents/julia/usr/lib/julia > -Cnative -J/Users/joosep/Documents/julia/usr/lib/julia/sys.dylib > --output-ji /Users/joosep/.julia/lib/v0.4/DataFrames.ji > --output-incremental=yes --startup-file=no --history-file=no --eval 'while > !eof(STDIN) > eval(Main, deserialize(STDIN)) > end > '`: permission denied (EACCES) > in _jl_spawn at process.jl:226 > ~~~ > > This is a clean build, but maybe I have missed something - has anyone else > seen this? > > I'm using > ~~~ > - DataFrames0.6.9 > julia> versioninfo() > Julia Version 0.4.0-dev+6723 > Commit b59b0dd* (2015-08-14 08:12 UTC) > Platform Info: > System: Darwin (x86_64-apple-darwin14.3.0) > CPU: Intel(R) Core(TM) i5-4278U CPU @ 2.60GHz > WORD_SIZE: 64 > BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell) > LAPACK: libopenblas > LIBM: libopenlibm > LLVM: libLLVM-3.3 > ~~~ >
[julia-users] C interop: pointer to elementary julia type
I want to modify the data stored in an elementary julia object (e.g. Int64)
using external C code. I can do this if the type is wrapped in an array, but is
it possible to do something like `pointer(x::Float64) -> address of data in x`
with the memory being guaranteed by julia?
Here’s a short snippet to describe:
~~~
#C side
void myfunc(long* x) {
x[0] += 1;
}
#julia side, works
x = Int64[0]
ccall((:myfunc, lib), Void, (Ptr{Void}, ), convert(Ptr{Void}, pointer(x)))
println(x)
>> [1]
#julia side, desired
x = 0
ccall((:myfunc, lib), Void, (Ptr{Void}, ), pointer(x))
println(x)
>> 1
~~~
I tried pointer_from_objref but as I understand this gives me the pointer of
the whole julia type with meta-data etc. If I write to this address in C, I
fear I’ll have problems.
Re: [julia-users] C interop: pointer to elementary julia type
Thanks Jameson,
I want to later use `x` in auto-generated expressions, and if it's a Ref I
will have to use `x.x` to get the actual data which makes generating the
expressions more tricky, I thought I could perhaps avoid it.
Here's a more complete snippet:
~~~
type Foo
x::Int64
y::Vector{Int64}
end
Foo() = Foo(0, zeros(Int64, 10))
bar = :(f.x + f.y[2])
expr = :(
f = Foo()
for i=1:10
ccall((:myfunc_x, lib), Void, (Ptr{Void}, ), pointer(h.x)) #fill the
scalar f.x
ccall((:myfunc_y, lib), Void, (Ptr{Void}, ), pointer(h.y)) #fill the
array f.y
r = $(eval(bar)) #do something with f.x and f.y
end
#done with f, OK to clean up
)
~~~
Does this seem like an OK approach with Refs and is there a way to avoid
needing to do `h.x.x` in the expressions?
On Tuesday, 1 September 2015 23:51:34 UTC+2, Jameson wrote:
>
> use `Ref{Clong}` for the calling convention to pass an auto-allocated
> boxed c-long.
>
> it's usually best to avoid `pointer` and `pointer_from_objref`. memory is
> not "guaranteed by julia": it will be cleaned up as soon as the gc detects
> that you are no longer referencing the julia object. using `pointer` and
> `pointer_from_objref` hides memory from the gc, making it easier for you to
> create memory bugs in your program.
>
>
> On Tue, Sep 1, 2015 at 5:46 PM Joosep Pata > wrote:
>
>> I want to modify the data stored in an elementary julia object (e.g.
>> Int64) using external C code. I can do this if the type is wrapped in an
>> array, but is it possible to do something like `pointer(x::Float64) ->
>> address of data in x` with the memory being guaranteed by julia?
>>
>> Here’s a short snippet to describe:
>> ~~~
>>
>> #C side
>> void myfunc(long* x) {
>> x[0] += 1;
>> }
>>
>> #julia side, works
>> x = Int64[0]
>> ccall((:myfunc, lib), Void, (Ptr{Void}, ), convert(Ptr{Void}, pointer(x)))
>> println(x)
>> >> [1]
>>
>> #julia side, desired
>> x = 0
>> ccall((:myfunc, lib), Void, (Ptr{Void}, ), pointer(x))
>> println(x)
>> >> 1
>> ~~~
>>
>> I tried pointer_from_objref but as I understand this gives me the pointer
>> of the whole julia type with meta-data etc. If I write to this address in
>> C, I fear I’ll have problems.
>
>
[julia-users] RFC: ROOT.jl, wrappers for the ROOT library from CERN
Hello, I’ve compiled a set of wrappers for the ROOT libraries used at CERN for storing the data from the Large Hadron Collider. https://github.com/jpata/ROOT.jl https://github.com/JuliaLang/METADATA.jl/pull/3550 These are a stop-gap until they can be ported to Keno’s Cxx.jl, but they are already useful since a few years, so I’d like to ask for more sets of eyes to look at them. Cheers, Joosep
[julia-users] embedding: julia repl as a shared library
Hi, I'd like to compile ui/repl.c into a shared library so that I could dlopen julia after some other initialization procedures that would otherwise conflict with the LLVM linked to julia. I succeeded in doing that on OSX using: ~~~ diff --git a/ui/Makefile b/ui/Makefile +julia-release: $(build_bindir)/julia$(EXE) $(build_private_libdir)/librepl.$(SHLIB_EXT) ... +$(build_private_libdir)/librepl.$(SHLIB_EXT): $(OBJS) + @$(call PRINT_LINK, $(CXXLD) -shared $(CXXFLAGS) $(CXXLDFLAGS) $(LINK_FLAGS) $(SHIPFLAGS) $^ -o $@ -L$(build_private_libdir) -L$(build_libdir) -L$(build_shlibdir) ~~~ so I can call julia dynamically as ~~~ my_init(); // initalize stuff that hides its LLVM symbols after loading ... void* handle_julia = dlopen(LIBJULIAREPL, RTLD_NOW | RTLD_GLOBAL); ... typedef int (*t_jl_main)(int, char**); t_jl_main jl_main = (t_jl_main)dlsym(handle_julia, "main"); return jl_main(argc, argv); ~~~ On linux, I get strange linker errors: `/usr/bin/ld: repl.o: relocation R_X86_64_TPOFF32 against `tls_states.12084' can not be used when making a shared object; recompile with -fPIC` As far as I can tell, julia uses fPIC throughout. Has anyone encountered something like this before? Google links to some old gcc bugs and a go linker issue but it's not evident if there is a fix. Cheers, Joosep
Re: [julia-users] embedding: julia repl as a shared library
I'd like to not re-implement all the REPL boiler-plate, like
~~~
ios_puts("\njulia> ", ios_stdout);
ios_flush(ios_stdout);
line = ios_readline(ios_stdin);
~~~
and so on.
In effect, I want to launch the usual julia REPL, but call some of my own
initialization procedures before julia_init.
My motivation is that I want to call an external library that dies horribly
due to the LLVM symbols present if loaded after julia_init is called, see
also https://github.com/JuliaLang/julia/issues/12644.
The only way I've managed to do it is to recompile the julia binary, I
figured I could re-use the repl code by just dynamically loading it.
On Sunday, 24 July 2016 19:55:00 UTC+2, Yichao Yu wrote:
>
> On Sun, Jul 24, 2016 at 1:21 PM, Joosep Pata > wrote:
> > Hi,
> >
> > I'd like to compile ui/repl.c into a shared library so that I could
> dlopen julia after some other initialization procedures that would
> otherwise conflict with the LLVM linked to julia.
>
> You should **NOT** compile `ui/repl.c` since it will fail as you saw.
> You should just use `libjulia.so`, why is it not enough? `ui/repl.c`
> is just a very thin wrapper and should have nothing to do with LLVM or
> whatever conflict you saw.
>
> >
> > I succeeded in doing that on OSX using:
> >
> > ~~~
> > diff --git a/ui/Makefile b/ui/Makefile
> > +julia-release: $(build_bindir)/julia$(EXE)
> $(build_private_libdir)/librepl.$(SHLIB_EXT)
> > ...
> > +$(build_private_libdir)/librepl.$(SHLIB_EXT): $(OBJS)
> > + @$(call PRINT_LINK, $(CXXLD) -shared $(CXXFLAGS) $(CXXLDFLAGS)
> $(LINK_FLAGS) $(SHIPFLAGS) $^ -o $@ -L$(build_private_libdir)
> -L$(build_libdir) -L$(build_shlibdir)
> > ~~~
> >
> > so I can call julia dynamically as
> > ~~~
> > my_init(); // initalize stuff that hides its LLVM symbols after loading
> > ...
> > void* handle_julia = dlopen(LIBJULIAREPL, RTLD_NOW | RTLD_GLOBAL);
> > ...
> > typedef int (*t_jl_main)(int, char**);
> > t_jl_main jl_main = (t_jl_main)dlsym(handle_julia, "main");
> > return jl_main(argc, argv);
> > ~~~
> >
> > On linux, I get strange linker errors:
> > `/usr/bin/ld: repl.o: relocation R_X86_64_TPOFF32 against
> `tls_states.12084' can not be used when making a shared object; recompile
> with -fPIC`
> >
> > As far as I can tell, julia uses fPIC throughout. Has anyone encountered
> something like this before? Google links to some old gcc bugs and a go
> linker issue but it's not evident if there is a fix.
> >
> > Cheers,
> > Joosep
>
Re: [julia-users] embedding: julia repl as a shared library
Right, thanks for the tip. To confirm: `ui/repl.c` is still the code that
gets compiled to the julia(-debug) binary, right?
If I call "Base._start()" via libjulia, I still need to reproduce the usual
argv logic of the julia binary.
I'll just patch `repl.c` to my needs then without changing the linking,
it's dirty but seems better that re-implementing.
On Sunday, 24 July 2016 20:43:54 UTC+2, Yichao Yu wrote:
>
> On Sun, Jul 24, 2016 at 2:39 PM, Yichao Yu >
> wrote:
> > On Sun, Jul 24, 2016 at 2:37 PM, Joosep Pata > wrote:
> >> I'd like to not re-implement all the REPL boiler-plate, like
> >> ~~~
> >> ios_puts("\njulia> ", ios_stdout);
> >> ios_flush(ios_stdout);
> >> line = ios_readline(ios_stdin);
> >> ~~~
> >> and so on.
> >
> > That's not the repl and you don't need to implement that.
>
> The only thing you need to do to get a repl after initialization is to
> call `Base._start()`. Simply `jl_eval_string("Base._start()")` should
> work.
>
> >
> >>
> >> In effect, I want to launch the usual julia REPL, but call some of my
> own
> >
> > Which is **NOT** in the repl.c
> >
> >> initialization procedures before julia_init.
> >> My motivation is that I want to call an external library that dies
> horribly
> >> due to the LLVM symbols present if loaded after julia_init is called,
> see
> >> also https://github.com/JuliaLang/julia/issues/12644.
> >> The only way I've managed to do it is to recompile the julia binary, I
> >> figured I could re-use the repl code by just dynamically loading it.
> >>
> >> On Sunday, 24 July 2016 19:55:00 UTC+2, Yichao Yu wrote:
> >>>
> >>> On Sun, Jul 24, 2016 at 1:21 PM, Joosep Pata
> wrote:
> >>> > Hi,
> >>> >
> >>> > I'd like to compile ui/repl.c into a shared library so that I could
> >>> > dlopen julia after some other initialization procedures that would
> otherwise
> >>> > conflict with the LLVM linked to julia.
> >>>
> >>> You should **NOT** compile `ui/repl.c` since it will fail as you saw.
> >>> You should just use `libjulia.so`, why is it not enough? `ui/repl.c`
> >>> is just a very thin wrapper and should have nothing to do with LLVM or
> >>> whatever conflict you saw.
> >>>
> >>> >
> >>> > I succeeded in doing that on OSX using:
> >>> >
> >>> > ~~~
> >>> > diff --git a/ui/Makefile b/ui/Makefile
> >>> > +julia-release: $(build_bindir)/julia$(EXE)
> >>> > $(build_private_libdir)/librepl.$(SHLIB_EXT)
> >>> > ...
> >>> > +$(build_private_libdir)/librepl.$(SHLIB_EXT): $(OBJS)
> >>> > + @$(call PRINT_LINK, $(CXXLD) -shared $(CXXFLAGS)
> $(CXXLDFLAGS)
> >>> > $(LINK_FLAGS) $(SHIPFLAGS) $^ -o $@ -L$(build_private_libdir)
> >>> > -L$(build_libdir) -L$(build_shlibdir)
> >>> > ~~~
> >>> >
> >>> > so I can call julia dynamically as
> >>> > ~~~
> >>> > my_init(); // initalize stuff that hides its LLVM symbols after
> loading
> >>> > ...
> >>> > void* handle_julia = dlopen(LIBJULIAREPL, RTLD_NOW | RTLD_GLOBAL);
> >>> > ...
> >>> > typedef int (*t_jl_main)(int, char**);
> >>> > t_jl_main jl_main = (t_jl_main)dlsym(handle_julia, "main");
> >>> > return jl_main(argc, argv);
> >>> > ~~~
> >>> >
> >>> > On linux, I get strange linker errors:
> >>> > `/usr/bin/ld: repl.o: relocation R_X86_64_TPOFF32 against
> >>> > `tls_states.12084' can not be used when making a shared object;
> recompile
> >>> > with -fPIC`
> >>> >
> >>> > As far as I can tell, julia uses fPIC throughout. Has anyone
> encountered
> >>> > something like this before? Google links to some old gcc bugs and a
> go
> >>> > linker issue but it's not evident if there is a fix.
> >>> >
> >>> > Cheers,
> >>> > Joosep
>
Re: [julia-users] embedding: julia repl as a shared library
> I was under the impression that by "patch repl.c" you mean to patch it > somehow so that you can compile as a shared library, that will be very > bad and is intentionally not supported. > If you are talking about adding your llvm initialization stuff in this > file and compile it still as an executable and if you current goal is > to get a julia binary that does not confuse LLVM then I think that's > the best way to do it and the approach itself is not "dirty" at all > (apart from possibly dirty things you need to do to "unconfuse" LLVM, > which you need to do anyway, independent of where you do it). Yes, that's what I wanted to do (re-compile the julia binary with my preinit code), sorry for not being clear. If only 3 (or some N<10) lines of code was needed to make a fully functional julia binary using libjulia, I suppose repl.c would be a bit shorter as well. In fact, I found the way to implement what I needed also via dlopening libjulia [1] as you suggested, but having a non-standard location for the julia binary (wrt. julia source tree) is a real pain, so I think I'll just go with the patch-and-recompile-binary approach. Thanks again for the clarifications! [1] https://github.com/jpata/ROOT.jl/blob/cxx/src/ui.cc
[julia-users] IJulia loses syntax coloring
Hi, Does anyone else have issues in persisting the syntax highlighting in IJulia? For me, the Julia-specific coloring seems to erratically disappear when I restart the kernel. When I convert each cell to something other than code and back again, it seems to work, until it again loses formatting. This is a minor annoyance and does not hinder productivity, but I would like my notebooks to keep all their nice features even when closed. For the record, I’m using HEAD on OSX 10.9.4, this happens both with Safari and Firefox under IPython 2.1. Joosep
