RE: [julia-users] Re: performance of parallel code
I can do it using pmap, but it doesn't run any faster (still 3 workers): julia> @time h10 = Calculus.hessian(myf, RB.troubleParamsSim1.raw) 186.704029 seconds (410.30 k allocations: 29.503 MB, 0.01% gc time) That was using this code: """ Return a function that takes one argument, the optimization parameters. It evaluates the -LL in parallel """ function makeParallelLL() np = nworkers() cuts = chunks(obs[:id], np) function myLL(x) args = [(cuts[i, 1], cuts[i,2], x) for i = 1:np] ll= pmap(x->cutLL(x...), args) -sum(ll) end end From: julia-users@googlegroups.com [julia-users@googlegroups.com] on behalf of Christopher Fisher [fishe...@miamioh.edu] Sent: Monday, June 20, 2016 4:45 AM To: julia-users Subject: [julia-users] Re: performance of parallel code Would using pmap() be suitable for your application? Usually when I use that I get slightly more than a 2X speed up with 4 cores. On Monday, June 20, 2016 at 2:27:41 AM UTC-4, Boylan, Ross wrote: I think I've taken steps to minimize parallel overhead by providing only one function call per process and passing really minimal arguments to the functions. But the gains in speed don't seem commensurate with the number of processors. I know that pure linear speedup is too much to hope for, but I suspect I'm doing something wrong--for example that large data is getting passed around despite my efforts. All my code is defined inside a module, though I exercise it from the main REPL. Single processor (times are representative of multiple samples and exclude burn-in runs): julia> @time h=Calculus.hessian(RB.mylike, RB.troubleParamsSim1.raw) 206.422562 seconds (2.43 G allocations: 83.867 GB, 10.14% gc time) #with 3 workers julia> myf = RB.makeParallelLL() # code below julia> @time h10 = Calculus.hessian(myf, RB.troubleParamsSim1.raw) 182.567647 seconds (1.48 M allocations: 111.622 MB, 0.02% gc time) #with 7 workers julia> @time h10 = Calculus.hessian(myf, RB.troubleParamsSim1.raw) 82.453033 seconds (3.43 M allocations: 259.838 MB, 0.08% gc time) Any suggestions? This on an 8 CPU VM; the underlying hardware has >>8 processors. Here's some of the code: mylike(v) = -logitlite(RB.obs, v, 7) """ return a matrix whose columns are the start and the end of range of observations, inclusive. these chunks are roughly equal sizes, and no id is split between chunks. n target number of chunks, and thus of rows returned id id for each row; generally one id may appear in several rows. all rows with the same id must be contiguous. """ function chunks(id, n::Int) nTot = length(id) s = nTot/n cuts = Vector{Int}(n) cuts[1] = 1 for i = 1:(n-1) cut::Int = floor(s*i) while id[cut] == id[cut-1] cut += 1 end cuts[i+1] = cut end stops = vcat(cuts[2:n]-1, nTot) hcat(cuts, stops) end """ Parallel helper: evaluate likelihood over limited range i0..i1, inclusive, is the range. This is an effort to avoid passing obs between processes. """ function cutLL(i0::Int, i1::Int, x) logitlite(obs[i0:i1,:], x, 7) #obs is a 6413x8 data.frame. Maybe this is doing an extra copy? end """ Return a function that takes one argument, the optimization parameters. It evaluates the -LL in parallel """ function makeParallelLL() np = nworkers() cuts = chunks(obs[:id], np) function myLL(x) ll= @parallel (+) for i=1:np cutLL(cuts[i,1], cuts[i,2], x) end -ll end end It might be relevant that logitlite uses a by() to process groups of observations. So I think the only thing I'm passing into the workers is a function call, cutLL, 2 integers, and a 10-element Vector{Float64}, and the only thing going back is 1 Float64. Or perhaps fact that I'm using an inner function def (myLL in the last function definition) is doing something like causing transmission of all variables in lexical scope?
RE: [julia-users] errors using packages in parallel
I've been encountering errors with packages in parallel, even though everything is happening on one machine (julia -p 4 or -p 3 under ESS). This isn't necessarily the same issue, but it seems suggestive it's arising at the same time. Originally I think the packages/cache were a bit out of date, and when I ran in parallel things went wrong and rebuilt everywhere. If they were all rebuilding in the same spot, as they seemed to, it's easy to imagine that would cause problems. I also had a single CPU copy of julia running at the same time (idle) throughout. After that I started up a single cpu version, did a Pkg.update(), and ran my code. Everything seemed OK. However, I then ran a parallel julia, and continued to get errors. The included code is all in a module and connect to a Postgresql DB to get some data. Here are excerpts from the sessions: | | | | | | |/ _' | | | | |_| | | | (_| | | Version 0.4.6-pre+28 (2016-04-22 00:59 UTC) _/ |\__'_|_|_|\__'_| | Commit 022917e (44 days old release-0.4) |__/ | x86_64-linux-gnu WARNING: Terminal not fully functional julia> julia> nprocs() 5 julia> now() 2016-06-05T00:04:52 # Pacific time US julia> @everywhere include("trouble.jl") INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Debug.ji for module Debug. WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/PDMats.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/PDMats.ji WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/PDMats.ji WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/PDMats.ji WARNING: Module PDMats uuid did not match cache file This is likely because module PDMats does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji WARNING: Module PDMats uuid did not match cache file This is likely because module PDMats does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsFuns.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsFuns.ji WARNING: Module PDMats uuid did not match cache file This is likely because module PDMats does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsFuns.ji WARNING: Module PDMats uuid did not match cache file This is likely because module PDMats does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsFuns.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from
RE: [julia-users] Re: Calculus.hessian result oddities
I believe Calculus.hessian does finite differences. I had not expected it to computer both h[i, j] and h[j, i], since they should be the same. That is, AFAIK they are the same mathematically; numerically some differences dependent on the order of operations seem inevitable. The double calculation has the potential advantage of alerting you if the numerical issues are significant, but returning a matrix that lacks the expected symmetry seems a pretty high price. Ross From: julia-users@googlegroups.com [julia-users@googlegroups.com] on behalf of Kristoffer Carlsson [kcarlsso...@gmail.com] Sent: Tuesday, May 31, 2016 1:26 AM To: julia-users Subject: [julia-users] Re: Calculus.hessian result oddities Doesn't Calculu's Hessian just do finite difference? There will naturally be floating point errors. isposdef returns false if the matrix is not hermitian, under the hood, potrf from Lapack is used which work with hermitian matrices. On Tuesday, May 31, 2016 at 7:04:04 AM UTC+2, Boylan, Ross wrote: The hessian produced by Calculus.hessian has some properties that seem odd to me. First, the resulting matrix is not symmetric (though it's really close). Second, it is not positive definite even though the eigenvalues are all positive. This might partly be the result of the first problem. Comments? julia> @time h6 = Calculus.hessian(RB.mylike, sol6[2]) 206.988247 seconds (2.43 G allocations: 83.830 GB, 15.48% gc time) 10x10 Array{Float64,2}: julia> issym(h6) false julia> isposdef(h6) false julia> eigvals(h6) 10-element Array{Float64,1}: 3.27548e5 41407.1 1873.44 809.515 385.469 336.118 269.82 64.4228 117.009 122.845 julia> [det(h6[1:i, 1:i]) for i in 1:10] # all submatrix determinants are positive 10-element Array{Any,1}: 583.356 2.93666e5 3.76334e7 4.26776e9 3.17529e12 6.53405e14 1.12477e17 8.54011e21 3.75658e26 6.65866e29 julia> showall(h6) [583.356378500377 -215.9519377704782 42.707753088841706 -12.046905619427504 100.26418659571075 30.907721733245413 5.1999446295105445 -3522.0797654066855 4410.401341876791 29.123944464570087 -215.95193777047822 583.3512842422355 -12.0619061856546 42.69860798380086 -82.01778794681003 -20.00373356894301 -8.643894790795585 1484.6195212852567 -3316.905502014349 -2.7622075180025965 42.7077530888417 -12.061906185654598 131.30467049554812 -44.14799847519552 16.748039679197223 -10.98960509547607 43.02830286169418 234.68906012694512 -1258.6622801992237 9.86994636336174 -12.046905619427504 42.69860798380086 -44.147998475195514 131.35743705036847 -17.498616283805696 4.724998443749094 -31.481387268916205 -95.390157895702 209.5245372890838 12.903636963625035 100.26418659571075 -82.01778794681003 16.748039679197223 -17.498616283805696 766.6651018177663 -20.766430168130857 -2.164074090378522 610.206909770971 -6927.858446549729 -200.4409796490271 30.907721733245413 -20.00373356894301 -10.98960509547607 4.724998443749093 -20.766430168130857 209.8283757716871 -60.222027409043555 280.3888402431943 -1638.62573982097 -158.60599205247166 5.199944629510544 -8.643894790795585 43.02830286169418 -31.4813872689162 -2.164074090378524 -60.222027409043555 204.34068072875016 -3.631354965653126 314.50383342119943 30.12788292183477 -3522.079765406686 1484.6195212852565 234.6890601269451 -95.39015789570199 610.2069097709712 280.3888402431943 -3.631354965653125 102596.98982032725 -117630.58331133435 267.23733374838395 4410.401341876791 -3316.9055020143487 -1258.6622801992235 209.5245372890838 -6927.858446549729 -1638.62573982097 314.5038334211995 -117630.58331133435 265600.50878023007 5508.687364627178 29.123944464570084 -2.762207518002596 9.86994636336174 12.903636963625033 -200.4409796490271 -158.6059920524717 30.12788292183477 267.23733374838395 5508.687364627178 2125.8954716959997] The largest absolute difference between symmetrical entries is < 5e-13. Ross Boylan
[julia-users] Calculus.hessian result oddities
The hessian produced by Calculus.hessian has some properties that seem odd to me. First, the resulting matrix is not symmetric (though it's really close). Second, it is not positive definite even though the eigenvalues are all positive. This might partly be the result of the first problem. Comments? julia> @time h6 = Calculus.hessian(RB.mylike, sol6[2]) 206.988247 seconds (2.43 G allocations: 83.830 GB, 15.48% gc time) 10x10 Array{Float64,2}: julia> issym(h6) false julia> isposdef(h6) false julia> eigvals(h6) 10-element Array{Float64,1}: 3.27548e5 41407.1 1873.44 809.515 385.469 336.118 269.82 64.4228 117.009 122.845 julia> [det(h6[1:i, 1:i]) for i in 1:10] # all submatrix determinants are positive 10-element Array{Any,1}: 583.356 2.93666e5 3.76334e7 4.26776e9 3.17529e12 6.53405e14 1.12477e17 8.54011e21 3.75658e26 6.65866e29 julia> showall(h6) [583.356378500377 -215.9519377704782 42.707753088841706 -12.046905619427504 100.26418659571075 30.907721733245413 5.1999446295105445 -3522.0797654066855 4410.401341876791 29.123944464570087 -215.95193777047822 583.3512842422355 -12.0619061856546 42.69860798380086 -82.01778794681003 -20.00373356894301 -8.643894790795585 1484.6195212852567 -3316.905502014349 -2.7622075180025965 42.7077530888417 -12.061906185654598 131.30467049554812 -44.14799847519552 16.748039679197223 -10.98960509547607 43.02830286169418 234.68906012694512 -1258.6622801992237 9.86994636336174 -12.046905619427504 42.69860798380086 -44.147998475195514 131.35743705036847 -17.498616283805696 4.724998443749094 -31.481387268916205 -95.390157895702 209.5245372890838 12.903636963625035 100.26418659571075 -82.01778794681003 16.748039679197223 -17.498616283805696 766.6651018177663 -20.766430168130857 -2.164074090378522 610.206909770971 -6927.858446549729 -200.4409796490271 30.907721733245413 -20.00373356894301 -10.98960509547607 4.724998443749093 -20.766430168130857 209.8283757716871 -60.222027409043555 280.3888402431943 -1638.62573982097 -158.60599205247166 5.199944629510544 -8.643894790795585 43.02830286169418 -31.4813872689162 -2.164074090378524 -60.222027409043555 204.34068072875016 -3.631354965653126 314.50383342119943 30.12788292183477 -3522.079765406686 1484.6195212852565 234.6890601269451 -95.39015789570199 610.2069097709712 280.3888402431943 -3.631354965653125 102596.98982032725 -117630.58331133435 267.23733374838395 4410.401341876791 -3316.9055020143487 -1258.6622801992235 209.5245372890838 -6927.858446549729 -1638.62573982097 314.5038334211995 -117630.58331133435 265600.50878023007 5508.687364627178 29.123944464570084 -2.762207518002596 9.86994636336174 12.903636963625033 -200.4409796490271 -158.6059920524717 30.12788292183477 267.23733374838395 5508.687364627178 2125.8954716959997] The largest absolute difference between symmetrical entries is < 5e-13. Ross Boylan
RE: [julia-users] Why isn't my constructor recognized?
Thank you; that did the trick (with `Coefficients{T}(vals::Vector{T}) = Coefficients{T}(vals)` defined outside the type). The fact that defining Coefficients{T} does not necessarily define Coefficients seems like an awfully subtle point, though it is mentioned in http://docs.julialang.org/en/release-0.4/manual/constructors/: Some features of parametric constructor definitions at work here deserve comment. First, inner constructor declarations always define methods of Point{T} rather than methods of the general Point constructor function. Since Point is not a concrete type, it makes no sense for it to even have inner constructor methods at all. Thus, the inner method declaration Point(x,y) = new(x,y) provides an inner constructor method for each value of T. It is this method declaration that defines the behavior of constructor calls with explicit type parameters like Point{Int64}(1,2) and Point{Float64}(1.0,2.0). The outer constructor declaration, on the other hand, defines a method for the general Point constructor which only applies to pairs of values of the same real type. It's even trickier because the definition that makes Coefficients without the type work looks exactly like (on the left) the declaration which does not have the desired effect, Coefficients{T}. Does the language have to work this way? Ross From: julia-users@googlegroups.com [julia-users@googlegroups.com] on behalf of Yichao Yu [yyc1...@gmail.com] Sent: Friday, May 27, 2016 6:02 PM To: Julia Users Subject: Re: [julia-users] Why isn't my constructor recognized? On Fri, May 27, 2016 at 7:53 PM, Boylan, Ross <ross.boy...@ucsf.edu> wrote: > A type is defined as > type Coefficients{T} > raw::Vector{T} > σ_0::T > σ_T::T > ρ::T > ν::T > Σ::Matrix{T} > # because of this c'tor the type must be mutable > function Coefficients(vals::Vector{T}) This defines a method for `Coefficients{T}` and needs to be called with `Coefficients{T}(...)` and not `Coefficients(...)` Defining a inner constructor turns off the automatically defined construction (method for `Coefficients`) that automatically deduce the type parameter so you need to manually define that constructor with `Coefficients{T}(vals::Vector{T}) = Coefficients{T}(vals)` (note that the first `{T}` is a parameter for this method of `Coefficients`, the last `{T}` uses this variable to instantiate `Coefficients` and then call the `Coefficients{T}` object instead.)
[julia-users] Why isn't my constructor recognized?
A type is defined as type Coefficients{T} raw::Vector{T} σ_0::T σ_T::T ρ::T ν::T Σ::Matrix{T} # because of this c'tor the type must be mutable function Coefficients(vals::Vector{T}) if length(vals) ≠ 10 error("Must have exactly 10 parameters") end self = new(vals) self.σ_0 = exp(lnσ_0(self)) self.σ_T = exp(lnσ_T(self)) self.ρ = tanh(atanhρ(self)) self.ν = exp(lnν(self)) # chol doesn't work on SymTridiagonal self.Σ = Matrix(SymTridiagonal([ self.σ_0, self.σ_T].^2, [self.ρ * self.σ_0 * self.σ_T])) self end end But julia> @time g = Calculus.gradient(RB.mylike, RB.troubleParamsInit.raw) ERROR: MethodError: `convert` has no method matching convert(::Type{RB.Coefficients{T}}, ::Array{Float64,1}) This may have arisen from a call to the constructor RB.Coefficients{T}(...), since type constructors fall back to convert methods. Closest candidates are: call{T}(::Type{T}, ::Any) convert{T}(::Type{T}, !Matched::T) in logitlite at /home/ross/PCORI/trouble.jl:316 in mylike at /home/ross/PCORI/trouble.jl:345 in finite_difference! at /home/ross/.julia/v0.4/Calculus/src/finite_difference.jl:126 in gradient at /home/ross/.julia/v0.4/Calculus/src/derivative.jl:14 The triggering line was c = Coefficients(params) Since Array{Float64,1} = Vector{Float64} I am puzzled that the inner constructor isn't called. I also tried with the inner constructor defined as function Coefficients{T}(vals::Vector{T}) When I call this from my own code things work OK. Can anyone help me identify the problem, or even the solution :) Thanks. Ross Boylan
[julia-users] old objects and new code
A file has code surrounded by a module declaration. It includes definitions of some types and some functions. In the REPL (ESS) I include the file and execute a function that returns me an object x of type MyModule.Foo. Then I change some of the code and include the module again. The type Foo is unchanged. It seems to me that when I execute one of the module functions on x the original version of the function, not the revised version, gets called. If I recreate x then I get the current function definitions aka what I wanted. First, is my understanding of what's going on correct? Second, any recommendations for dealing with this? In this case regenerating x was not a big deal, but I can imagine situations that weren't so trivial. If there were lots of objects involved, one could keep a script that creates them and rerun it. Thanks Ross Boylan
RE: [julia-users] Re: newbie questions (was undefined reference error)
But if I make an array of immutables I won't be able to change them afterwards. Though I can replace them. With the old code except for immutable Stuff a::Int b::Int end I now have julia> v=Array(TT.Stuff, 2) 2-element Array{TT.Stuff,1}: TT.Stuff(140124021836848,140124000460928) TT.Stuff(140124048730192,0) julia> v[1].a=44 # what I'd like to do ERROR: type Stuff is immutable julia> v[1]=TT.Stuff(3, 4) # work-around TT.Stuff(3,4) !julia> v 2-element Array{TT.Stuff,1}: TT.Stuff(3,4) TT.Stuff(140124048730192,0) With immutable, is everything in the array laid out contiguous in memory? It seems sort of odd that the type is immutable yet I can overwrite data of that type. I suppose that would only be prohibited if the array were immutable. Thanks for the tip about using Type to get the zero to work. Ross From: julia-users@googlegroups.com [julia-users@googlegroups.com] on behalf of Cedric St-Jean [cedric.stj...@gmail.com] Sent: Tuesday, May 10, 2016 1:20 PM To: julia-users Subject: [julia-users] Re: newbie questions (was undefined reference error) "normal" types are by definition heap-allocated, and are always manipulated them through pointers. What you want is immutables<http://docs.julialang.org/en/release-0.4/manual/types/#immutable-composite-types> immutable Stuff a::Int b::Int end # Also, for zeros to work, function zero(x::Type{Stuff}) Stuff(0, 0) end On Tuesday, May 10, 2016 at 4:04:37 PM UTC-4, Boylan, Ross wrote: I'm puzzled that a type consisting only of 2 integers doesn't qualify as "bitstype". Further experiment shows that the array seems to be an array of references, I don't know how to implement zero, and generally that I'm a bit lost :) My goal is to get a densely packed array of data. I assume that will use less memory and generate faster code; if not, maybe I should change my goal. BTW, my real use has a type more heterogeneous than 2 Int's, so a solution that uses a 2D array doesn't really generalize appropriately for me. module TT import Base.zero type Stuff a::Int b::Int end function zero(x::Stuff) Stuff(0, 0) end end julia> v=Array(TT.Stuff, 3) #as before 3-element Array{TT.Stuff,1}: #undef #undef #undef julia> s=TT.Stuff(3, 5) TT.Stuff(3,5) julia> v=fill(s, 2) 2-element Array{TT.Stuff,1}: TT.Stuff(3,5) TT.Stuff(3,5) julia> s.a=900 900 julia> v ###OOPS: every array element points to the same instance 2-element Array{TT.Stuff,1}: TT.Stuff(900,5) TT.Stuff(900,5) julia> zero(TT.Stuff) # This is probably what needs to work for zeros() to work ERROR: MethodError: `zero` has no method matching zero(::Type{TT.Stuff}) !julia> zero(TT.Stuff(1, 1)) # this at least calls the right c'tor TT.Stuff(0,0) Ross From: julia...@googlegroups.com [julia...@googlegroups.com] on behalf of Lutfullah Tomak [tomak...@gmail.com] Sent: Tuesday, May 10, 2016 12:04 AM To: julia-users Subject: [julia-users] undefined reference error You need to initialize array entries if you don't have eltype as bitstype. Here, undefined reference means you had not initialize the entry. And, full type assignment works because it initializes the entry.
[julia-users] newbie questions (was undefined reference error)
I'm puzzled that a type consisting only of 2 integers doesn't qualify as "bitstype". Further experiment shows that the array seems to be an array of references, I don't know how to implement zero, and generally that I'm a bit lost :) My goal is to get a densely packed array of data. I assume that will use less memory and generate faster code; if not, maybe I should change my goal. BTW, my real use has a type more heterogeneous than 2 Int's, so a solution that uses a 2D array doesn't really generalize appropriately for me. module TT import Base.zero type Stuff a::Int b::Int end function zero(x::Stuff) Stuff(0, 0) end end julia> v=Array(TT.Stuff, 3) #as before 3-element Array{TT.Stuff,1}: #undef #undef #undef julia> s=TT.Stuff(3, 5) TT.Stuff(3,5) julia> v=fill(s, 2) 2-element Array{TT.Stuff,1}: TT.Stuff(3,5) TT.Stuff(3,5) julia> s.a=900 900 julia> v ###OOPS: every array element points to the same instance 2-element Array{TT.Stuff,1}: TT.Stuff(900,5) TT.Stuff(900,5) julia> zero(TT.Stuff) # This is probably what needs to work for zeros() to work ERROR: MethodError: `zero` has no method matching zero(::Type{TT.Stuff}) !julia> zero(TT.Stuff(1, 1)) # this at least calls the right c'tor TT.Stuff(0,0) Ross From: julia-users@googlegroups.com [julia-users@googlegroups.com] on behalf of Lutfullah Tomak [tomaklu...@gmail.com] Sent: Tuesday, May 10, 2016 12:04 AM To: julia-users Subject: [julia-users] undefined reference error You need to initialize array entries if you don't have eltype as bitstype. Here, undefined reference means you had not initialize the entry. And, full type assignment works because it initializes the entry.
[julia-users] undefined reference error
julia 0.4: module TT type Stuff a::Int b::Int end v = Array(Stuff, 3) end Produces julia> TT.v[1].a = 5 ERROR: UndefRefError: access to undefined reference problem in getindex at array.jl:282 !julia> TT.v 3-element Array{TT.Stuff,1}: #undef #undef #undef Could anyone help me to understand what's going on, and what I can do about it? I know Array() produces uninitialized elements, but I don't understand why I can't write values into them. In contrast, replacing the entire object works: julia> TT.v[1] = TT.Stuff(4, 78) TT.Stuff(4,78) !julia> TT.v[1] TT.Stuff(4,78) Thanks. Ross Boylan
RE: [julia-users] Can't add PostgreSQL
For the PostgreSQL package, when I do fetchdf against the result of execute, instead of the stmt, it does work. Another problem solved! The DataFrame returned by fetchdf seems to have lost the type information. Am I understanding that right? It seems like undesirable behavior. --- stmt = prepare(conn, "SELECT isim::int, id::int, x::int, t::double precision, y::int, regular::int, censor::int, wait::double precision from obs2619 where isim=1") result = execute(stmt) obs = fetchdf(result) produces julia> eltypes(obs) 8-element Array{Type{T},1}: Any Any Any Any Any Any Any Any Ross From: julia-users@googlegroups.com [julia-users@googlegroups.com] on behalf of Cameron McBride [cameron.mcbr...@gmail.com] Sent: Wednesday, May 04, 2016 3:17 AM To: julia-users Subject: Re: [julia-users] Can't add PostgreSQL I've been using PostgreSQL and DBI just fine for a few months now. The implementation has a few edges that need polishing and work, but the basics work just fine for the julia 0.4.x branch. The problem you ran into with fetchdf() not being implemented is that you tried to run it on the stmt (before running execute). This is one of the rough edges that not all convenience functions are implemented. In short, I just confirmed the following example runs on julia 0.4.5 and the latest DBI.jl / PostgreSQL.jl: # slightly modified example from https://github.com/JuliaDB/PostgreSQL.jl using DBI using PostgreSQL conn = connect(Postgres, "localhost", "username", "password", "dbname", 5432) stmt = prepare(conn, "SELECT 1::bigint, 2.0::double precision, 'foo'::character varying, " * "'foo'::character(10);") result = execute(stmt) # this was missing in your use! df = fetchdf(result) finish(stmt) disconnect(conn) Perhaps that helps clarify? Happy to help further, here or offline. Cameron
RE: [julia-users] Can't add PostgreSQL
ODBC is now working for me with the documented interface (after updates). Thanks! I do get a lot of chatter as it loads, mostly from the DecFP package. The documentation doesn't mention how to shutdown. I used disconnect!(dsn) after peeking at the code. Does it clean up the connection if you just exit the julia session without disconnect? Also, in this sequence 1. datatable = ODBC.query(dsn, "show databases") # convert result to a DataFrame for additional data manipulation functionality 2. df = DataFrame(datatable) 3. do stuff with df at what point is it safe to disconnect? Is the odbc interface in general, or particularly with postgresql, thread safe? On a more general topic I got lots of notices my packages were out of date, even immediately after I installed them. What's that about? E.g., julia> Pkg.add("Cairo") INFO: Nothing to be done INFO: METADATA is out-of-date — you may not have the latest version of Cairo INFO: Use 'Pkg.update()' to get the latest versions of your packages julia> Pkg.update() INFO: Updating METADATA... INFO: Updating DBI... INFO: Updating PostgreSQL... INFO: Updating Postgres... INFO: Computing changes... INFO: No packages to install, update or remove Ross From: karbar...@gmail.com [karbar...@gmail.com] on behalf of Jacob Quinn [quinn.jac...@gmail.com] Sent: Wednesday, May 04, 2016 7:47 AM To: julia-users@googlegroups.com Subject: Re: [julia-users] Can't add PostgreSQL Hey Ross, It actually sounds like you're still on the older release of ODBC (you can check by running Pkg.installed()). To get the latest version (that matches the current documentation), you'll have to run: Pkg.update() Then restart Julia to make sure you have the latest code. -Jacob On Tue, May 3, 2016 at 9:58 PM, Boylan, Ross <ross.boy...@ucsf.edu> wrote: I got it working before your update. Yay! Thank you! There were a couple of issues, mostly documentation, leaving aside getting ODBC setup outside of julia. The docs (the help at https://github.com/JuliaDB/ODBC.jl) say to use DSN to get things going. As far as I can tell, it doesn't exist. I used connect. The docs mention a connection string, but provide no pointers to its content. It would be nice to have one; the odbc packages I installed were almost documentation free. The docs imply the results of a query need to be turned into a DataFrame, but it seems to come out as one. The source code suggested one could get a dataframe with the output argument. I'm not really sure what that's supposed to be or do, but when I tried julia> raw = ODBC.query("SELECT isim, id, x, t, y, regular, censor, wait from obs2619 where isim=1", conn, output=df ) ERROR: TypeError: typeassert: expected Union{AbstractString,DataType}, got Function I had not set df to anything before the call, AFAIK. Ross From: karbar...@gmail.com [karbar...@gmail.com] on behalf of Jacob Quinn [quinn.jac...@gmail.com] Sent: Tuesday, May 03, 2016 7:57 PM To: julia-users@googlegroups.com Subject: Re: [julia-users] Can't add PostgreSQL Hey Ross, Just confirmed that connecting to Postgres does indeed work; haven't seen any issues yet. I'm using the PostgreSQL ODBC Driver from their website with unixODBC on a mac. I'm using a connection string I built by following the guidelines here: http://www.connectionstrings.com/postgresql-odbc-driver-psqlodbc/ Here's a little sample code (I have more testing all the various data types in Postgres, but I won't post the full snippet here): using ODBC dsn = ODBC.DSN("Driver={PostgreSQL};Server=[server];Port=5432;Database=testdb;Uid=username;Pwd=password") # Check some basic queries dbs = ODBC.query(dsn, "SELECT datname FROM pg_database WHERE datistemplate = false;") data = ODBC.query(dsn, "SELECT table_schema,table_name FROM information_schema.tables ORDER BY table_schema,table_name;") The latest ODBC release was just merged in METADATA.jl, so if you do Pkg.add("ODBC") you should get the latest release/code. If you already had it installed, just run Pkg.update() to get the latest release. Good luck! -Jacob On Tue, May 3, 2016 at 7:11 PM, Boylan, Ross <ross.boy...@ucsf.edu> wrote: Attempting to post comments inline, even though my mailer is allergic. Look for >> From: karbar...@gmail.com [karbar...@gmail.com] on behalf of Jacob Quinn [quinn.jac...@gmail.com] Sent: Tuesday, May 03, 2016 5:47 PM I'm not sure I understand: I don't see DBI.jl nor PostgreSQL.jl at pkg.julialang.org, which tells me they aren't officially registered as packages. The difficulties you're seeing then aren't that surprising because they haven't felt comfortable enough to officially register. >> Yo
RE: [julia-users] Can't add PostgreSQL
I got it working before your update. Yay! Thank you! There were a couple of issues, mostly documentation, leaving aside getting ODBC setup outside of julia. The docs (the help at https://github.com/JuliaDB/ODBC.jl) say to use DSN to get things going. As far as I can tell, it doesn't exist. I used connect. The docs mention a connection string, but provide no pointers to its content. It would be nice to have one; the odbc packages I installed were almost documentation free. The docs imply the results of a query need to be turned into a DataFrame, but it seems to come out as one. The source code suggested one could get a dataframe with the output argument. I'm not really sure what that's supposed to be or do, but when I tried julia> raw = ODBC.query("SELECT isim, id, x, t, y, regular, censor, wait from obs2619 where isim=1", conn, output=df ) ERROR: TypeError: typeassert: expected Union{AbstractString,DataType}, got Function I had not set df to anything before the call, AFAIK. Ross From: karbar...@gmail.com [karbar...@gmail.com] on behalf of Jacob Quinn [quinn.jac...@gmail.com] Sent: Tuesday, May 03, 2016 7:57 PM To: julia-users@googlegroups.com Subject: Re: [julia-users] Can't add PostgreSQL Hey Ross, Just confirmed that connecting to Postgres does indeed work; haven't seen any issues yet. I'm using the PostgreSQL ODBC Driver from their website with unixODBC on a mac. I'm using a connection string I built by following the guidelines here: http://www.connectionstrings.com/postgresql-odbc-driver-psqlodbc/ Here's a little sample code (I have more testing all the various data types in Postgres, but I won't post the full snippet here): using ODBC dsn = ODBC.DSN("Driver={PostgreSQL};Server=[server];Port=5432;Database=testdb;Uid=username;Pwd=password") # Check some basic queries dbs = ODBC.query(dsn, "SELECT datname FROM pg_database WHERE datistemplate = false;") data = ODBC.query(dsn, "SELECT table_schema,table_name FROM information_schema.tables ORDER BY table_schema,table_name;") The latest ODBC release was just merged in METADATA.jl, so if you do Pkg.add("ODBC") you should get the latest release/code. If you already had it installed, just run Pkg.update() to get the latest release. Good luck! -Jacob On Tue, May 3, 2016 at 7:11 PM, Boylan, Ross <ross.boy...@ucsf.edu<mailto:ross.boy...@ucsf.edu>> wrote: Attempting to post comments inline, even though my mailer is allergic. Look for >> From: karbar...@gmail.com<mailto:karbar...@gmail.com> [karbar...@gmail.com<mailto:karbar...@gmail.com>] on behalf of Jacob Quinn [quinn.jac...@gmail.com<mailto:quinn.jac...@gmail.com>] Sent: Tuesday, May 03, 2016 5:47 PM I'm not sure I understand: I don't see DBI.jl nor PostgreSQL.jl at pkg.julialang.org<http://pkg.julialang.org>, which tells me they aren't officially registered as packages. The difficulties you're seeing then aren't that surprising because they haven't felt comfortable enough to officially register. >> You're right. I got to them through https://github.com/JuliaDB ODBC actually works fine on Linux/OSX through the unixODBC library. It can be installed through a variety of means, but works quite well from what I've seen. Postgres is actually next on my list to test with ODBC (so far the testing has focused on Teradata, SQL Server, and MySQL), so if I find time in the next few days, I'll try to put together an example and share it here. >> The status page indicates errors with julia 0.4 and Linux. Does it matter? >> Clicking on the build logs it appears the most recent build for 0.4, 124.4 >> (https://travis-ci.org/JuliaDB/ODBC.jl) succeeded, so I'm not sure why the >> problem is showing (also the previous error looks like a problem with the >> configuration of the build machine rather than a problem with the package). >> No joy with the alternate Postgresl package; >> https://github.com/NCarson/Postgres.jl/issues/3 has the details. >> So I guess I'll give ODBC a whirl. Thanks for your help.
RE: [julia-users] Can't add PostgreSQL
Attempting to post comments inline, even though my mailer is allergic. Look for >> From: karbar...@gmail.com [karbar...@gmail.com] on behalf of Jacob Quinn [quinn.jac...@gmail.com] Sent: Tuesday, May 03, 2016 5:47 PM I'm not sure I understand: I don't see DBI.jl nor PostgreSQL.jl at pkg.julialang.org, which tells me they aren't officially registered as packages. The difficulties you're seeing then aren't that surprising because they haven't felt comfortable enough to officially register. >> You're right. I got to them through https://github.com/JuliaDB ODBC actually works fine on Linux/OSX through the unixODBC library. It can be installed through a variety of means, but works quite well from what I've seen. Postgres is actually next on my list to test with ODBC (so far the testing has focused on Teradata, SQL Server, and MySQL), so if I find time in the next few days, I'll try to put together an example and share it here. >> The status page indicates errors with julia 0.4 and Linux. Does it matter? >> Clicking on the build logs it appears the most recent build for 0.4, 124.4 >> (https://travis-ci.org/JuliaDB/ODBC.jl) succeeded, so I'm not sure why the >> problem is showing (also the previous error looks like a problem with the >> configuration of the build machine rather than a problem with the package). >> No joy with the alternate Postgresl package; >> https://github.com/NCarson/Postgres.jl/issues/3 has the details. >> So I guess I'll give ODBC a whirl. Thanks for your help.
RE: [julia-users] Can't add PostgreSQL
That's odd. After cloning the Postgres package, and then executing the script from the command line, it gets further: $ julia trouble.jl ERROR: LoadError: DBI API not fully implemented in fetchdf at /home/ross/.julia/v0.4/DBI/src/DBI.jl:97 in include at ./boot.jl:261 in include_from_node1 at ./loading.jl:320 in process_options at ./client.jl:280 in _start at ./client.jl:378 while loading /home/ross/PCORI/trouble.jl, in expression starting on line 6 Here are the 3 additional lines: stmt = prepare(conn, "SELECT isim::int, id::int, x::int, t::double precision, y::int, regular::int, censor::int, wait::double precision from obs2619 where isim=1") execute(stmt) obs = fetchdf(stmt) So the fetchdf fails because it's not implemented. ____ From: Boylan, Ross Sent: Tuesday, May 03, 2016 4:06 PM To: julia-users@googlegroups.com Subject: RE: [julia-users] Can't add PostgreSQL Thank you, Jacob, for the pointer. I'm further along but still not there. I did the clone and then build. I'm not sure if the build was necessary, but it took awhile. This process did not install DBI although it's listed as a requirement, and the installer gave no warning about it. When that didn't work I cloned DBI. I'm surprised that packages shown on http://pkg.julialang.org/ are not necessarily in https://github.com/JuliaLang/METADATA.jl. Neither DBI nor PostgreSQL are there. Does it mean anything if a package is not in METADATA? After all that I got julia> Pkg.clone("https://github.com/JuliaDB/DBI.jl.git;) INFO: Cloning DBI from https://github.com/JuliaDB/DBI.jl.git INFO: Computing changes... !julia> include("/home/ross/PCORI/trouble.jl") INFO: Precompiling module DataArrays... INFO: Precompiling module DataFrames... ERROR: LoadError: UndefVarError: Postgres not defined in include at ./boot.jl:261 in include_from_node1 at ./loading.jl:320 while loading /home/ross/PCORI/trouble.jl, in expression starting on line 3 trouble.jl begins using DBI using PostgreSQL conn = connect(Postgres, "localhost", "user", "word", "table") Ideas? Isn't ODBC just for MS Windows? There's also https://github.com/NCarson/Postgres.jl; an announcement said it didn't implement the full DBI spec. Ross From: karbar...@gmail.com [karbar...@gmail.com] on behalf of Jacob Quinn [quinn.jac...@gmail.com] Sent: Tuesday, May 03, 2016 3:23 PM To: julia-users@googlegroups.com Subject: Re: [julia-users] Can't add PostgreSQL I don't think the PostgreSQL.jl package was ever officially registered. You could try Pkg.clone("https://github.com/JuliaDB/PostgreSQL.jl;) to manually download/install it; you may need to do Pkg.build("PostgreSQL") as well after cloning. Alternatively, you might try the latest master of https://github.com/JuliaDB/ODBC.jl for interacting with databases. -Jacob On Tue, May 3, 2016 at 4:19 PM, Boylan, Ross <ross.boy...@ucsf.edu<mailto:ross.boy...@ucsf.edu>> wrote: Am I misunderstanding how things work? Is the package only available for julia 0.3? I have a julia obtained via git a few days ago, on the release-0.4 branch and built locally. Seeing https://github.com/JuliaDB/PostgreSQL.jl I tried, from within an ESS session, julia> Pkg.add("PostgreSQL") ERROR: unknown package PostgreSQL in error at ./error.jl:21 [inlined code] from pkg/entry.jl:49 in anonymous at task.jl:447 in sync_end at ./task.jl:413 [inlined code] from task.jl:422 in add at pkg/entry.jl:64 in add at pkg/entry.jl:73 in anonymous at pkg/dir.jl:31 in cd at file.jl:22 in cd at pkg/dir.jl:31 in add at pkg.jl:23 To see if this was a spelling mistake: julia> Pkg.add("PostgresSQL") ERROR: unknown package PostgresSQL ... I was able to add the DataFrames package and dependencies. This is on a current Debian system, amd64 architecture.
RE: [julia-users] Can't add PostgreSQL
Thank you, Jacob, for the pointer. I'm further along but still not there. I did the clone and then build. I'm not sure if the build was necessary, but it took awhile. This process did not install DBI although it's listed as a requirement, and the installer gave no warning about it. When that didn't work I cloned DBI. I'm surprised that packages shown on http://pkg.julialang.org/ are not necessarily in https://github.com/JuliaLang/METADATA.jl. Neither DBI nor PostgreSQL are there. Does it mean anything if a package is not in METADATA? After all that I got julia> Pkg.clone("https://github.com/JuliaDB/DBI.jl.git;) INFO: Cloning DBI from https://github.com/JuliaDB/DBI.jl.git INFO: Computing changes... !julia> include("/home/ross/PCORI/trouble.jl") INFO: Precompiling module DataArrays... INFO: Precompiling module DataFrames... ERROR: LoadError: UndefVarError: Postgres not defined in include at ./boot.jl:261 in include_from_node1 at ./loading.jl:320 while loading /home/ross/PCORI/trouble.jl, in expression starting on line 3 trouble.jl begins using DBI using PostgreSQL conn = connect(Postgres, "localhost", "user", "word", "table") Ideas? Isn't ODBC just for MS Windows? There's also https://github.com/NCarson/Postgres.jl; an announcement said it didn't implement the full DBI spec. Ross From: karbar...@gmail.com [karbar...@gmail.com] on behalf of Jacob Quinn [quinn.jac...@gmail.com] Sent: Tuesday, May 03, 2016 3:23 PM To: julia-users@googlegroups.com Subject: Re: [julia-users] Can't add PostgreSQL I don't think the PostgreSQL.jl package was ever officially registered. You could try Pkg.clone("https://github.com/JuliaDB/PostgreSQL.jl;) to manually download/install it; you may need to do Pkg.build("PostgreSQL") as well after cloning. Alternatively, you might try the latest master of https://github.com/JuliaDB/ODBC.jl for interacting with databases. -Jacob On Tue, May 3, 2016 at 4:19 PM, Boylan, Ross <ross.boy...@ucsf.edu<mailto:ross.boy...@ucsf.edu>> wrote: Am I misunderstanding how things work? Is the package only available for julia 0.3? I have a julia obtained via git a few days ago, on the release-0.4 branch and built locally. Seeing https://github.com/JuliaDB/PostgreSQL.jl I tried, from within an ESS session, julia> Pkg.add("PostgreSQL") ERROR: unknown package PostgreSQL in error at ./error.jl:21 [inlined code] from pkg/entry.jl:49 in anonymous at task.jl:447 in sync_end at ./task.jl:413 [inlined code] from task.jl:422 in add at pkg/entry.jl:64 in add at pkg/entry.jl:73 in anonymous at pkg/dir.jl:31 in cd at file.jl:22 in cd at pkg/dir.jl:31 in add at pkg.jl:23 To see if this was a spelling mistake: julia> Pkg.add("PostgresSQL") ERROR: unknown package PostgresSQL ... I was able to add the DataFrames package and dependencies. This is on a current Debian system, amd64 architecture.
[julia-users] Can't add PostgreSQL
Am I misunderstanding how things work? Is the package only available for julia 0.3? I have a julia obtained via git a few days ago, on the release-0.4 branch and built locally. Seeing https://github.com/JuliaDB/PostgreSQL.jl I tried, from within an ESS session, julia> Pkg.add("PostgreSQL") ERROR: unknown package PostgreSQL in error at ./error.jl:21 [inlined code] from pkg/entry.jl:49 in anonymous at task.jl:447 in sync_end at ./task.jl:413 [inlined code] from task.jl:422 in add at pkg/entry.jl:64 in add at pkg/entry.jl:73 in anonymous at pkg/dir.jl:31 in cd at file.jl:22 in cd at pkg/dir.jl:31 in add at pkg.jl:23 To see if this was a spelling mistake: julia> Pkg.add("PostgresSQL") ERROR: unknown package PostgresSQL ... I was able to add the DataFrames package and dependencies. This is on a current Debian system, amd64 architecture.
RE: [julia-users] Roadmap
Somewhere, not necessarily on the front page, some tips for people wondering where the project is heading would be good. Not a list of plans, but orienting info like no one knows, and an explanation of how to get a sense of current (i.e., for 0.4 right now) and future issues from the bug tracker. I found the tip I got awhile ago to search on a particular tag (0.4?) and sort by bug/issue activity pretty helpful. BTW, is 0.4 still in a you don't want to go there state for users of julia? Ross From: julia-users@googlegroups.com [julia-users@googlegroups.com] on behalf of David Anthoff [anth...@berkeley.edu] Sent: Wednesday, December 10, 2014 5:38 PM To: julia-users@googlegroups.com Subject: RE: [julia-users] Roadmap Thanks, this is really useful! I would very much cherish an email like this maybe after a release from some of the core team members. It just gives a nice insight into the current plans. Cheers, David From: julia-users@googlegroups.com [mailto:julia-users@googlegroups.com] On Behalf Of John Myles White Sent: Wednesday, December 10, 2014 4:56 PM To: julia-users@googlegroups.com Subject: Re: [julia-users] Roadmap FWIW, my sense is that no one really knows what's going to happen between 0.4 and 1.0. There are lots of projects that are seen as essential before 1.0, but many of those are tenatively on the 0.4 release targets (static compilation, array views, package documentation, etc.). At JuliaCon, I realized that I was one of the longest standing users of Julia -- many people at JuliaCon had never tried Julia 0.1 and therefore don't remember how much the 0.2 release improved the language and redefined the way Julia code was written. I feel like 0.4 is going to be a similar release: a lot of the most egregious problems with the current version of Julia are going to be fixed. But once those problems are solved, it seems hard to believe that we won't start realizing that there are lots of parts of the language that could be cleaned up before 1.0. My sense is that Julia, like ggplot2, will start to be mature enough for almost all users well before 1.0 is released, but that1.0 will still thankfully have the freedom to make any changes that are necessary before something gets declared as the finished product. -- John On Dec 10, 2014, at 4:45 PM, David Anthoff anth...@berkeley.edumailto:anth...@berkeley.edu wrote: I hear you, and I didn’t think much before sending my email. Couple of points: I totally agree this should certainly not be on the homepage. I also agree that there is no need for a detailed schedule, deadlines or anything like that. I think the only thing that would be immensely helpful at least for me is just a very high level idea of what the core team is thinking about a roadmap/timing. Do you expect a 1.0 more in 10 years, or more in 1 year? Do you right now expect there to be a 0.5, 0.6, or many more releases before a 1.0? My gut guess is that the core team has an idea about those kinds of questions, and it would be great if you could share that kind of stuff from time to time. Maybe one idea here would be that the core team just sends out a brief email after a major release what the current thinking is about the next version and the road to 1.0? Such an email could be fuzzy and non-committal if the plans are fuzzy, but that in itself would also be valuable information for us users. I am following the issue tracker and am subscribed to the email lists, and I don’t get any sense/picture about those kind of high level questions from those sources. Cheers, David From: julia-users@googlegroups.commailto:julia-users@googlegroups.com [mailto:julia-users@googlegroups.com] On Behalf Of Tony Kelman Sent: Wednesday, December 10, 2014 4:31 PM To: julia-users@googlegroups.commailto:julia-users@googlegroups.com Subject: Re: [julia-users] Re: home page content -1 on trying to put plans, schedule, roadmap on the website. This week in Julia was a great contribution to the community but evidently took more effort than Matt had time to keep up with. New features get developed as the PR's for them get worked on and finished. You can subscribe to just the subset of issues/PR's for things you (along with everyone else) are eagerly awaiting. Better yet, help with testing and code review if you can. We have been doing a good job of monthly backport bugfix releases, we should be able to continue doing that. But 0.4 is still unstable and has several big-ticket items still open and being worked on (check the milestones on github). It's too early to try to make time estimates, if people are impatient and want a release sooner it's not going to be possible without punting on a number of targeted features and pushing them back to 0.5 or later. On Wednesday, December 10, 2014 1:58:52 PM UTC-8, Randy Zwitch wrote: I think it would please everyone if you moved daily televised scrums. On Wednesday, December 10, 2014