Re: [julia-users] linspace-like range generators and keyword ordering

[Josh Langsfeld]

On Saturday, November 14, 2015 at 10:59:26 AM UTC-5, MA Laforge wrote:
>
>
>>   - `immutable KD{Symbol} end` does not force the type parameter to be a 
>> symbol. It does the same thing as the more typical 'T' (which could be 
>> :tfund, Int, Array{Symbol,2}, or anything else).
>>
> Hmm... I have been having trouble with this one.  I know I have succeeded 
> with things like "abstract KD{T<:Number}" - but I don't seem to be able to 
> do "abstract KD{T<:Symbol}", or "abstract KD{T::Symbol}".  I don't 
> understand why I cannot get this level of control.  I would appreciate if 
> you would have more insight on this part.
>

I've also wanted to use the "abstract KD{S::Symbol}" syntax before and it 
has come up on github once or twice 
(https://github.com/JuliaLang/julia/issues/9580#issuecomment-101408507). 
Mostly it doesn't seem to let you do anything that you can't do now (by 
leaving off the type qualifier) and so it hasn't been prioritized on 
anyone's list.

[julia-users] Re: how to specify degrees of freedom for OneSampleTTest?

[Evan]

My data are not statistically independent, that is why I want to use N*.

So what I've done is modify a function I found here:
http://stackoverflow.com/questions/20425473/calculate-confidence-interval-in-julia

to do what I want:

function t_test(x::Array{Float64,1};
nstar=0, conf_level=0.95)
alpha = (1. - conf_level)
if nstar == 0
nstar=length(x) - 1
end
tstar = quantile(TDist(nstar - 1), 1 - alpha / 2)
SE = std(x) / sqrt(nstar)
lo, hi = mean(x) + [-1, 1] * tstar * SE
end





On Tuesday, November 10, 2015 at 4:48:39 PM UTC+1, j verzani wrote:
>
> If you have the sample data, it is just the length of the data minus 1, so 
> does not need to be specified, as it is computed.
>
> On Tuesday, November 10, 2015 at 9:47:43 AM UTC-5, Evan wrote:
>>
>> I want to do the following with some time series data:
>>
>> Calculate 95% confidence interval for mean defined as 
>> +/-sigma(x)qt(0.025, N*-1) / sqrt(N*) where sigma(x) is the standard 
>> deviation at any point x and qt(0.025, N*-1) is the 2.5 percentage point of 
>> the Student-t distribution with N*-1 degrees of freedom.
>>
>> I have been trying to do this with HypothesisTests' ci() and 
>> OneSampleTTest() functions, but as far as I can see OneSampleTTest() does 
>> not allow me to choose the number of degrees of freedom.
>>
>> julia> t = OneSampleTTest(arr)
>> One sample t-test
>> -
>> Population details:
>> parameter of interest:   Mean
>> value under h_0: 0
>> point estimate:  0.3835
>> 95% confidence interval: (0.14217301685460967,0.6248269831453903)
>>
>> Test summary:
>> outcome with 95% confidence: reject h_0
>> two-sided p-value:   0.0057946078675091515 (very significant)
>>
>> Details:
>> number of observations:   10
>> t-statistic:  3.5948622927526235
>> degrees of freedom:   9
>> empirical standard error: 0.10668002520517972
>>
>>
>> julia> t.df
>> 9
>> julia> ci(t, 0.025, tail=:both)
>> (0.09706297518543938,0.6699370248145606)
>>
>>
>> Can anybody point me in the right direction?
>>
>>
>>
>>

Re: [julia-users] Creating a stable version of Julia + Packages for a semester long course?

[Avik Sengupta]

It is now possible to run Juliabox locally, but it's not a trivial 
setup: https://github.com/JuliaLang/JuliaBox/blob/master/docs/INSTALL.MD

On Monday, 16 November 2015 09:14:57 UTC, Sheehan Olver wrote:
>
>
> You can do this locally?   I tried using JuliaBox in my classes last year, 
> but it was a bit of disaster, as it was unreliable.
>
>
>
>
> On 16 Nov 2015, at 7:53 PM, Sisyphuss  
> wrote:
>
> Run a JuliaBox server?
>
> On Monday, November 16, 2015 at 9:05:38 AM UTC+1, Sheehan Olver wrote:
>>
>> Another requirement is that the packages are shared across users, to save 
>> disk space.  Gadfly + PyPlot + IJulia (with Conda.jl version of Jupyter) 
>> takes over 750MB.   Does .julia need to be writable?  If not, I guess both 
>> options are still possible.
>>
>> On Monday, November 16, 2015 at 2:05:45 PM UTC+11, Sheehan Olver wrote:
>>>
>>>
>>> I'm trying to figure out the "best" way to create a stable version of 
>>> Julia + Gadfly + PyPlot + IJulia (+ other packages?) for a semester long 
>>> course.  I don't want to have the students run Pkg.add(...)/Pkg.update(), 
>>> as packages have a tendency to occasionally break on updates, and it's a 
>>> headache dealing with this during the lecture.
>>>
>>> Two possible solutions I can think of of are:
>>>
>>> 1)  Prebake a .julia folder that contains all the necessary resources, 
>>> with a script to reset in case the students break it with Pkg.update().
>>> 2)  Use system image
>>>
>>> http://docs.julialang.org/en/release-0.4/devdocs/sysimg/
>>>
>>> that includes all the necessary packages.   It's not really clear how to 
>>> do this from the documentation, though.   I'm also not sure how that would 
>>> interact with Pkg.update() though, so probably instructions to delete 
>>> .julia would also need to be given.
>>>
>>>
>>> Any other options I'm missing?  If 2 is recommended, any tutorial how to 
>>> do this?
>>>
>>
>

[julia-users] Re: Progress meter for parallel workers

[bernhard]

related to this I would welcome if it were possible to show the progress of 
a pmap() statement.
It is easy, to have each worker display the instance number which is being 
processed (say if pmap goes over a range 1:n). But I do not know how to 
show progress and estimated time left

Am Sonntag, 15. November 2015 16:35:24 UTC+1 schrieb Jason Eckstein:
>
>
>
> I was playing around with the ProgressMeter.jl package by Tim Holy and it 
> works great for a single threaded loop.  I was wondering if there's any way 
> to run several loops on multiple workers and have a set of progress bars 
> update in real time.  When I tried this with Tim's package it just said 
> Progress: 100% Time:  after each was complete but nothing during 
> intermediate progress.  What I had in mind was a display like
> Worker 1:  Progress: X%...
> Worker 2: Progress: Y% 
> that update in real time but have multiple lines to represent the 
> different workers.
>

Re: [julia-users] linspace-like range generators and keyword ordering

[Matt Bauman]

Instead of keyword argument ordering, you could use an "Exactly" wrapper 
type.  This would allow you to dispatch and potentially error if the exact 
constraints cannot be satisfied.

immutable Exactly{T}
val::T
end

foo(Exactly(1.0), 2.0, Exactly(3.0)) # the second argument might get fudged

(Or, conversely, consider "bare" arguments exact and use an "Approximately" 
wrapper).

On Saturday, November 14, 2015 at 10:59:26 AM UTC-5, MA Laforge wrote:

> This looks like overengineering of what should be a simple problem. Why do 
>> you want turn the keywords into arguments for dispatch? Dispatch is best 
>> when you need different behavior for different types but here all your 
>> input and output types are fixed.
>>
>
> I suppose there is some truth to that.  I also look at Julia's dispatch 
> system as an alternative to a case statement.  But, unlike the case 
> statement, you don't need to register the new function in the case 
> statement.  The function signature (that includes KD) takes care of that.
>
> You may have already known, but you can work with your keywords by doing 
>> `kwdict = Dict(kwargs)`. That will let you do stuff like `haskey(kwdict, 
>> :ts)` to decide which timespace algorithm to use.
>>
> Interesting... I never thought of that (not sure why).  Thanks.
>  
>
>> Some other points for future reference:
>>
>>   - `immutable KD{Symbol} end` does not force the type parameter to be a 
>> symbol. It does the same thing as the more typical 'T' (which could be 
>> :tfund, Int, Array{Symbol,2}, or anything else).
>>
> Hmm... I have been having trouble with this one.  I know I have succeeded 
> with things like "abstract KD{T<:Number}" - but I don't seem to be able to 
> do "abstract KD{T<:Symbol}", or "abstract KD{T::Symbol}".  I don't 
> understand why I cannot get this level of control.  I would appreciate if 
> you would have more insight on this part.
>  
>
>>   - timespace() and timespace(; kwargs...) are not separate methods. The 
>> second one actually overrides any definition in the first one.
>>
> I should have known better than this.  I did not notice that mistake.  
> Thanks.
>

[julia-users] Re: push! performance

[Tomas Lycken]

Making sure that precompilation and gc don’t factor into the result, I get 
quite different timings:

julia> gc(); @time sizehint!(a, 10_000_000);
  0.001493 seconds (46 allocations: 76.296 MB, 304.61% gc time)

julia> gc(); @time b = zeros(Int, 10_000_000);
  0.021997 seconds (38 allocations: 76.296 MB, 0.70% gc time)

julia> versioninfo()
Julia Version 0.4.1
Commit cbe1bee* (2015-11-08 10:33 UTC)
Platform Info:
  System: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

The gc count on sizehint!() is obviously bogus, but the difference in 
timing is much smaller. I wouldn’t doubt, however, that there is 
performance to be gained from the fact that sizehint! doesn’t have to write 
zeros to all the memory it allocates. My point was rather to illustrate 
*when* allocation occurs, and how *usage* of the array differs depending on 
approach, than the minute implementation details.

// T

On Monday, November 16, 2015 at 4:09:54 PM UTC+1, Seth wrote:

I'm not sure the equivalence is entirely accurate:
>
> julia> a = Vector{Int}()
> 0-element Array{Int64,1}
>
> julia> Base.summarysize(a)
> 0
>
> julia> @time sizehint!(a,10_000_000)
> 0.96 seconds (149 allocations: 76.304 MB)
> 0-element Array{Int64,1}
>
> julia> Base.summarysize(a)
> 0
>
> julia> @time b = zeros(Int,10_000_000);
> 0.037202 seconds (35 allocations: 76.295 MB, 64.13% gc time)
>
> julia> Base.summarysize(b)
> 8000
>
>
>
> On Monday, November 16, 2015 at 1:10:36 AM UTC-8, Tomas Lycken wrote:
>>
>> sizehint! preallocates for you, with comparable cost to calling e.g. 
>> zeroes, but lets you treat the array semantically the same way as a 
>> non-preallocated one (but without the cost for reallocation). Hopefully, 
>> these comments highlight the differences between the various appraches:
>>
>> N = 10_000
>> A = Array(Float64,0) 
>> sizehint!(A, 10_000) # this preallocates memory for 10k elements
>> B = Array(Float64,0)
>> C = zeros(10_000) # this also preallocates memory for 10k elements
>>
>> # now, A and C are pre-allocated, while B is not# however, A and B are 
>> semantically equivalent (0-length) vectors,
>> # while C is already of length 10 000:
>> println(length(A)) # 0
>> println(length(B)) # 0
>> println(length(C)) # 1
>>
>> for i in 1:10_000
>>push!(A, i) # no reallocation happens here, because we did it with 
>> sizehint!
>>push!(B, i) # this will re-allocate B every now and then
>>C[i] = i # can't use push! here, but must manually track index instead
>> end
>>
>> I don't know what `dynamic` does in this context, and I can't find it in 
>> the docs, so can't help you there :)
>>
>> // T
>>
>> On Monday, November 16, 2015 at 2:07:13 AM UTC+1, Seth wrote:
>>
>> What happens if you use sizehint!() with dynamic()?
>>>
>>> On Sunday, November 15, 2015 at 3:35:45 PM UTC-8, Steven G. Johnson 
>>> wrote:

 function prealloc(n)
 a = Array(Int, n)
 for i = 1:n
 a[i] = i
 end
 return a
 end
 function dynamic(n)
 a = Int[]
 for i = 1:n
 push!(a, i)
 end
 return a
 end
 @time prealloc(10^7);
 @time dynamic(10^7);


 On my machine, the preallocated version is 2.5–3x faster.  A 
 significant but not overwhelming margin.

>>> ​
>>
> ​

[julia-users] Re: Progress meter for parallel workers

[Tomas Lycken]

There has been some discussion about this, 
see https://github.com/timholy/ProgressMeter.jl/issues/9 
and https://github.com/timholy/ProgressMeter.jl/issues/32

// T

On Monday, November 16, 2015 at 1:53:22 PM UTC+1, bernhard wrote:
>
> related to this I would welcome if it were possible to show the progress 
> of a pmap() statement.
> It is easy, to have each worker display the instance number which is being 
> processed (say if pmap goes over a range 1:n). But I do not know how to 
> show progress and estimated time left
>
> Am Sonntag, 15. November 2015 16:35:24 UTC+1 schrieb Jason Eckstein:
>>
>>
>>
>> I was playing around with the ProgressMeter.jl package by Tim Holy and it 
>> works great for a single threaded loop.  I was wondering if there's any way 
>> to run several loops on multiple workers and have a set of progress bars 
>> update in real time.  When I tried this with Tim's package it just said 
>> Progress: 100% Time:  after each was complete but nothing during 
>> intermediate progress.  What I had in mind was a display like
>> Worker 1:  Progress: X%...
>> Worker 2: Progress: Y% 
>> that update in real time but have multiple lines to represent the 
>> different workers.
>>
>

[julia-users] Julia users at SC'15?

[Mark Dewing]

SC15 (Supercomputing) is happening in Austin this week.
Are there any Julia users/developers/fans here who would like to meet up?

- Mark

[julia-users] Fitting with x- and y-errors

[Daniel Hauck]

Hi,

is there any possibility to consider x- and y-errors when fitting (e.g. 
using absolute least square method)? In ROOT I would simply use 
TGraphErrors ...

LsqFit and other packages I found only support y-errors.

Daniel

[julia-users] Re: Google releases TensorFlow as open source

[Ravish Mishra]

Hi Viral, 

I want to be a part of JuliaML.

~ Ravish

On Wednesday, November 11, 2015 at 4:48:07 PM UTC+5:30, Viral Shah wrote:
>
> I think TensorFlow.jl is a great idea. Also their distributed computation 
> framework is also the kind that we want to have in Julia.
>
> I have created JuliaML. Send me email if you want to be part of it, and I 
> will make you an owner. Perhaps we can even move some of the JuliaStats ML 
> projects to JuliaML.
>
> -viral
>
> On Wednesday, November 11, 2015 at 11:27:21 AM UTC+5:30, Valentin Churavy 
> wrote:
>>
>> It fits in the same niche that Mocha.jl and MXNet.jl are filling right 
>> now. MXNet is a ML library that shares many of the same design ideas of 
>> TensorFlow and has great Julia support https://github.com/dmlc/MXNet.jl
>>
>>
>> On Wednesday, 11 November 2015 01:04:00 UTC+9, Randy Zwitch wrote:
>>>
>>> For me, the bigger question is how does TensorFlow fit in/fill in gaps 
>>> in currently available Julia libraries? I'm not saying that someone who is 
>>> sufficiently interested shouldn't wrap the library, but it'd be great to 
>>> identify what major gaps remain in ML for Julia and figure out if 
>>> TensorFlow is the right way to proceed. 
>>>
>>> We're certainly nowhere near the R duplication problem yet, but 
>>> certainly we're already repeating ourselves in many areas.
>>>
>>> On Monday, November 9, 2015 at 4:02:36 PM UTC-5, Phil Tomson wrote:

 Google has released it's deep learning library called TensorFlow as 
 open source code:

 https://github.com/tensorflow/tensorflow

 They include Python bindings, Any ideas about how easy/difficult it 
 would be to create Julia bindings?

 Phil

>>>

[julia-users] Re: Creating a stable version of Julia + Packages for a semester long course?

[John Lambert]

It may be easiest to use Sagemath Cloud.  At very least it would take the 
burden off of the students to have to worry about installing software at 
all.  There are also a lot of other benefits including unlimited undo a.k.a 
Google Docs, some class administration support and being web hosted so 
hardware failures don't mean lost work.  Check it out here 
- https://cloud.sagemath.com/ - 
http://sagemath.blogspot.com/2013/12/the-sagemath-cloud-minute-elevator.html 
- http://sagemath.blogspot.com/2014/10/a-non-technical-overview-of.html 
 for teaching 
- http://www.beezers.org/blog/bb/2015/09/grading-in-sagemathcloud/ 
or https://www.youtube.com/watch?v=-SqzarfTqRY

On Sunday, November 15, 2015 at 10:05:45 PM UTC-5, Sheehan Olver wrote:
>
>
> I'm trying to figure out the "best" way to create a stable version of 
> Julia + Gadfly + PyPlot + IJulia (+ other packages?) for a semester long 
> course.  I don't want to have the students run Pkg.add(...)/Pkg.update(), 
> as packages have a tendency to occasionally break on updates, and it's a 
> headache dealing with this during the lecture.
>
> Two possible solutions I can think of of are:
>
> 1)  Prebake a .julia folder that contains all the necessary resources, 
> with a script to reset in case the students break it with Pkg.update().
> 2)  Use system image
>
> http://docs.julialang.org/en/release-0.4/devdocs/sysimg/
>
> that includes all the necessary packages.   It's not really clear how to 
> do this from the documentation, though.   I'm also not sure how that would 
> interact with Pkg.update() though, so probably instructions to delete 
> .julia would also need to be given.
>
>
> Any other options I'm missing?  If 2 is recommended, any tutorial how to 
> do this?
>

[julia-users] Re: indexing with non Integer Reals is deprecated

[Peter Kovesi]

Can I add a plea for allowing indexing with integers that are represented 
by a floating point type

Following the principle of 'minimum surprise': If I have an integer just 
that happens to be represented via a floating point type I would still like 
to be able to use it as an integer.

After all we are happy with the idea of allowing mathematical operations 
between integer and float types via  automatic promotion of the integer. 
 That is, we are not required to cast the integer to a float in order to 
multiply the two.  Indexing with integer values that are represented by 
floats would seem an appropriate place for automatic demotion. Please do 
not make me write a[round(Int, 1.0)] !

As a long time Matlab user who is attempting to reform himself I confess to 
having some mixed feelings about Julia.  Much of the time I find Julia very 
elegant and I find coding flows nicely but there are also times when I find 
Julia to be excessively finicky and a generator of too much 'surprise'. 
Indeed, sometimes I feel as if I might as well be coding in C.  I am hoping 
much of this will go as I learn the language better.   Still, let me say 
Julia is an impressive language and I greatly appreciate the work of the 
developers.  I am looking forward to its future development.

Cheers
Peter

[julia-users] Re: Creating a stable version of Julia + Packages for a semester long course?

[Seth]

Avik,

what happens with precompilation if .julia isn't writable? (Too chicken to 
try it out here.)


On Monday, November 16, 2015 at 4:30:34 AM UTC-8, Avik Sengupta wrote:
>
> .julia does not need to be writable for running Julia code. It needs to be 
> writable for package operations (add, update etc). I run a shared .julia on 
> a multiuser linux machine, it works well. The only issue is that doing 
> Pkg.update() is a bit of a pain, but we only do that very rarely. 
>
> Regards
> -
> Avik
>
> On Monday, 16 November 2015 08:05:38 UTC, Sheehan Olver wrote:
>>
>> Another requirement is that the packages are shared across users, to save 
>> disk space.  Gadfly + PyPlot + IJulia (with Conda.jl version of Jupyter) 
>> takes over 750MB.   Does .julia need to be writable?  If not, I guess both 
>> options are still possible.
>>
>> On Monday, November 16, 2015 at 2:05:45 PM UTC+11, Sheehan Olver wrote:
>>>
>>>
>>> I'm trying to figure out the "best" way to create a stable version of 
>>> Julia + Gadfly + PyPlot + IJulia (+ other packages?) for a semester long 
>>> course.  I don't want to have the students run Pkg.add(...)/Pkg.update(), 
>>> as packages have a tendency to occasionally break on updates, and it's a 
>>> headache dealing with this during the lecture.
>>>
>>> Two possible solutions I can think of of are:
>>>
>>> 1)  Prebake a .julia folder that contains all the necessary resources, 
>>> with a script to reset in case the students break it with Pkg.update().
>>> 2)  Use system image
>>>
>>> http://docs.julialang.org/en/release-0.4/devdocs/sysimg/
>>>
>>> that includes all the necessary packages.   It's not really clear how to 
>>> do this from the documentation, though.   I'm also not sure how that would 
>>> interact with Pkg.update() though, so probably instructions to delete 
>>> .julia would also need to be given.
>>>
>>>
>>> Any other options I'm missing?  If 2 is recommended, any tutorial how to 
>>> do this?
>>>
>>

Re: [julia-users] how to make my own 3d array type

[Evan]

Thank you, Tim, I understand the motivation for the approach in the link 
you sent.

I'm still unable to create the type that I want. I actually want two arrays 
in my type, one 2d and the other 3d.

First though, I'm still stuck on just the 2d when I try to implement your 
suggestion:
julia> type MyType{T,N,A<:AbstractArray} <: AbstractArray{T,N}
   var::A
   end

julia> MyType{T,N}(var::AbstractArray{T,N}) = MyType{T,N,typeof(var)}(var)
MyType{T,N,A<:AbstractArray{T,N}}

julia> aa = MyType(zeros(2,3))
Error showing value of type MyType{Float64,2,Array{Float64,2}}:
ERROR: MethodError: `size` has no method matching size(::MyType{Float64,2,
Array{Float64,2}})
Closest candidates are:
  size{T,n}(::AbstractArray{T,n}, ::Any)
  size(::Any, ::Integer, ::Integer, ::Integer...)
 in showarray at show.jl:1231
 in anonymous at replutil.jl:29
 in with_output_limit at ./show.jl:1271
 in writemime at replutil.jl:28
 in display at REPL.jl:114
 in display at REPL.jl:117
 [inlined code] from multimedia.jl:151
 in display at multimedia.jl:162
 in print_response at REPL.jl:134
 in print_response at REPL.jl:121
 in anonymous at REPL.jl:624
 in run_interface at ./LineEdit.jl:1610
 in run_frontend at ./REPL.jl:863
 in run_repl at ./REPL.jl:167
 in _start at ./client.jl:420

julia>

Performance is important, but it's not clear what I should use in place of 
abstract types; I've tried replacing AbstractArray with just Array but that 
does not appear to work.




On Monday, November 16, 2015 at 1:03:05 PM UTC+1, Tim Holy wrote:
>
> This fixes two problems: 
>
> type MyType{T,N,A<:AbstractArray} <: AbstractArray{T,N} 
> var::A 
> end 
>
> MyType{T,N}(var::AbstractArray{T,N}) = MyType{T,N,typeof(var)}(var) 
>
>
>
> If performance matters, you should not use abstract types for fields. See 
>
> http://docs.julialang.org/en/stable/manual/faq/#how-do-abstract-or-ambiguous-fields-in-types-interact-with-the-compiler
>  
> and the section after that. 
>
> --Tim 
>
>
> On Monday, November 16, 2015 03:54:15 AM Evan wrote: 
> > For a 2d array the following works: 
> > 
> > type mytype{T} 
> > var :: AbstractMatrix{T} 
> > end 
> > 
> > julia> t = mytype(zeros(4, 3)) 
> > mytype{Float64}(4x3 Array{Float64,2}: 
> >  0.0  0.0  0.0 
> >  0.0  0.0  0.0 
> >  0.0  0.0  0.0 
> >  0.0  0.0  0.0) 
> > 
> > 
> > 
> > But how do I extend this to a 3d array? 
> > 
> > julia> t = mytype(zeros(2, 4, 3)) 
> > ERROR: MethodError: `convert` has no method matching 
> convert(::Type{mytype{T 
> > }}, ::Array{Float64,3}) 
> > This may have arisen from a call to the constructor mytype{T}(...), 
> > since type constructors fall back to convert methods. 
> > Closest candidates are: 
> >   call{T}(::Type{T}, ::Any) 
> >   convert{T}(::Type{T}, ::T) 
> >   mytype{T}(::AbstractArray{T,2}) 
> >  in call at essentials.jl:56 
>
>

Re: [julia-users] how to make my own 3d array type

[Tim Holy]

Totally different error (it's a missing size method). You also need to read 
this:
http://docs.julialang.org/en/stable/manual/interfaces/#abstract-arrays

--Tim

On Monday, November 16, 2015 06:08:15 AM Evan wrote:
> Thank you, Tim, I understand the motivation for the approach in the link
> you sent.
> 
> I'm still unable to create the type that I want. I actually want two arrays
> in my type, one 2d and the other 3d.
> 
> First though, I'm still stuck on just the 2d when I try to implement your
> suggestion:
> julia> type MyType{T,N,A<:AbstractArray} <: AbstractArray{T,N}
>var::A
>end
> 
> julia> MyType{T,N}(var::AbstractArray{T,N}) = MyType{T,N,typeof(var)}(var)
> MyType{T,N,A<:AbstractArray{T,N}}
> 
> julia> aa = MyType(zeros(2,3))
> Error showing value of type MyType{Float64,2,Array{Float64,2}}:
> ERROR: MethodError: `size` has no method matching size(::MyType{Float64,2,
> Array{Float64,2}})
> Closest candidates are:
>   size{T,n}(::AbstractArray{T,n}, ::Any)
>   size(::Any, ::Integer, ::Integer, ::Integer...)
>  in showarray at show.jl:1231
>  in anonymous at replutil.jl:29
>  in with_output_limit at ./show.jl:1271
>  in writemime at replutil.jl:28
>  in display at REPL.jl:114
>  in display at REPL.jl:117
>  [inlined code] from multimedia.jl:151
>  in display at multimedia.jl:162
>  in print_response at REPL.jl:134
>  in print_response at REPL.jl:121
>  in anonymous at REPL.jl:624
>  in run_interface at ./LineEdit.jl:1610
>  in run_frontend at ./REPL.jl:863
>  in run_repl at ./REPL.jl:167
>  in _start at ./client.jl:420
> 
> julia>
> 
> Performance is important, but it's not clear what I should use in place of
> abstract types; I've tried replacing AbstractArray with just Array but that
> does not appear to work.
> 
> On Monday, November 16, 2015 at 1:03:05 PM UTC+1, Tim Holy wrote:
> > This fixes two problems:
> > 
> > type MyType{T,N,A<:AbstractArray} <: AbstractArray{T,N}
> > 
> > var::A
> > 
> > end
> > 
> > MyType{T,N}(var::AbstractArray{T,N}) = MyType{T,N,typeof(var)}(var)
> > 
> > 
> > 
> > If performance matters, you should not use abstract types for fields. See
> > 
> > http://docs.julialang.org/en/stable/manual/faq/#how-do-abstract-or-ambiguo
> > us-fields-in-types-interact-with-the-compiler and the section after that.
> > 
> > --Tim
> > 
> > On Monday, November 16, 2015 03:54:15 AM Evan wrote:
> > > For a 2d array the following works:
> > > 
> > > type mytype{T}
> > > 
> > > var :: AbstractMatrix{T}
> > > 
> > > end
> > > 
> > > julia> t = mytype(zeros(4, 3))
> > > 
> > > mytype{Float64}(4x3 Array{Float64,2}:
> > >  0.0  0.0  0.0
> > >  0.0  0.0  0.0
> > >  0.0  0.0  0.0
> > >  0.0  0.0  0.0)
> > > 
> > > But how do I extend this to a 3d array?
> > > 
> > > julia> t = mytype(zeros(2, 4, 3))
> > > ERROR: MethodError: `convert` has no method matching
> > 
> > convert(::Type{mytype{T
> > 
> > > }}, ::Array{Float64,3})
> > > This may have arisen from a call to the constructor mytype{T}(...),
> > > since type constructors fall back to convert methods.
> > > 
> > > Closest candidates are:
> > >   call{T}(::Type{T}, ::Any)
> > >   convert{T}(::Type{T}, ::T)
> > >   mytype{T}(::AbstractArray{T,2})
> > >  
> > >  in call at essentials.jl:56

[julia-users] Re: push! performance

[Seth]

I'm not sure the equivalence is entirely accurate:

julia> a = Vector{Int}()
0-element Array{Int64,1}

julia> Base.summarysize(a)
0

julia> @time sizehint!(a,10_000_000)
0.96 seconds (149 allocations: 76.304 MB)
0-element Array{Int64,1}

julia> Base.summarysize(a)
0

julia> @time b = zeros(Int,10_000_000);
0.037202 seconds (35 allocations: 76.295 MB, 64.13% gc time)

julia> Base.summarysize(b)
8000



On Monday, November 16, 2015 at 1:10:36 AM UTC-8, Tomas Lycken wrote:
>
> sizehint! preallocates for you, with comparable cost to calling e.g. 
> zeroes, but lets you treat the array semantically the same way as a 
> non-preallocated one (but without the cost for reallocation). Hopefully, 
> these comments highlight the differences between the various appraches:
>
> N = 10_000
> A = Array(Float64,0) 
> sizehint!(A, 10_000) # this preallocates memory for 10k elements
> B = Array(Float64,0)
> C = zeros(10_000) # this also preallocates memory for 10k elements
>
> # now, A and C are pre-allocated, while B is not# however, A and B are 
> semantically equivalent (0-length) vectors,
> # while C is already of length 10 000:
> println(length(A)) # 0
> println(length(B)) # 0
> println(length(C)) # 1
>
> for i in 1:10_000
>push!(A, i) # no reallocation happens here, because we did it with 
> sizehint!
>push!(B, i) # this will re-allocate B every now and then
>C[i] = i # can't use push! here, but must manually track index instead
> end
>
> I don't know what `dynamic` does in this context, and I can't find it in 
> the docs, so can't help you there :)
>
> // T
>
> On Monday, November 16, 2015 at 2:07:13 AM UTC+1, Seth wrote:
>
> What happens if you use sizehint!() with dynamic()?
>>
>> On Sunday, November 15, 2015 at 3:35:45 PM UTC-8, Steven G. Johnson wrote:
>>>
>>> function prealloc(n)
>>> a = Array(Int, n)
>>> for i = 1:n
>>> a[i] = i
>>> end
>>> return a
>>> end
>>> function dynamic(n)
>>> a = Int[]
>>> for i = 1:n
>>> push!(a, i)
>>> end
>>> return a
>>> end
>>> @time prealloc(10^7);
>>> @time dynamic(10^7);
>>>
>>>
>>> On my machine, the preallocated version is 2.5–3x faster.  A significant 
>>> but not overwhelming margin.
>>>
>> ​
>

[julia-users] Re: Creating a stable version of Julia + Packages for a semester long course?

[Avik Sengupta]

.julia does not need to be writable for running Julia code. It needs to be 
writable for package operations (add, update etc). I run a shared .julia on 
a multiuser linux machine, it works well. The only issue is that doing 
Pkg.update() is a bit of a pain, but we only do that very rarely. 

Regards
-
Avik

On Monday, 16 November 2015 08:05:38 UTC, Sheehan Olver wrote:
>
> Another requirement is that the packages are shared across users, to save 
> disk space.  Gadfly + PyPlot + IJulia (with Conda.jl version of Jupyter) 
> takes over 750MB.   Does .julia need to be writable?  If not, I guess both 
> options are still possible.
>
> On Monday, November 16, 2015 at 2:05:45 PM UTC+11, Sheehan Olver wrote:
>>
>>
>> I'm trying to figure out the "best" way to create a stable version of 
>> Julia + Gadfly + PyPlot + IJulia (+ other packages?) for a semester long 
>> course.  I don't want to have the students run Pkg.add(...)/Pkg.update(), 
>> as packages have a tendency to occasionally break on updates, and it's a 
>> headache dealing with this during the lecture.
>>
>> Two possible solutions I can think of of are:
>>
>> 1)  Prebake a .julia folder that contains all the necessary resources, 
>> with a script to reset in case the students break it with Pkg.update().
>> 2)  Use system image
>>
>> http://docs.julialang.org/en/release-0.4/devdocs/sysimg/
>>
>> that includes all the necessary packages.   It's not really clear how to 
>> do this from the documentation, though.   I'm also not sure how that would 
>> interact with Pkg.update() though, so probably instructions to delete 
>> .julia would also need to be given.
>>
>>
>> Any other options I'm missing?  If 2 is recommended, any tutorial how to 
>> do this?
>>
>

Re: [julia-users] how to make my own 3d array type

[Tim Holy]

This fixes two problems:

type MyType{T,N,A<:AbstractArray} <: AbstractArray{T,N}
var::A
end

MyType{T,N}(var::AbstractArray{T,N}) = MyType{T,N,typeof(var)}(var)



If performance matters, you should not use abstract types for fields. See 
http://docs.julialang.org/en/stable/manual/faq/#how-do-abstract-or-ambiguous-fields-in-types-interact-with-the-compiler
 and the section after that.

--Tim


On Monday, November 16, 2015 03:54:15 AM Evan wrote:
> For a 2d array the following works:
> 
> type mytype{T}
> var :: AbstractMatrix{T}
> end
> 
> julia> t = mytype(zeros(4, 3))
> mytype{Float64}(4x3 Array{Float64,2}:
>  0.0  0.0  0.0
>  0.0  0.0  0.0
>  0.0  0.0  0.0
>  0.0  0.0  0.0)
> 
> 
> 
> But how do I extend this to a 3d array?
> 
> julia> t = mytype(zeros(2, 4, 3))
> ERROR: MethodError: `convert` has no method matching convert(::Type{mytype{T
> }}, ::Array{Float64,3})
> This may have arisen from a call to the constructor mytype{T}(...),
> since type constructors fall back to convert methods.
> Closest candidates are:
>   call{T}(::Type{T}, ::Any)
>   convert{T}(::Type{T}, ::T)
>   mytype{T}(::AbstractArray{T,2})
>  in call at essentials.jl:56

Re: [julia-users] Julia users at SC'15?

[Erik Schnetter]

Mark

You beat me to sending such an email by a few minutes.

Is there a particular Julian place where we could meet? Is there e.g. an
MIT booth at tonight's reception?

-erik

On Sun, Nov 15, 2015 at 8:42 AM, Mark Dewing  wrote:

> SC15 (Supercomputing) is happening in Austin this week.
> Are there any Julia users/developers/fans here who would like to meet up?
>
> - Mark
>



-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

Re: [julia-users] Colormaps composing images

[Tobias Knopp]

Thanks a lot for you input. I appreciate all this wonderful infrastructure 
around Images.jl

Since I am no expert in image coloring I was simply not sure whether 
everything is already there or not. 
It seems in the case of alpha blending there are some missing parts. I 
still do not get how the "Overlay" functions work without an alpha channel 
but I will try to understand it.

And yes I mean that kind of interpolation. If have implemented now a 
function 

  colorize{T<:Colorant}(x::Real,wmin::Real,wmax::Real,cmap::Vector{T})

that takes as input a real value and two window parameters that are mapped 
to a certain color. And since the Colors.jl package has only RGB colormaps 
I have additionally implemented RGBA colormaps. I am currently not sure if 
this would fit into the existing package infrastructure...

Cheers,

Tobias



Am Montag, 16. November 2015 12:39:30 UTC+1 schrieb Tim Holy:
>
> On Monday, November 16, 2015 02:25:43 AM Tobias Knopp wrote: 
> > After investigating this a little further I found that the key things I 
> > am/was missing are: 
> > - How to compose two RGBA values a and b. It turns out that this is the 
> > "over" operation given by 
> > 
> >   c = alpha*b + (1-alpha)*a 
> > 
> > if alpha is the alpha channel of b and a has alpha=1. Do I have missed 
> it 
> > or is the over operation somewhere implemented in a package? 
>
> I'm not aware of a package that implements alpha-compositing. Perhaps it 
> could 
> be added to ColorVectorSpace (which might then need a new name), or 
> perhaps 
> better it could be its own package. If you're interested in this, I'd 
> recommend starting by creating a type that makes it clear whether you're 
> working with pre-multiplied alpha or not, see 
> https://en.wikipedia.org/wiki/Alpha_compositing#Description 
>
> > - The second thing is colormaps that include an alpha channel. This 
> allows 
> > to overlay data in a very flexible way. Otherwise its not really clear 
> how 
> > to compose things 
>
> ImageCmap lets you define a colormap that includes an alpha channel, but 
> it 
> doesn't implement any special operations. 
>
> > 
> > - The third is: given a colormap, how to interpolate into it (+ window 
> > level, window brightness). 
>
> Not quite sure what you mean, are you asking if you can have a fractional 
> value that lies between two points in the colormap? If the operations you 
> want 
> to implement can be expressed in terms of + and * via the standard rules 
> of 
> interpolation, this one should be easy :-): just use Interpolations.jl to 
> create the colormap vector, e.g., 
>
> icmap = interpolate(cmap, BSpline(Linear())) 
>
> and pass that to ImageCmap. We can easily generalize it to allow an 
> AbstractVector. 
>
> Note that currently img[i,j] for an ImageCmap returns the index, not the 
> color 
> value. I don't remember why I did it that way, but I think that's 
> something 
> that will have to change. Just need to think of the right migration 
> strategy. 
>
> Best, 
> --Tim 
>
> > 
> > Cheers, 
> > 
> > Tobias 
> > 
> > Am Donnerstag, 12. November 2015 21:54:18 UTC+1 schrieb Tim Holy: 
> > > On Thursday, November 12, 2015 12:15:46 PM Tobias Knopp wrote: 
> > > > Thanks Tim, I tried to look into the code of Overlay but it wasn't 
> to 
> > > 
> > > clear 
> > > 
> > > > to me. In particular I am missing where the RGB(A) data is combined. 
> > > 
> > > Here: 
> > > 
> > > 
> https://github.com/timholy/Images.jl/blob/cac28026250814f6ae6594dd26e92707 
> > > 6177db60/src/overlays.jl#L60-L67> 
> > > > Is it 
> > > > really as simple as adding the individual RGB values and preventing 
> > > > overflow by clamp? 
> > > 
> > > Yes. 
> > > 
> > > > Or is there some infrastructure for color mixing in Colors.jl 
> > > 
> > > There is, but it's considerably slower. See the ColorVectorSpace.jl 
> README 
> > > for 
> > > discussion. 
> > > 
> > > > I further looked for functions for gray value mapping 
> > > 
> > > (Contrast/Brightness) 
> > > 
> > > > in Colors.jl but could not find anything. This is of course not 
> > > 
> > > complicated 
> > > 
> > > > to code but I don't want to miss an existing solution. 
> > > 
> > > The whole "MapInfo" structure is a very flexible and powerful. Search 
> for 
> > > it on 
> > > this page: 
> > > http://timholy.github.io/Images.jl/function_reference.html 
> > > From my standpoint, the best feature is that it's "lazy": you specify 
> the 
> > > transformation you want, but don't execute it until you need it. For 
> > > people 
> > > like me who routinely browse 1TB images but probably look at <1% of 
> the 
> > > raw 
> > > data in any given dataset (and who also don't have 1TB worth of 
> RAM...), 
> > > this 
> > > is quite an advantage. 
> > > 
> > > In my own work, I pretty routinely design custom MapInfo types/map 
> > > functions 
> > > for visualization purposes. For example, I can color individual blobs 
> in 
> > > each 
> > > frame of a movie with something along the lines of 
> > > 

[julia-users] Re: push! performance

[Steven G. Johnson]

On Monday, November 16, 2015 at 11:24:20 AM UTC-5, Tomas Lycken wrote:
>
> Making sure that precompilation and gc don’t factor into the result, I get 
> quite different timings:
>
> julia> gc(); @time sizehint!(a, 10_000_000);
>   0.001493 seconds (46 allocations: 76.296 MB, 304.61% gc time)
>
> julia> gc(); @time b = zeros(Int, 10_000_000);
>   0.021997 seconds (38 allocations: 76.296 MB, 0.70% gc time)
>
>
Compare to Array(Int,  10_000_000), which allocates an uninitialized array.

> ​
>>>
>> ​
>

[julia-users] Re: push! performance

[Tomas Lycken]

Yeah, that makes it even more clear that they’re very similar:

julia> gc(); @time for i=1:1_000; sizehint!(Vector{Int}(0), 1_000); end
  0.000186 seconds (2.00 k allocations: 7.721 MB)

julia> gc(); @time for i=1:1_000; Vector{Int}(1_000); end
  0.000181 seconds (1000 allocations: 7.706 MB)

(broken into smaller chunks, because the timings started to get awfully 
close to the resolution of the clock, and the gc percentage count got 
really borked…)

// T

On Monday, November 16, 2015 at 6:26:04 PM UTC+1, Steven G. Johnson wrote:


>
> On Monday, November 16, 2015 at 11:24:20 AM UTC-5, Tomas Lycken wrote:
>>
>> Making sure that precompilation and gc don’t factor into the result, I 
>> get quite different timings:
>>
>> julia> gc(); @time sizehint!(a, 10_000_000);
>>   0.001493 seconds (46 allocations: 76.296 MB, 304.61% gc time)
>>
>> julia> gc(); @time b = zeros(Int, 10_000_000);
>>   0.021997 seconds (38 allocations: 76.296 MB, 0.70% gc time)
>>
>>
> Compare to Array(Int,  10_000_000), which allocates an uninitialized array.
>
>> ​

>>> ​
>>
> ​

[julia-users] Re: indexing with non Integer Reals is deprecated

[Steven G. Johnson]

On Monday, November 16, 2015 at 11:54:39 AM UTC-5, Peter Kovesi wrote:
>
> Following the principle of 'minimum surprise': If I have an integer just 
> that happens to be represented via a floating point type I would still like 
> to be able to use it as an integer.
>

Can you give an example of where you would like to use a floating-point 
index that does not involve doing integer division with / (where you should 
probably use ÷ instead)? 

Re: [julia-users] Re: Google releases TensorFlow as open source

[Stefan Karpinski]

JuliaML is a collection of repos, not people. If you create a package that
ends up in JuliaML or make significant contributions to one of them, then
the owner of some JuliaML package may give you commit access to that
package.

On Mon, Nov 16, 2015 at 4:24 AM, Ravish Mishra 
wrote:

> Hi Viral,
>
> I want to be a part of JuliaML.
>
> ~ Ravish
>
>
> On Wednesday, November 11, 2015 at 4:48:07 PM UTC+5:30, Viral Shah wrote:
>>
>> I think TensorFlow.jl is a great idea. Also their distributed computation
>> framework is also the kind that we want to have in Julia.
>>
>> I have created JuliaML. Send me email if you want to be part of it, and I
>> will make you an owner. Perhaps we can even move some of the JuliaStats ML
>> projects to JuliaML.
>>
>> -viral
>>
>> On Wednesday, November 11, 2015 at 11:27:21 AM UTC+5:30, Valentin Churavy
>> wrote:
>>>
>>> It fits in the same niche that Mocha.jl and MXNet.jl are filling right
>>> now. MXNet is a ML library that shares many of the same design ideas of
>>> TensorFlow and has great Julia support https://github.com/dmlc/MXNet.jl
>>>
>>>
>>> On Wednesday, 11 November 2015 01:04:00 UTC+9, Randy Zwitch wrote:

 For me, the bigger question is how does TensorFlow fit in/fill in gaps
 in currently available Julia libraries? I'm not saying that someone who is
 sufficiently interested shouldn't wrap the library, but it'd be great to
 identify what major gaps remain in ML for Julia and figure out if
 TensorFlow is the right way to proceed.

 We're certainly nowhere near the R duplication problem yet, but
 certainly we're already repeating ourselves in many areas.

 On Monday, November 9, 2015 at 4:02:36 PM UTC-5, Phil Tomson wrote:
>
> Google has released it's deep learning library called TensorFlow as
> open source code:
>
> https://github.com/tensorflow/tensorflow
>
> They include Python bindings, Any ideas about how easy/difficult it
> would be to create Julia bindings?
>
> Phil
>

Re: [julia-users] Colormaps composing images

[Tim Holy]

On Monday, November 16, 2015 09:10:21 AM Tobias Knopp wrote:
> It seems in the case of alpha blending there are some missing parts. I
> still do not get how the "Overlay" functions work without an alpha channel
> but I will try to understand it.

The "Overlay" functions are barely any more sophisticated than this (and in 
fact I almost implemented them this way, then decided to try to make them a 
little more flexible):

# generate a green/magenta image from two grayscale channels
imgout[1, : , :] = img1   # magenta = red + blue
imgout[3, :,  :] = img1
imgout[2, :,  :] = img2   # the green channel
imgoutc = reinterpret(RGB{T}, imgout)

This requires no alpha blending or anything.

> And yes I mean that kind of interpolation. If have implemented now a
> function
> 
>   colorize{T<:Colorant}(x::Real,wmin::Real,wmax::Real,cmap::Vector{T})
> 
> that takes as input a real value and two window parameters that are mapped
> to a certain color. And since the Colors.jl package has only RGB colormaps
> I have additionally implemented RGBA colormaps. I am currently not sure if
> this would fit into the existing package infrastructure...

Colors.jl has "only RGB colormaps" in the sense that perhaps no one has 
written convenience functions to _generate_ a colormap with an alpha channel. 
But since we're talking about a Vector{C} with C<:Colorant rather than 
C<:Color, the generalization is very natural.

I would definitely say that what you're doing belongs in those packages. As 
always it just comes down to what people have needed and what they've bothered 
to implement.

--Tim


> 
> Cheers,
> 
> Tobias
> 
> Am Montag, 16. November 2015 12:39:30 UTC+1 schrieb Tim Holy:
> > On Monday, November 16, 2015 02:25:43 AM Tobias Knopp wrote:
> > > After investigating this a little further I found that the key things I
> > > am/was missing are:
> > > - How to compose two RGBA values a and b. It turns out that this is the
> > > "over" operation given by
> > > 
> > >   c = alpha*b + (1-alpha)*a
> > > 
> > > if alpha is the alpha channel of b and a has alpha=1. Do I have missed
> > 
> > it
> > 
> > > or is the over operation somewhere implemented in a package?
> > 
> > I'm not aware of a package that implements alpha-compositing. Perhaps it
> > could
> > be added to ColorVectorSpace (which might then need a new name), or
> > perhaps
> > better it could be its own package. If you're interested in this, I'd
> > recommend starting by creating a type that makes it clear whether you're
> > working with pre-multiplied alpha or not, see
> > https://en.wikipedia.org/wiki/Alpha_compositing#Description
> > 
> > > - The second thing is colormaps that include an alpha channel. This
> > 
> > allows
> > 
> > > to overlay data in a very flexible way. Otherwise its not really clear
> > 
> > how
> > 
> > > to compose things
> > 
> > ImageCmap lets you define a colormap that includes an alpha channel, but
> > it
> > doesn't implement any special operations.
> > 
> > > - The third is: given a colormap, how to interpolate into it (+ window
> > > level, window brightness).
> > 
> > Not quite sure what you mean, are you asking if you can have a fractional
> > value that lies between two points in the colormap? If the operations you
> > want
> > to implement can be expressed in terms of + and * via the standard rules
> > of
> > interpolation, this one should be easy :-): just use Interpolations.jl to
> > create the colormap vector, e.g.,
> > 
> > icmap = interpolate(cmap, BSpline(Linear()))
> > 
> > and pass that to ImageCmap. We can easily generalize it to allow an
> > AbstractVector.
> > 
> > Note that currently img[i,j] for an ImageCmap returns the index, not the
> > color
> > value. I don't remember why I did it that way, but I think that's
> > something
> > that will have to change. Just need to think of the right migration
> > strategy.
> > 
> > Best,
> > --Tim
> > 
> > > Cheers,
> > > 
> > > Tobias
> > > 
> > > Am Donnerstag, 12. November 2015 21:54:18 UTC+1 schrieb Tim Holy:
> > > > On Thursday, November 12, 2015 12:15:46 PM Tobias Knopp wrote:
> > > > > Thanks Tim, I tried to look into the code of Overlay but it wasn't
> > 
> > to
> > 
> > > > clear
> > > > 
> > > > > to me. In particular I am missing where the RGB(A) data is combined.
> > > > 
> > > > Here:
> > https://github.com/timholy/Images.jl/blob/cac28026250814f6ae6594dd26e92707
> > 
> > > > 6177db60/src/overlays.jl#L60-L67>
> > > > 
> > > > > Is it
> > > > > really as simple as adding the individual RGB values and preventing
> > > > > overflow by clamp?
> > > > 
> > > > Yes.
> > > > 
> > > > > Or is there some infrastructure for color mixing in Colors.jl
> > > > 
> > > > There is, but it's considerably slower. See the ColorVectorSpace.jl
> > 
> > README
> > 
> > > > for
> > > > discussion.
> > > > 
> > > > > I further looked for functions for gray value mapping
> > > > 
> > > > (Contrast/Brightness)
> > > > 
> > > > > in Colors.jl but could not find anything. This is of course not

[julia-users] Re: Creating a stable version of Julia + Packages for a semester long course?

[Steven G. Johnson]

On Monday, November 16, 2015 at 10:14:52 AM UTC-5, Seth wrote:
>
> Avik,
>
> what happens with precompilation if .julia isn't writable? (Too chicken to 
> try it out here.)
>

You can use a different cache path.  However, even ignoring 
precompilations, a lot of packages need to write to their  "deps" directory 
during Pkg.build.

Re: [julia-users] Re: Google releases TensorFlow as open source

[Ravish Mishra]

Thanks Stefan. Just starting on Julia. Hope to start contributing soon.

~ Ravish

On Mon, Nov 16, 2015 at 11:16 PM, Stefan Karpinski 
wrote:

> JuliaML is a collection of repos, not people. If you create a package that
> ends up in JuliaML or make significant contributions to one of them, then
> the owner of some JuliaML package may give you commit access to that
> package.
>
> On Mon, Nov 16, 2015 at 4:24 AM, Ravish Mishra 
> wrote:
>
>> Hi Viral,
>>
>> I want to be a part of JuliaML.
>>
>> ~ Ravish
>>
>>
>> On Wednesday, November 11, 2015 at 4:48:07 PM UTC+5:30, Viral Shah wrote:
>>>
>>> I think TensorFlow.jl is a great idea. Also their distributed
>>> computation framework is also the kind that we want to have in Julia.
>>>
>>> I have created JuliaML. Send me email if you want to be part of it, and
>>> I will make you an owner. Perhaps we can even move some of the JuliaStats
>>> ML projects to JuliaML.
>>>
>>> -viral
>>>
>>> On Wednesday, November 11, 2015 at 11:27:21 AM UTC+5:30, Valentin
>>> Churavy wrote:

 It fits in the same niche that Mocha.jl and MXNet.jl are filling right
 now. MXNet is a ML library that shares many of the same design ideas of
 TensorFlow and has great Julia support https://github.com/dmlc/MXNet.jl


 On Wednesday, 11 November 2015 01:04:00 UTC+9, Randy Zwitch wrote:
>
> For me, the bigger question is how does TensorFlow fit in/fill in gaps
> in currently available Julia libraries? I'm not saying that someone who is
> sufficiently interested shouldn't wrap the library, but it'd be great to
> identify what major gaps remain in ML for Julia and figure out if
> TensorFlow is the right way to proceed.
>
> We're certainly nowhere near the R duplication problem yet, but
> certainly we're already repeating ourselves in many areas.
>
> On Monday, November 9, 2015 at 4:02:36 PM UTC-5, Phil Tomson wrote:
>>
>> Google has released it's deep learning library called TensorFlow as
>> open source code:
>>
>> https://github.com/tensorflow/tensorflow
>>
>> They include Python bindings, Any ideas about how easy/difficult it
>> would be to create Julia bindings?
>>
>> Phil
>>
>
>

[julia-users] Re: General question on indexing

[hustf]

Seth,
thanks for that, and yes we can move over there, but I need some time to 
study it before I contribute in the context. I've been looking at 
GraphLayout - spring.jl, and thinking along the same lines about speed. I'm 
delaying a commit because the package has a coherent coding style but which 
I find hard to follow - and an interface problem with Compose. Also, I 
don't realistically expect to make a real improvement except in my own 
skills.

>From the documentation and small tests, I did not expect any improvements 
using @simd as long as there are "if"-sentences in the middle of it. So 
perhaps this format of the adjacency matrix may help? This for both 
directed and non-directed graphs. The type definition AdjMatrix just a 
small adaption of Maq7 above. Each node could be processed from one row and 
a submatrix pointing to the positions of just the other nodes.

# using 'current node -neighbour node ' notation:
4x4 Array{Int64,2}:
  0  21  31  41
 12   0  32  42
 13  23   0  43
 14  24  34   0

julia> ama =AdjMatrix(am)
3x4 AdjMatrix:
 12  21  31  41
 13  23  32  42
 14  24  34  43

Re: [julia-users] Re: Google releases TensorFlow as open source

[Phil Tomson]

On Monday, November 16, 2015 at 11:46:14 AM UTC-8, George Coles wrote:
>
> Does MXNet provide features that are analogous with Theano? I would rather 
> do machine learning in one language, than a mix of python + c + a DSL like 
> Theano. 


MXNet.jl is a wrapper around libmxnet so there is c (C++) in the 
background. 

MXNet.jl would be analogous to Theano in some ways. It also seems similar 
to TensorFlow. 

It is always cool to be able to quickly wrap native libraries, but Julia 
> would really gain momentum if it could obviate Theano et al (as cool as the 
> python ecosystem is, it is all quite ungainly.)
>

You should give some of the MXNet.jl examples a try.

Phil 

[julia-users] Re: Masters/PhD programs to continue work on Julia

[Alex Mellnik]

Agreed.  I would also note that Julia might be used in a wide range of 
graduate fields, from linguistics to economics to computational physics to 
math, so if you're only interested in working in Julia you needn't limit 
yourself to Comp Sci/Eng unless that's the field you wanted to go into 
anyways.  

On Friday, November 13, 2015 at 9:17:53 AM UTC-8, Steven G. Johnson wrote:
>
> On Thursday, November 12, 2015 at 11:43:38 AM UTC-5, Rohit Thankachan 
> wrote:
>>
>> I am planning on applying to masters and/or PhD programs that start in 
>> Fall 2016. I am a JSoC 2015 student and have been doing work related to 3D 
>> visualizations from Julia - ThreeJS.jl 
>>  and Compose3D.jl 
>> . I am hoping to continue 
>> working on these projects along with newer projects in Julia as part of a 
>> master's or PhD program. Does anyone know of any programs that would allow 
>> me to do this? Pointers would be appreciated too. :)
>>
>
> As a general rule, you should apply to PhD engineering programs with the 
> expectation that you can choose your general area of research (e.g. 
> visualization), but your specific projects will be chosen by your advisor 
> in the beginning.   You can generally spend a certain fraction of your time 
> on personal projects, and as you become a more senior graduate student you 
> often have more input in steering your research direction, but your advisor 
> will typically hire you with projects already in mind for you to work on.
>
> (Also, as a general tip for admissions: by all means, express an interest 
> in specific areas of research, but in your application statement you should 
> appear flexible in what specific problems you want to work on.   If you 
> appear too determined to work on a specific problem, no one will want to 
> admit you except in the unlikely event that your problem exactly coincides 
> with a project already in their research program/grants.)
>

[julia-users] Re: Creating a stable version of Julia + Packages for a semester long course?

[Sheehan Olver]

I guess if the students are limited to the pre-assigned packages all 
running on the same machine, then the cache can be shared and the Pkg.build 
done before hand?  I.e., whoever owns the shared .julia file calls ''using 
Gadfly,PyPlot,IJulia" and then the build and cache happens.

Re: Cloud based solutions.  I think being in Australia its risky relying on 
a trans-pacific internet connection.  I'm only concerned with the setup on 
the departmental lab computers, which are actually multiple log ins to a 
single machine, so I don't have to worry about students installing on their 
own machines.



On Tuesday, November 17, 2015 at 4:19:13 AM UTC+11, Steven G. Johnson wrote:
>
>
>
> On Monday, November 16, 2015 at 10:14:52 AM UTC-5, Seth wrote:
>>
>> Avik,
>>
>> what happens with precompilation if .julia isn't writable? (Too chicken 
>> to try it out here.)
>>
>
> You can use a different cache path.  However, even ignoring 
> precompilations, a lot of packages need to write to their  "deps" directory 
> during Pkg.build.
>

[julia-users] Re: Creating a stable version of Julia + Packages for a semester long course?

[Alireza Nejati]

John Lambert: Unless you can run julia in sagemath cloud, I fail to see how 
it is relevant to the discussion at hand. Sheehan clearly wants to run 
julia, not sage.

[julia-users] Re: sparse rank, sparse nullspace, sparse linear algebra over rationals?

[Seth]

well, nullspace appears to be defined for dense arrays only - and rank is 
defined for abstract arrays but will fail in svdvals!() when passed a 
sparse array (this is arguably a bug).

On Monday, November 16, 2015 at 1:12:04 PM UTC-8, Laurent Bartholdi wrote:
>
> Hello world,
> I'm new at julia, and trying it out as a replacement for matlab and other 
> computer algebra systems. I'm a bit stuck with sparse matrices: I have 
> largeish matrices (10^6 x 10^6) that are very sparse, and
> want to know their rank and nullspace. (the matrices decompose by blocks, 
> so the nullspace should be expressible as a sparse matrix).
>
> I tried with the latest (github) julia 0.5:
>
> julia> nullspace(sparse([1],[1],[1]))
> ERROR: MethodError: `nullspace` has no method matching 
> nullspace(::SparseMatrixCSC{Int64,Int64})
>
> julia> nullspace(full(sparse([1],[1],[1])))
> 1x0 Array{Float64,2} # I'm a bit unhappy here, I was hoping to get a 
> rational answer.
>
> julia> nullspace([1//1])
> 1x0 Array{Float32,2} # yikes! I'm down to 32 bits floats now.
>
> julia> rank(sparse([1],[1],[1.0]))
> ERROR: MethodError: `svdvals!` has no method matching 
> svdvals!(::SparseMatrixCSC{Float64,Int64})
>
> julia> rank([1.0])
> ERROR: MethodError: `rank` has no method matching rank(::Array{Float64,1})
>
> julia> rank([1 0;0 1])
> 2 # finally something that works...
>
> Many thanks in advance! Laurent
>
>

[julia-users] Re: push! performance

[Seth]

Yup, I'm convinced. Though if we ever get 
https://github.com/JuliaLang/julia/issues/9147 implemented perhaps it'll be 
faster in the case of zeros too :)

On Monday, November 16, 2015 at 9:42:40 AM UTC-8, Tomas Lycken wrote:
>
> Yeah, that makes it even more clear that they’re very similar:
>
> julia> gc(); @time for i=1:1_000; sizehint!(Vector{Int}(0), 1_000); end
>   0.000186 seconds (2.00 k allocations: 7.721 MB)
>
> julia> gc(); @time for i=1:1_000; Vector{Int}(1_000); end
>   0.000181 seconds (1000 allocations: 7.706 MB)
>
> (broken into smaller chunks, because the timings started to get awfully 
> close to the resolution of the clock, and the gc percentage count got 
> really borked…)
>
> // T
>
> On Monday, November 16, 2015 at 6:26:04 PM UTC+1, Steven G. Johnson wrote:
>
>
>
> On Monday, November 16, 2015 at 11:24:20 AM UTC-5, Tomas Lycken wrote:
>
> Making sure that precompilation and gc don’t factor into the result, I get 
> quite different timings:
>
> julia> gc(); @time sizehint!(a, 10_000_000);
>   0.001493 seconds (46 allocations: 76.296 MB, 304.61% gc time)
>
> julia> gc(); @time b = zeros(Int, 10_000_000);
>   0.021997 seconds (38 allocations: 76.296 MB, 0.70% gc time)
>
>
> Compare to Array(Int,  10_000_000), which allocates an uninitialized array.
>
> ​
>
> ​
>
> ...

[julia-users] Re: build_executable.jl fails in OSX yosemite

[Daniel Høegh]

Thank you for the fix Tom. I have made a PR to master to fix this see, 
https://github.com/JuliaLang/julia/pull/14018. I have started a new repository 
for build_executable to fix it and keep it up to date. It would be nice if you 
could verify that the branch in 
https://github.com/dhoegh/BuildExecutable.jl/pull/2 build successfully on OSX. 
I have a plan to merge it soon, everything seems to work on Travis with your 
fix of build_sysimg.jl

[julia-users] sparse rank, sparse nullspace, sparse linear algebra over rationals?

[Laurent Bartholdi]

Hello world,
I'm new at julia, and trying it out as a replacement for matlab and other 
computer algebra systems. I'm a bit stuck with sparse matrices: I have 
largeish matrices (10^6 x 10^6) that are very sparse, and
want to know their rank and nullspace. (the matrices decompose by blocks, 
so the nullspace should be expressible as a sparse matrix).

I tried with the latest (github) julia 0.5:

julia> nullspace(sparse([1],[1],[1]))
ERROR: MethodError: `nullspace` has no method matching 
nullspace(::SparseMatrixCSC{Int64,Int64})

julia> nullspace(full(sparse([1],[1],[1])))
1x0 Array{Float64,2} # I'm a bit unhappy here, I was hoping to get a 
rational answer.

julia> nullspace([1//1])
1x0 Array{Float32,2} # yikes! I'm down to 32 bits floats now.

julia> rank(sparse([1],[1],[1.0]))
ERROR: MethodError: `svdvals!` has no method matching 
svdvals!(::SparseMatrixCSC{Float64,Int64})

julia> rank([1.0])
ERROR: MethodError: `rank` has no method matching rank(::Array{Float64,1})

julia> rank([1 0;0 1])
2 # finally something that works...

Many thanks in advance! Laurent

Re: [julia-users] Re: Google releases TensorFlow as open source

[George Coles]

Does MXNet provide features that are analogous with Theano? I would rather do 
machine learning in one language, than a mix of python + c + a DSL like Theano. 

It is always cool to be able to quickly wrap native libraries, but Julia would 
really gain momentum if it could obviate Theano et al (as cool as the python 
ecosystem is, it is all quite ungainly.)

[julia-users] Re: General question on indexing

[hustf]

Seth, I appreciate that. As a novice programmer I appreciate to have 
exchanges with you guys who are making this. I'm using commercial software 
every day that hasn't made progress since the nineties. 

I worked through something like fifteen types for the adjacency matrix 
(and, by the way, tuples for storage worked out, but was slower). I suppose 
Julia is fast so that we don't actually need to optimize like this all the 
time, but it's an excellent lesson. 

In the example I gave above you may have noted some confusion about 
rows-columns order. Those algorithms which were adapted from text books or 
other languages seem to walk along rows. In Julia, you walk faster 
downhill. And with @simd. 

Re: [julia-users] Re: Google releases TensorFlow as open source

[George Coles]

Does MXNet provide features that are analogous with Theano? I would rather do 
machine learning in one language, than a mix of python + c + a DSL like Theano. 

It is always cool to be able to quickly wrap native libraries, but Julia would 
really gain momentum if it could obviate Theano et al (as cool as the python 
ecosystem is, it is all quite ungainly.)

[julia-users] Re: General question on indexing

[Seth]

I wish I walked faster with @simd - for some reason I'm getting the same 
performance as described 
here: https://groups.google.com/d/msg/julia-users/DycY6jwDcWs/yYXHsWF9AwAJ

One of the key objectives for LightGraphs is to be as fast as possible, so 
while Julia itself is pretty darn good, we want to eke every possible 
performance gain out of it so that we can scale to very large graphs. This 
also includes memory allocation, so fast and small. We're getting to the 
point where improving one results in bad performance in the other. That's 
both a good sign (that we've optimized away the low-hanging fruit) and a 
bad sign (that we may be hitting the limits of what we can do). I'm sure 
we've missed some stuff, though.

Thanks for the discussion. Feel free to join us over 
at https://github.com/JuliaGraphs/LightGraphs.jl/issues if this sort of 
stuff interests you :)

Seth.



On Monday, November 16, 2015 at 11:28:15 AM UTC-8, hustf wrote:
>
> Seth, I appreciate that. As a novice programmer I appreciate to have 
> exchanges with you guys who are making this. I'm using commercial software 
> every day that hasn't made progress since the nineties. 
>
> I worked through something like fifteen types for the adjacency matrix 
> (and, by the way, tuples for storage worked out, but was slower). I suppose 
> Julia is fast so that we don't actually need to optimize like this all the 
> time, but it's an excellent lesson. 
>
> In the example I gave above you may have noted some confusion about 
> rows-columns order. Those algorithms which were adapted from text books or 
> other languages seem to walk along rows. In Julia, you walk faster 
> downhill. And with @simd. 
>

[julia-users] Brew: No such keg for ImageMagick

[Cedric St-Jean]

My update to OSX El Capitan brought along a bunch of brew updates which 
broke my ImageMagick install. I followed someone's advice to 

rm -rf ~/.julia/v0.4/Homebrew/deps/usr/Cellar/imagemagick

but maybe that was bad advice :) Now I can't build it at all:

> Pkg.build("ImageMagick")

INFO: Building Homebrew
>From https://github.com/Homebrew/homebrew
   e7a3153..160f3a3  master -> origin/master

HEAD is now at 160f3a3 audit: respect umask in formula file mode check
HEAD is now at 4a13095 Merge pull request #85 from staticfloat/staging

INFO: Building ImageMagick
Error: No such keg: 
/Users/cedric/.julia/v0.4/Homebrew/deps/usr/Cellar/imagemagick
=[ ERROR: ImageMagick ]=

LoadError: failed process: 
Process(`/Users/cedric/.julia/v0.4/Homebrew/deps/usr/bin/brew unlink --quiet 
imagemagick`, ProcessExited(1)) [1]
while loading /Users/cedric/.julia/v0.4/ImageMagick/deps/build.jl, in 
expression starting on line 92


How should I "bring it back"? I've tried `Pkg.rm("ImageMagick")` then 
`Pkg.add("ImageMagick")`, but that just fails. I could brew install it at 
the command line, but apparently that's a bad idea for ImageMagick 
specifically.

Cédric

[julia-users] Re: Creating a stable version of Julia + Packages for a semester long course?

[cdm]

more on courses over SMC:

https://github.com/sagemathinc/smc/wiki/Teaching

Re: [julia-users] Julia users at SC'15?

[Eric Forgy]

Way to plan ahead guys :D

On Tuesday, November 17, 2015 at 1:09:30 AM UTC+8, Erik Schnetter wrote:
>
> Mark
>
> You beat me to sending such an email by a few minutes.
>
> Is there a particular Julian place where we could meet? Is there e.g. an 
> MIT booth at tonight's reception?
>
> -erik
>
> On Sun, Nov 15, 2015 at 8:42 AM, Mark Dewing  > wrote:
>
>> SC15 (Supercomputing) is happening in Austin this week.
>> Are there any Julia users/developers/fans here who would like to meet up?
>>
>> - Mark
>>
>
>
>
> -- 
> Erik Schnetter  
> http://www.perimeterinstitute.ca/personal/eschnetter/
>

[julia-users] Re: Creating a stable version of Julia + Packages for a semester long course?

[cdm]

the Sage Math Cloud is on:


   _
   _   _ _(_)_ |  A fresh approach to technical computing
  (_) | (_) (_)|  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.4.1 (2015-11-08 10:33 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org release
|__/   |  x86_64-linux-gnu



that capture is from a Julia session that is live on Sage Math Cloud now.

best,

cdm


On Monday, November 16, 2015 at 2:46:53 PM UTC-8, Alireza Nejati wrote:
>
> John Lambert: Unless you can run julia in sagemath cloud, I fail to see 
> how it is relevant to the discussion at hand. Sheehan clearly wants to run 
> julia, not sage.
>

Re: [julia-users] Colormaps composing images

[Tim Holy]

On Monday, November 16, 2015 02:25:43 AM Tobias Knopp wrote:
> After investigating this a little further I found that the key things I
> am/was missing are:
> - How to compose two RGBA values a and b. It turns out that this is the
> "over" operation given by
> 
>   c = alpha*b + (1-alpha)*a
> 
> if alpha is the alpha channel of b and a has alpha=1. Do I have missed it
> or is the over operation somewhere implemented in a package?

I'm not aware of a package that implements alpha-compositing. Perhaps it could 
be added to ColorVectorSpace (which might then need a new name), or perhaps 
better it could be its own package. If you're interested in this, I'd 
recommend starting by creating a type that makes it clear whether you're 
working with pre-multiplied alpha or not, see 
https://en.wikipedia.org/wiki/Alpha_compositing#Description

> - The second thing is colormaps that include an alpha channel. This allows
> to overlay data in a very flexible way. Otherwise its not really clear how
> to compose things

ImageCmap lets you define a colormap that includes an alpha channel, but it 
doesn't implement any special operations.

> 
> - The third is: given a colormap, how to interpolate into it (+ window
> level, window brightness).

Not quite sure what you mean, are you asking if you can have a fractional 
value that lies between two points in the colormap? If the operations you want 
to implement can be expressed in terms of + and * via the standard rules of 
interpolation, this one should be easy :-): just use Interpolations.jl to 
create the colormap vector, e.g.,

icmap = interpolate(cmap, BSpline(Linear()))

and pass that to ImageCmap. We can easily generalize it to allow an 
AbstractVector.

Note that currently img[i,j] for an ImageCmap returns the index, not the color 
value. I don't remember why I did it that way, but I think that's something 
that will have to change. Just need to think of the right migration strategy.

Best,
--Tim

> 
> Cheers,
> 
> Tobias
> 
> Am Donnerstag, 12. November 2015 21:54:18 UTC+1 schrieb Tim Holy:
> > On Thursday, November 12, 2015 12:15:46 PM Tobias Knopp wrote:
> > > Thanks Tim, I tried to look into the code of Overlay but it wasn't to
> > 
> > clear
> > 
> > > to me. In particular I am missing where the RGB(A) data is combined.
> > 
> > Here:
> > 
> > https://github.com/timholy/Images.jl/blob/cac28026250814f6ae6594dd26e92707
> > 6177db60/src/overlays.jl#L60-L67> 
> > > Is it
> > > really as simple as adding the individual RGB values and preventing
> > > overflow by clamp?
> > 
> > Yes.
> > 
> > > Or is there some infrastructure for color mixing in Colors.jl
> > 
> > There is, but it's considerably slower. See the ColorVectorSpace.jl README
> > for
> > discussion.
> > 
> > > I further looked for functions for gray value mapping
> > 
> > (Contrast/Brightness)
> > 
> > > in Colors.jl but could not find anything. This is of course not
> > 
> > complicated
> > 
> > > to code but I don't want to miss an existing solution.
> > 
> > The whole "MapInfo" structure is a very flexible and powerful. Search for
> > it on
> > this page:
> > http://timholy.github.io/Images.jl/function_reference.html
> > From my standpoint, the best feature is that it's "lazy": you specify the
> > transformation you want, but don't execute it until you need it. For
> > people
> > like me who routinely browse 1TB images but probably look at <1% of the
> > raw
> > data in any given dataset (and who also don't have 1TB worth of RAM...),
> > this
> > is quite an advantage.
> > 
> > In my own work, I pretty routinely design custom MapInfo types/map
> > functions
> > for visualization purposes. For example, I can color individual blobs in
> > each
> > frame of a movie with something along the lines of
> > 
> > immutable ColorizeBlobs
> > 
> > blobpixels::Vector   # of length nblobs
> > blobcolors::Vector{RGB{U8}}  # color assigned to each blob
> > 
> > end
> > 
> > and passing that to ImageView using the "scalei" keyword argument (a
> > legacy of
> > the days when this was called ScaleInfo rather than MapInfo). It's a nice
> > way
> > of getting custom visualization while leaving all the stupid zoom &
> > navigation
> > functionality up to ImageView.
> > 
> > --Tim
> > 
> > > Tobi
> > > 
> > > Am Donnerstag, 12. November 2015 16:42:00 UTC+1 schrieb Tim Holy:
> > > > Probably the easiest thing would be to just extend the code in Images
> > 
> > and
> > 
> > > > submit a PR (the code is not very complicated).
> > > > 
> > > > However, you can do very fancy things with MapInfo objects. This is
> > > > untested,
> > > > but it should be close:
> > > > 
> > > > immutable TwoColormap <: MapInfo
> > > > 
> > > > colormap1
> > > > colormap2
> > > > 
> > > > end
> > > > 
> > > > function map!(dest, mapi::TwoColormap,
> > > > src::Tuple{AbstractArray,AbstractArray})
> > > > 
> > > > img1, img2 = src
> > > > for I in eachindex(dest)
> > > > 
> > > > dest[I] = 

[julia-users] how to make my own 3d array type

[Evan]

For a 2d array the following works:

type mytype{T}
var :: AbstractMatrix{T}
end

julia> t = mytype(zeros(4, 3))
mytype{Float64}(4x3 Array{Float64,2}:
 0.0  0.0  0.0
 0.0  0.0  0.0
 0.0  0.0  0.0
 0.0  0.0  0.0)



But how do I extend this to a 3d array?

julia> t = mytype(zeros(2, 4, 3))
ERROR: MethodError: `convert` has no method matching convert(::Type{mytype{T
}}, ::Array{Float64,3})
This may have arisen from a call to the constructor mytype{T}(...),
since type constructors fall back to convert methods.
Closest candidates are:
  call{T}(::Type{T}, ::Any)
  convert{T}(::Type{T}, ::T)
  mytype{T}(::AbstractArray{T,2})
 in call at essentials.jl:56

[julia-users] Re: how to make my own 3d array type

[Patrick Kofod Mogensen]

Well, try running

 isa(zeros(2,4,3), AbstractMatrix)

The problem here is that a(n) (Abstract)Matrix is 2-dimensional array.

On Monday, November 16, 2015 at 12:54:15 PM UTC+1, Evan wrote:
>
> For a 2d array the following works:
>
> type mytype{T}
> var :: AbstractMatrix{T}
> end
>
> julia> t = mytype(zeros(4, 3))
> mytype{Float64}(4x3 Array{Float64,2}:
>  0.0  0.0  0.0
>  0.0  0.0  0.0
>  0.0  0.0  0.0
>  0.0  0.0  0.0)
>
>
>
> But how do I extend this to a 3d array?
>
> julia> t = mytype(zeros(2, 4, 3))
> ERROR: MethodError: `convert` has no method matching convert(::Type{mytype
> {T}}, ::Array{Float64,3})
> This may have arisen from a call to the constructor mytype{T}(...),
> since type constructors fall back to convert methods.
> Closest candidates are:
>   call{T}(::Type{T}, ::Any)
>   convert{T}(::Type{T}, ::T)
>   mytype{T}(::AbstractArray{T,2})
>  in call at essentials.jl:56
>
>
>
>
>

[julia-users] 2-by-2 dense linear system performance

[Pavel]

Greetings,

I have a 2x2 linear algebraic system to be solved at each step of an 
iterative process. Julia code profiling shows that

linalg/dense.jl; \; line: 448

takes the most run time (from my A\b call), probably calling LU 
decomposition from lapack. Is there any way to speed up the calculation 
specifically due to such a small system size? I could of course use exact 
solution (also saving memory allocation by not forming arrays) but 
numerical issues matter for a few ways to write it (potential instability). 
There is no a priori knowledge about specifics of the matrix (could 
sometimes be close to singular). Any suggestions for performance 
enhancements?

Thanks,
Pavel

[julia-users] Re: indexing with non Integer Reals is deprecated

[Eric Forgy]

Hi Peter,

I've learned that Julia is flexible enough to make it do just about 
whatever you want it to do. If you want Julia to handle non-integer 
indices, you can very easily create a type MatlabArray (or something) and 
have its indices be non-integer and the matrix performance would be on par 
in terms of performance as native arrays.

The fact that base Julia doesn't allow non-integer indices does not really 
limit you or anyone in any way because you can simply create your own type 
that does what you want it to do.

On Tuesday, November 17, 2015 at 1:22:18 AM UTC+8, Steven G. Johnson wrote:
>
>
>
> On Monday, November 16, 2015 at 11:54:39 AM UTC-5, Peter Kovesi wrote:
>>
>> Following the principle of 'minimum surprise': If I have an integer just 
>> that happens to be represented via a floating point type I would still like 
>> to be able to use it as an integer.
>>
>
> Can you give an example of where you would like to use a floating-point 
> index that does not involve doing integer division with / (where you should 
> probably use ÷ instead)? 
>

[julia-users] Re: Brew: No such keg for ImageMagick

[Cedric St-Jean]

Solved it

cd ~/.julia/v0.4/Homebrew/deps/usr/bin
./brew install imagemagick

This brought the keg back, then I could Pkg.build("ImageMagick")

On Monday, November 16, 2015 at 3:22:18 PM UTC-5, Cedric St-Jean wrote:
>
> My update to OSX El Capitan brought along a bunch of brew updates which 
> broke my ImageMagick install. I followed someone's advice to 
>
> rm -rf ~/.julia/v0.4/Homebrew/deps/usr/Cellar/imagemagick
>
> but maybe that was bad advice :) Now I can't build it at all:
>
> > Pkg.build("ImageMagick")
>
> INFO: Building Homebrew
> From https://github.com/Homebrew/homebrew
>e7a3153..160f3a3  master -> origin/master
>
> HEAD is now at 160f3a3 audit: respect umask in formula file mode check
> HEAD is now at 4a13095 Merge pull request #85 from staticfloat/staging
>
> INFO: Building ImageMagick
> Error: No such keg: 
> /Users/cedric/.julia/v0.4/Homebrew/deps/usr/Cellar/imagemagick
> =[ ERROR: ImageMagick 
> ]=
>
> LoadError: failed process: 
> Process(`/Users/cedric/.julia/v0.4/Homebrew/deps/usr/bin/brew unlink --quiet 
> imagemagick`, ProcessExited(1)) [1]
> while loading /Users/cedric/.julia/v0.4/ImageMagick/deps/build.jl, in 
> expression starting on line 92
>
>
> How should I "bring it back"? I've tried `Pkg.rm("ImageMagick")` then 
> `Pkg.add("ImageMagick")`, but that just fails. I could brew install it at 
> the command line, but apparently that's a bad idea for ImageMagick 
> specifically.
>
> Cédric
>

[julia-users] Re: indexing with non Integer Reals is deprecated

[Eric Forgy]

Here is a quick example I threw together (with some clues from Tim here):

https://github.com/EricForgy/RoundingIndexArrays.jl

Indexing only works for 1-d at the moment, but hopefully it helps get the 
point.

On Tuesday, November 17, 2015 at 8:51:34 AM UTC+8, Eric Forgy wrote:
>
> Hi Peter,
>
> I've learned that Julia is flexible enough to make it do just about 
> whatever you want it to do. If you want Julia to handle non-integer 
> indices, you can very easily create a type MatlabArray (or something) and 
> have its indices be non-integer and the matrix performance would be on par 
> in terms of performance as native arrays.
>
> The fact that base Julia doesn't allow non-integer indices does not really 
> limit you or anyone in any way because you can simply create your own type 
> that does what you want it to do.
>
> On Tuesday, November 17, 2015 at 1:22:18 AM UTC+8, Steven G. Johnson wrote:
>>
>>
>>
>> On Monday, November 16, 2015 at 11:54:39 AM UTC-5, Peter Kovesi wrote:
>>>
>>> Following the principle of 'minimum surprise': If I have an integer just 
>>> that happens to be represented via a floating point type I would still like 
>>> to be able to use it as an integer.
>>>
>>
>> Can you give an example of where you would like to use a floating-point 
>> index that does not involve doing integer division with / (where you should 
>> probably use ÷ instead)? 
>>
>

[julia-users] Re: optimization of sorted vector / insert and dedup?

[Seth]

I managed to squeeze a bit more performance out of it:

_insert_and_dedup!(v::Vector{Int}, x::Int) = isempty(splice!(v, searchsorted
(v,x), x))

If there's a better way of doing it, please let me know.



On Monday, November 16, 2015 at 3:37:34 PM UTC-8, Seth wrote:
>
> I found this stackoverflow question: 
> http://stackoverflow.com/questions/25678112/insert-item-into-a-sorted-list-with-julia-with-and-without-duplicates
>
> with a great solution by Stefan.
>
> Moving from [x] to x (see his followup) reduced the time to do this 
> insert/dedup by ~30%.
>
> Is there a more efficient way to do this? Here's the code I have right now:
>
> function _insert_and_dedup!(v::Vector{Int}, x::Int)
> oldv = length(v)
> splice!(v, searchsorted(v,x), x)
> return (length(v) == oldv+1)
> end
>
> I'm returning true if the value was added; false if it was a duplicate 
> (needed for some other function).
>
>

[julia-users] Re: Masters/PhD programs to continue work on Julia

[Rohit Thankachan]

Hi Andre,

On Friday, 13 November 2015 20:06:58 UTC+5:30, André Lage wrote:
>
> hi Rohit,
> Cc Nikos Parlavantzas
>
> There is a master position open at the Myriads Team [1] (in France, but 
> classes are in English). It's a joint-advisorship: IRISA-INRIA labs 
> (France) and Federal University of Alagoas (Brazil).
>
 
Thanks for letting me know about the position. CloudArray sounds very 
exciting! However the admissions page states that it requires applicants to 
have completed one year of master studies, which is a requirement I 
wouldn't satisfy.

Hi Steven, 
On Friday, 13 November 2015 22:47:53 UTC+5:30, Steven G. Johnson wrote:
>
>
> As a general rule, you should apply to PhD engineering programs with the 
> expectation that you can choose your general area of research (e.g. 
> visualization), but your specific projects will be chosen by your advisor 
> in the beginning.   You can generally spend a certain fraction of your time 
> on personal projects, and as you become a more senior graduate student you 
> often have more input in steering your research direction, but your advisor 
> will typically hire you with projects already in mind for you to work on.
>

> (Also, as a general tip for admissions: by all means, express an interest 
> in specific areas of research, but in your application statement you should 
> appear flexible in what specific problems you want to work on.   If you 
> appear too determined to work on a specific problem, no one will want to 
> admit you except in the unlikely event that your problem exactly coincides 
> with a project already in their research program/grants.)
>

Thanks for some great advice!
 
Hi Alex,
On Tuesday, 17 November 2015 04:18:33 UTC+5:30, Alex Mellnik wrote:
>
> Agreed.  I would also note that Julia might be used in a wide range of 
> graduate fields, from linguistics to economics to computational physics to 
> math, so if you're only interested in working in Julia you needn't limit 
> yourself to Comp Sci/Eng unless that's the field you wanted to go into 
> anyways.  
>

Comp Sci/Eng is the field I want to go into. :) 

[julia-users] Re: Creating a stable version of Julia + Packages for a semester long course?

[Sisyphuss]

Run a JuliaBox server?

On Monday, November 16, 2015 at 9:05:38 AM UTC+1, Sheehan Olver wrote:
>
> Another requirement is that the packages are shared across users, to save 
> disk space.  Gadfly + PyPlot + IJulia (with Conda.jl version of Jupyter) 
> takes over 750MB.   Does .julia need to be writable?  If not, I guess both 
> options are still possible.
>
> On Monday, November 16, 2015 at 2:05:45 PM UTC+11, Sheehan Olver wrote:
>>
>>
>> I'm trying to figure out the "best" way to create a stable version of 
>> Julia + Gadfly + PyPlot + IJulia (+ other packages?) for a semester long 
>> course.  I don't want to have the students run Pkg.add(...)/Pkg.update(), 
>> as packages have a tendency to occasionally break on updates, and it's a 
>> headache dealing with this during the lecture.
>>
>> Two possible solutions I can think of of are:
>>
>> 1)  Prebake a .julia folder that contains all the necessary resources, 
>> with a script to reset in case the students break it with Pkg.update().
>> 2)  Use system image
>>
>> http://docs.julialang.org/en/release-0.4/devdocs/sysimg/
>>
>> that includes all the necessary packages.   It's not really clear how to 
>> do this from the documentation, though.   I'm also not sure how that would 
>> interact with Pkg.update() though, so probably instructions to delete 
>> .julia would also need to be given.
>>
>>
>> Any other options I'm missing?  If 2 is recommended, any tutorial how to 
>> do this?
>>
>

[julia-users] Re: push! performance

[Tomas Lycken]

sizehint! preallocates for you, with comparable cost to calling e.g. zeroes, 
but lets you treat the array semantically the same way as a 
non-preallocated one (but without the cost for reallocation). Hopefully, 
these comments highlight the differences between the various appraches:

N = 10_000
A = Array(Float64,0) 
sizehint!(A, 10_000) # this preallocates memory for 10k elements
B = Array(Float64,0)
C = zeros(10_000) # this also preallocates memory for 10k elements

# now, A and C are pre-allocated, while B is not# however, A and B are 
semantically equivalent (0-length) vectors,
# while C is already of length 10 000:
println(length(A)) # 0
println(length(B)) # 0
println(length(C)) # 1

for i in 1:10_000
   push!(A, i) # no reallocation happens here, because we did it with sizehint!
   push!(B, i) # this will re-allocate B every now and then
   C[i] = i # can't use push! here, but must manually track index instead
end

I don't know what `dynamic` does in this context, and I can't find it in 
the docs, so can't help you there :)

// T

On Monday, November 16, 2015 at 2:07:13 AM UTC+1, Seth wrote:

What happens if you use sizehint!() with dynamic()?
>
> On Sunday, November 15, 2015 at 3:35:45 PM UTC-8, Steven G. Johnson wrote:
>>
>> function prealloc(n)
>> a = Array(Int, n)
>> for i = 1:n
>> a[i] = i
>> end
>> return a
>> end
>> function dynamic(n)
>> a = Int[]
>> for i = 1:n
>> push!(a, i)
>> end
>> return a
>> end
>> @time prealloc(10^7);
>> @time dynamic(10^7);
>>
>>
>> On my machine, the preallocated version is 2.5–3x faster.  A significant 
>> but not overwhelming margin.
>>
> ​

Re: [julia-users] Creating a stable version of Julia + Packages for a semester long course?

[Sheehan Olver]

You can do this locally?   I tried using JuliaBox in my classes last 
year, but it was a bit of disaster, as it was unreliable.




> On 16 Nov 2015, at 7:53 PM, Sisyphuss  wrote:
> 
> Run a JuliaBox server?
> 
> On Monday, November 16, 2015 at 9:05:38 AM UTC+1, Sheehan Olver wrote:
> Another requirement is that the packages are shared across users, to save 
> disk space.  Gadfly + PyPlot + IJulia (with Conda.jl version of Jupyter) 
> takes over 750MB.   Does .julia need to be writable?  If not, I guess both 
> options are still possible.
> 
> On Monday, November 16, 2015 at 2:05:45 PM UTC+11, Sheehan Olver wrote:
> 
> I'm trying to figure out the "best" way to create a stable version of Julia + 
> Gadfly + PyPlot + IJulia (+ other packages?) for a semester long course.  I 
> don't want to have the students run Pkg.add(...)/Pkg.update(), as packages 
> have a tendency to occasionally break on updates, and it's a headache dealing 
> with this during the lecture.
> 
> Two possible solutions I can think of of are:
> 
> 1)  Prebake a .julia folder that contains all the necessary resources, with a 
> script to reset in case the students break it with Pkg.update().
> 2)  Use system image
> 
> http://docs.julialang.org/en/release-0.4/devdocs/sysimg/ 
> 
> 
> that includes all the necessary packages.   It's not really clear how to do 
> this from the documentation, though.   I'm also not sure how that would 
> interact with Pkg.update() though, so probably instructions to delete .julia 
> would also need to be given.
> 
> 
> Any other options I'm missing?  If 2 is recommended, any tutorial how to do 
> this?

Re: [julia-users] Colormaps composing images

[Tobias Knopp]

After investigating this a little further I found that the key things I 
am/was missing are:
- How to compose two RGBA values a and b. It turns out that this is the 
"over" operation given by

  c = alpha*b + (1-alpha)*a

if alpha is the alpha channel of b and a has alpha=1. Do I have missed it 
or is the over operation somewhere implemented in a package?

- The second thing is colormaps that include an alpha channel. This allows 
to overlay data in a very flexible way. Otherwise its not really clear how 
to compose things

- The third is: given a colormap, how to interpolate into it (+ window 
level, window brightness).

Cheers,

Tobias
 



Am Donnerstag, 12. November 2015 21:54:18 UTC+1 schrieb Tim Holy:
>
> On Thursday, November 12, 2015 12:15:46 PM Tobias Knopp wrote: 
> > Thanks Tim, I tried to look into the code of Overlay but it wasn't to 
> clear 
> > to me. In particular I am missing where the RGB(A) data is combined. 
>
> Here: 
>
> https://github.com/timholy/Images.jl/blob/cac28026250814f6ae6594dd26e927076177db60/src/overlays.jl#L60-L67
>  
>
> > Is it 
> > really as simple as adding the individual RGB values and preventing 
> > overflow by clamp? 
>
> Yes. 
>
> > Or is there some infrastructure for color mixing in Colors.jl 
>
> There is, but it's considerably slower. See the ColorVectorSpace.jl README 
> for 
> discussion. 
>
> > I further looked for functions for gray value mapping 
> (Contrast/Brightness) 
> > in Colors.jl but could not find anything. This is of course not 
> complicated 
> > to code but I don't want to miss an existing solution. 
>
> The whole "MapInfo" structure is a very flexible and powerful. Search for 
> it on 
> this page: 
> http://timholy.github.io/Images.jl/function_reference.html 
> From my standpoint, the best feature is that it's "lazy": you specify the 
> transformation you want, but don't execute it until you need it. For 
> people 
> like me who routinely browse 1TB images but probably look at <1% of the 
> raw 
> data in any given dataset (and who also don't have 1TB worth of RAM...), 
> this 
> is quite an advantage. 
>
> In my own work, I pretty routinely design custom MapInfo types/map 
> functions 
> for visualization purposes. For example, I can color individual blobs in 
> each 
> frame of a movie with something along the lines of 
>
> immutable ColorizeBlobs 
> blobpixels::Vector   # of length nblobs 
> blobcolors::Vector{RGB{U8}}  # color assigned to each blob 
> end 
>
> and passing that to ImageView using the "scalei" keyword argument (a 
> legacy of 
> the days when this was called ScaleInfo rather than MapInfo). It's a nice 
> way 
> of getting custom visualization while leaving all the stupid zoom & 
> navigation 
> functionality up to ImageView. 
>
> --Tim 
>
> > 
> > Tobi 
> > 
> > Am Donnerstag, 12. November 2015 16:42:00 UTC+1 schrieb Tim Holy: 
> > > Probably the easiest thing would be to just extend the code in Images 
> and 
> > > submit a PR (the code is not very complicated). 
> > > 
> > > However, you can do very fancy things with MapInfo objects. This is 
> > > untested, 
> > > but it should be close: 
> > > 
> > > immutable TwoColormap <: MapInfo 
> > > 
> > > colormap1 
> > > colormap2 
> > > 
> > > end 
> > > 
> > > function map!(dest, mapi::TwoColormap, 
> > > src::Tuple{AbstractArray,AbstractArray}) 
> > > 
> > > img1, img2 = src 
> > > for I in eachindex(dest) 
> > > 
> > > dest[I] = clamp(RGBmapi.colormap1[img1[I]] + 
> > > 
> > > mapi.colormap2[img2[I]]) 
> > > 
> > > end 
> > > dest 
> > > 
> > > end 
> > > 
> > > --Tim 
> > > 
> > > On Thursday, November 12, 2015 07:23:07 AM Tobias Knopp wrote: 
> > > > Hi, 
> > > > 
> > > > I am using the OverlayImage type from the Images.jl package to 
> overlay 
> > > 
> > > two 
> > > 
> > > > different grayscale images (tomographic data). 
> > > > If I understand it correctly OverlayImage is restricted to colormaps 
> > > 
> > > that 
> > > 
> > > > go from black to a certain RGB value. Has anybody an idea how this 
> could 
> > > 
> > > be 
> > > 
> > > > extended to Colormaps provided by Colors.jl? 
> > > > 
> > > > So my need is: 
> > > > Input: two 3D datasets (FloatingPoint) + two Colormaps + WindowWidth 
> > > > WindowLevel for each 
> > > > Output: Combined 3D dataset as RGBA values. 
> > > > 
> > > > Thanks 
> > > > 
> > > > Tobias 
>
>

[julia-users] Re: sparse matrix bug

[ming . chau]

ok, i guess performance is then improved.

however, the conversion between sparse matrix line and sparse vecteor is 
not correct


julia> aa = sparse([10. 0 30.])
1x3 sparse matrix with 2 Float64 entries:
[1, 1]  =  10.0
[1, 3]  =  30.0


julia> full(aa[1,:])
3-element Array{Float64,1}:
 10.0
 30.0
  0.0

the zeros coefficient has been misplaced in the sparse vector.

bests,
M.

Le samedi 14 novembre 2015 08:05:18 UTC+1, Tony Kelman a écrit :
>
> These are intended changes, and not specific to sparse matrices. See the 
> NEWS.md file.

[julia-users] Re: sparse matrix bug

[Tony Kelman]

Ah, you're right, I missed that the values were wrong. Can you open an issue?

[julia-users] Re: sparse matrix bug

[ming . chau]

oops. I have just realized that I messed up with git checkout v0.***

I recompiled the good release (not the development branch).

the behavior described above does not happen in v0.4.1

I agree the "packing bug" in dev branch is normal in development stage.

I have read https://github.com/JuliaLang/julia/pull/13440

the direction taken by sparse arrays (matrix or vector) library is still 
questionable in my opinion. if a line or column slice of a sparse matrix is 
transformed into a sparse vector, with a different internal implementation, 
then you will prevent people from passing slices like a[k,:] to libraries. 
in my opinion, more generally, the implicit conversion of matrix slices 
like a[k,:] or a[:,k] into vectors will reduce drastically the main quality 
of Julia : "it's very easy to plug algorithms between them". we will have 
to write special case for slices every time.

anyway, just let me figure out how to open an issue :) i am noob with big 
open source projects.

bests,
M.

Le lundi 16 novembre 2015 11:12:22 UTC+1, Tony Kelman a écrit :
>
> Ah, you're right, I missed that the values were wrong. Can you open an 
> issue?

[julia-users] Re: Creating a stable version of Julia + Packages for a semester long course?

[Sheehan Olver]

Another requirement is that the packages are shared across users, to save 
disk space.  Gadfly + PyPlot + IJulia (with Conda.jl version of Jupyter) 
takes over 750MB.   Does .julia need to be writable?  If not, I guess both 
options are still possible.

On Monday, November 16, 2015 at 2:05:45 PM UTC+11, Sheehan Olver wrote:
>
>
> I'm trying to figure out the "best" way to create a stable version of 
> Julia + Gadfly + PyPlot + IJulia (+ other packages?) for a semester long 
> course.  I don't want to have the students run Pkg.add(...)/Pkg.update(), 
> as packages have a tendency to occasionally break on updates, and it's a 
> headache dealing with this during the lecture.
>
> Two possible solutions I can think of of are:
>
> 1)  Prebake a .julia folder that contains all the necessary resources, 
> with a script to reset in case the students break it with Pkg.update().
> 2)  Use system image
>
> http://docs.julialang.org/en/release-0.4/devdocs/sysimg/
>
> that includes all the necessary packages.   It's not really clear how to 
> do this from the documentation, though.   I'm also not sure how that would 
> interact with Pkg.update() though, so probably instructions to delete 
> .julia would also need to be given.
>
>
> Any other options I'm missing?  If 2 is recommended, any tutorial how to 
> do this?
>