[julia-users] Advice on programs with large data types

[Nathan Smith]

Hi 

I'm looking for some advice for some on writing methods on large 
(memory-wise) data types. Is there an overhead difference in the following 
two methods?

type State
   hugearray1::Array{Float64, 2}
   hugearray2::Array{Float64, 2}
   ... # Many more components
end

function style1!(s::State)
 complicated_function(s.hugearray1)
end

function style2!(arr::Array{Float64, 2})
 complicated_function(arr)
end

# difference in calls:

s = State(arr1, arr2, ...)
style1!(s)   # Pretty but is it slower?
style2!(s.hugearray1)# Ugly but is it faster?


Basically, is there an overhead in throwing the whole state datatype into a 
function even if i only manipulate one or two of the potentially large 
amount of components or should I be making functions in the second style 
where arguments are exactly of the type of the components being manipulated.

Thanks!
Nathan

ps: sorry for the double post, I hit enter before I finished writing..

[julia-users] Advice on programs with a huge state

[Nathan Smith]

Hi,

I'm looking for some advice on writing some simulation software. The 
simulation has an enormous state with many parameters and arrays that are 
updated iteratively and I was wondering if there is any difference in 
overhead in the following two styles of methods:

type State

 

[julia-users] Re: Making a New Repl Mode

[Nathan Smith]

Thanks for the heads up! I didn't even know that SJulia existed. 

[julia-users] Making a New Repl Mode

[Nathan Smith]

Hey all, 

I'm looking at making a new repl mode. I can't find much documentation on 
this process. Does anyone have any resources or suggestions to get me 
started before I dig through the source in REPL.jl, Cxx, RCall etc?

Best, 
Nathan

Re: [julia-users] Re: Multimedia I/O

[Nathan Smith]

What does "mime-type agnostic" mean in practice? 

[julia-users] Re: Multimedia I/O

[Nathan Smith]

ah, wonderful. I didn't realize that the "text/plain" mime was different 
than the default 1D string! Thanks for your help!

Nathan

[julia-users] Multimedia I/O

[Nathan Smith]

Hi all, 

I'm still a little confused about how to use julia's multimedia i/o. I'm 
writing a module to interact with Maxima CAS and I want to provide i/o for 
my maxima expression types. There are three different contexts that I'm 
interested in: 

(1) During string interpolation (among other things) I need to print out a 
1D version of the expression
(2) In the REPL I want to print out a 2D version of the expression
(3) In Jupyter I want to print out a pretty Tex/MathML version of the 
expression

Example:

if the Maxima expression is "sin(x)/x" i'd like to see

(1) sin(x)/x

(2) sin(x)
 ---
   x

(3)  <>

I've overloaded show(io, x::MaximaExpr) to get the appropriate behaviour 
for (1), but I don't understand how to get (2) and (3) to work...

Thanks for any help!

Nathan

[julia-users] Re: I can't believe this spped-up !

[Nathan Smith]

in Jupyer notebook, add processors with addprocs(N) 

On Thursday, 21 July 2016 12:59:02 UTC-4, Nathan Smith wrote:
>
> To be clear, you need to compare the final 'z' not the final 'A' to check 
> if your calculations are consistent. The matrix A does not change through 
> out this calculation, but the matrix z does.
> Also, there is no parallelism with the @parallel loop unless your start 
> julia with 'julia -np N' where N is the number of processes you'd like to 
> use.
>
> On Thursday, 21 July 2016 12:45:17 UTC-4, Ferran Mazzanti wrote:
>>
>> Hi Nathan,
>>
>> I posted the codes, so you can check if they do the same thing or not. 
>> These went to separate cells in Jupyter, nothing more and nothing less.
>> Not even a single line I didn't post. And yes I understand your line of 
>> reasoning, so that's why I got astonished also.
>> But I can see what is making this huge difference, and I'd like to know :)
>>
>> Best,
>>
>> Ferran.
>>
>> On Thursday, July 21, 2016 at 6:31:57 PM UTC+2, Nathan Smith wrote:
>>>
>>> Hey Ferran, 
>>>
>>> You should be suspicious when your apparent speed up surpasses the level 
>>> of parallelism available on your CPU. I looks like your codes don't 
>>> actually compute the same thing.
>>>
>>> I'm assuming you're trying to compute the matrix exponential of A 
>>> (A^10) by repeatedly multiplying A. In your parallel code, each 
>>> process gets a local copy of 'z' and
>>> uses that. This means each process is computing something like 
>>> (A^(10/# of procs)). Check out this 
>>> <http://docs.julialang.org/en/release-0.4/manual/parallel-computing/#parallel-map-and-loops>
>>>  section 
>>> of the documentation on parallel map and loops to see what I mean.
>>>
>>> That said, that doesn't explain your speed up completely, you should 
>>> also make sure that each part of your script is wrapped in a function and 
>>> that you 'warm-up' each function by running it once before comparing.
>>>
>>> Cheers, 
>>> Nathan
>>>
>>> On Thursday, 21 July 2016 12:00:47 UTC-4, Ferran Mazzanti wrote:
>>>>
>>>> Hi,
>>>>
>>>> mostly showing my astonishment, but I can even understand the figures 
>>>> in this stupid parallelization code
>>>>
>>>> A = [[1.0 1.0001];[1.0002 1.0003]]
>>>> z = A
>>>> tic()
>>>> for i in 1:10
>>>> z *= A
>>>> end
>>>> toc()
>>>> A
>>>>
>>>> produces
>>>>
>>>> elapsed time: 105.458639263 seconds
>>>>
>>>> 2x2 Array{Float64,2}:
>>>>  1.0 1.0001
>>>>  1.0002  1.0003
>>>>
>>>>
>>>>
>>>> But then add @parallel in the for loop
>>>>
>>>> A = [[1.0 1.0001];[1.0002 1.0003]]
>>>> z = A
>>>> tic()
>>>> @parallel for i in 1:10
>>>> z *= A
>>>> end
>>>> toc()
>>>> A
>>>>
>>>> and get 
>>>>
>>>> elapsed time: 0.008912282 seconds
>>>>
>>>> 2x2 Array{Float64,2}:
>>>>  1.0 1.0001
>>>>  1.0002  1.0003
>>>>
>>>>
>>>> look at the elapsed time differences! And I'm running this on my Xeon 
>>>> desktop, not even a cluster
>>>> Of course A-B reports
>>>>
>>>> 2x2 Array{Float64,2}:
>>>>  0.0  0.0
>>>>  0.0  0.0
>>>>
>>>>
>>>> So is this what one should expect from this kind of simple 
>>>> paralleizations? If so, I'm definitely *in love* with Julia :):):)
>>>>
>>>> Best,
>>>>
>>>> Ferran.
>>>>
>>>>
>>>>

[julia-users] Re: I can't believe this spped-up !

[Nathan Smith]

To be clear, you need to compare the final 'z' not the final 'A' to check 
if your calculations are consistent. The matrix A does not change through 
out this calculation, but the matrix z does.
Also, there is no parallelism with the @parallel loop unless your start 
julia with 'julia -np N' where N is the number of processes you'd like to 
use.

On Thursday, 21 July 2016 12:45:17 UTC-4, Ferran Mazzanti wrote:
>
> Hi Nathan,
>
> I posted the codes, so you can check if they do the same thing or not. 
> These went to separate cells in Jupyter, nothing more and nothing less.
> Not even a single line I didn't post. And yes I understand your line of 
> reasoning, so that's why I got astonished also.
> But I can see what is making this huge difference, and I'd like to know :)
>
> Best,
>
> Ferran.
>
> On Thursday, July 21, 2016 at 6:31:57 PM UTC+2, Nathan Smith wrote:
>>
>> Hey Ferran, 
>>
>> You should be suspicious when your apparent speed up surpasses the level 
>> of parallelism available on your CPU. I looks like your codes don't 
>> actually compute the same thing.
>>
>> I'm assuming you're trying to compute the matrix exponential of A 
>> (A^10) by repeatedly multiplying A. In your parallel code, each 
>> process gets a local copy of 'z' and
>> uses that. This means each process is computing something like 
>> (A^(10/# of procs)). Check out this 
>> <http://docs.julialang.org/en/release-0.4/manual/parallel-computing/#parallel-map-and-loops>
>>  section 
>> of the documentation on parallel map and loops to see what I mean.
>>
>> That said, that doesn't explain your speed up completely, you should also 
>> make sure that each part of your script is wrapped in a function and that 
>> you 'warm-up' each function by running it once before comparing.
>>
>> Cheers, 
>> Nathan
>>
>> On Thursday, 21 July 2016 12:00:47 UTC-4, Ferran Mazzanti wrote:
>>>
>>> Hi,
>>>
>>> mostly showing my astonishment, but I can even understand the figures in 
>>> this stupid parallelization code
>>>
>>> A = [[1.0 1.0001];[1.0002 1.0003]]
>>> z = A
>>> tic()
>>> for i in 1:10
>>> z *= A
>>> end
>>> toc()
>>> A
>>>
>>> produces
>>>
>>> elapsed time: 105.458639263 seconds
>>>
>>> 2x2 Array{Float64,2}:
>>>  1.0 1.0001
>>>  1.0002  1.0003
>>>
>>>
>>>
>>> But then add @parallel in the for loop
>>>
>>> A = [[1.0 1.0001];[1.0002 1.0003]]
>>> z = A
>>> tic()
>>> @parallel for i in 1:10
>>> z *= A
>>> end
>>> toc()
>>> A
>>>
>>> and get 
>>>
>>> elapsed time: 0.008912282 seconds
>>>
>>> 2x2 Array{Float64,2}:
>>>  1.0 1.0001
>>>  1.0002  1.0003
>>>
>>>
>>> look at the elapsed time differences! And I'm running this on my Xeon 
>>> desktop, not even a cluster
>>> Of course A-B reports
>>>
>>> 2x2 Array{Float64,2}:
>>>  0.0  0.0
>>>  0.0  0.0
>>>
>>>
>>> So is this what one should expect from this kind of simple 
>>> paralleizations? If so, I'm definitely *in love* with Julia :):):)
>>>
>>> Best,
>>>
>>> Ferran.
>>>
>>>
>>>

[julia-users] Re: I can't believe this spped-up !

[Nathan Smith]

Try comparing these two function:

function serial_example()
A = [[1.0 1.001];[1.002 1.003]
z = A 
for i in 1:10
z *= A
end
return z
end

function parallel_example()
A = [[1.0 1.001]; [1.002 1.003]]
z = @parallel (*) for i in 1:10
A
end
return z
end

[julia-users] Re: I can't believe this spped-up !

[Nathan Smith]

Hey Ferran, 

You should be suspicious when your apparent speed up surpasses the level of 
parallelism available on your CPU. I looks like your codes don't actually 
compute the same thing.

I'm assuming you're trying to compute the matrix exponential of A 
(A^10) by repeatedly multiplying A. In your parallel code, each 
process gets a local copy of 'z' and
uses that. This means each process is computing something like 
(A^(10/# of procs)). Check out this 

 section 
of the documentation on parallel map and loops to see what I mean.

That said, that doesn't explain your speed up completely, you should also 
make sure that each part of your script is wrapped in a function and that 
you 'warm-up' each function by running it once before comparing.

Cheers, 
Nathan

On Thursday, 21 July 2016 12:00:47 UTC-4, Ferran Mazzanti wrote:
>
> Hi,
>
> mostly showing my astonishment, but I can even understand the figures in 
> this stupid parallelization code
>
> A = [[1.0 1.0001];[1.0002 1.0003]]
> z = A
> tic()
> for i in 1:10
> z *= A
> end
> toc()
> A
>
> produces
>
> elapsed time: 105.458639263 seconds
>
> 2x2 Array{Float64,2}:
>  1.0 1.0001
>  1.0002  1.0003
>
>
>
> But then add @parallel in the for loop
>
> A = [[1.0 1.0001];[1.0002 1.0003]]
> z = A
> tic()
> @parallel for i in 1:10
> z *= A
> end
> toc()
> A
>
> and get 
>
> elapsed time: 0.008912282 seconds
>
> 2x2 Array{Float64,2}:
>  1.0 1.0001
>  1.0002  1.0003
>
>
> look at the elapsed time differences! And I'm running this on my Xeon 
> desktop, not even a cluster
> Of course A-B reports
>
> 2x2 Array{Float64,2}:
>  0.0  0.0
>  0.0  0.0
>
>
> So is this what one should expect from this kind of simple 
> paralleizations? If so, I'm definitely *in love* with Julia :):):)
>
> Best,
>
> Ferran.
>
>
>

Re: [julia-users] A question of style: type constructors vs methods

[Nathan Smith]

Hi Gabriel, 

If you're types often have default values you may want to take a look 
Parameters.jl  for a neat and 
useful implementation of types with default values.

As for the Bar() constructor, is may be odd for future people looking at 
your code to have a constructor like method that doesn't correspond to a 
type. 

-Nathan

On Tuesday, 21 June 2016 14:49:40 UTC-4, Gabriel Gellner wrote:
>
> Ugh keyboard error.
>
> So lets say I have the type
>
> type Foo
> field1
> field2
> end
>
> So I have my nice constructors dealing with all the ways I want to create 
> Foo types, but then I want a specialized Foo that has field2 set to some 
> special value that has a special sub meaning from my default Foo. Say I 
> have a default constructor for Foo like
>
> Foo(;field1=, field2=)
>
> But I want to have a different default, so I call this
> Bar(;field1=, field2=) = Foo(field1, field2)
>
> My question! Really this is an abuse of the naming guide, as Bar is really 
> a method, not a type constructor with the same name as the type, but I 
> guess I am thinking about this as a kind of specialized type of Foo, hence 
> the uppercase, but should I instead do
>
> bar(...)
>
> when I am returning these kind of specialized versions of Foo?
>
> On Tuesday, June 21, 2016 at 11:44:07 AM UTC-7, Gabriel Gellner wrote:
>>
>> So going deeper into using my own types, and coding style in julia.
>>
>> Lets say I have the container type
>>
>> type Foo
>>
>>
>> On Monday, June 20, 2016 at 7:10:15 AM UTC-7, Gabriel Gellner wrote:
>>>
>>> That was what I was feeling. That this was a legacy issue for lowercase 
>>> type "constructors". Thanks so much.
>>>
>>> On Monday, June 20, 2016 at 1:11:41 AM UTC-7, Mauro wrote:

 I think you should use the constructors if the user expects to 
 construct 
 a certain type, e.g. `Dict()`.  Conversely if the user cares about the 
 action then use a function, e.g.: 

 julia> keys(Dict()) 
 Base.KeyIterator for a Dict{Any,Any} with 0 entries 

 here I don't care about the type, I just want to iterate the keys. 

 But of course, it's not that clear-cut: `linspace` has history, so does 
 `zeros`.  So, for a new container type I'd use the constructor 
 `FooBar`. 

 On Sun, 2016-06-19 at 23:12, Gabriel Gellner  
 wrote: 
 > I am currently making some container like types, so I am using the 
 > convention of studly caps for  the types ie `FooBar`. For usage I am 
 > confused on what the julian convention is for having expressive type 
 > constructors like for `Dict` and `DataFrame`, versus using methods 
 like 
 > `linspace`. Clearly I could use either, but it is not clear to me 
 when I 
 > should use one convention over the other. 
 > 
 > Clearly I can have my api be like: 
 > 
 > f = FooBar(...) 
 > 
 > or 
 > 
 > f = foobar() 
 > 
 > but is one preferred over the other? Is it just random when to use 
 one or 
 > the other when making container like types? 

>>>

[julia-users] Re: Junitex-Builder: Unicode tab completions in Gnome-Builder

[Nathan Smith]

Update: subscript and superscript characters work now

On Saturday, 7 May 2016 11:06:51 UTC-4, Nathan Smith wrote:
>
> HI Everyone!
>
> I've built a plugin to provide latex to unicode tab completions in the Gnome 
> Builder IDE. <https://wiki.gnome.org/Apps/Builder> Works with *very* 
> recent versions of gnome builder (>=3.20). 
>
> Checkout it out at: https://github.com/nsmith5/Junitex-Builder 
>
> Currently, the plugin won't support subscript or superscript characters 
> for some reason, but otherwise everything else should work. 
>
> I'm hoping to have Junitex-Gedit finished soon as well to help create a 
> better Julia experience on the gnome desktop. Happy to hear any feedback!
>
> Cheers, 
> Nathan
>

[julia-users] Junitex-Builder: Unicode tab completions in Gnome-Builder

[Nathan Smith]

HI Everyone!

I've built a plugin to provide latex to unicode tab completions in the Gnome 
Builder IDE.  Works with *very* recent 
versions of gnome builder (>=3.20). 

Checkout it out at: https://github.com/nsmith5/Junitex-Builder 

Currently, the plugin won't support subscript or superscript characters for 
some reason, but otherwise everything else should work. 

I'm hoping to have Junitex-Gedit finished soon as well to help create a 
better Julia experience on the gnome desktop. Happy to hear any feedback!

Cheers, 
Nathan

[julia-users] MPI-FFTW on Julia

[Nathan Smith]

Hi Folks, 

I'm interesting in trying to call MPI implementations of FFTW from Julia 
using the current MPI.jl and C-interface infrastructure. I'm a somewhat new 
user to julia and was wondering if these will be feasible or if I'm jumping 
in way over my head or if perhaps this isn't even possible at this point in 
time?

If its possible, any pointers towards the ingredients I might need to make 
this happen would be greatly appreciated.

Thanks for any advice, 
Nathan

PS: This language is killer, I've never had so much fun with my simulation 
work!