Re: [julia-users] [Offtopic:] Why you did choose the name "Julia"?

[Jason Merrill]

Last time this came up, Godwin's law manifested early:

https://groups.google.com/d/msg/julia-users/c6rhZHMcIDs/62JB-u2MppMJ

On Monday, November 24, 2014 11:40:14 PM UTC-8, Francesco Bonazzi wrote:
>
> The problem with the name Julia is that you find a lot of other stuff 
> unrelated to programming when you look on Google.

Re: [julia-users] [Offtopic:] Why you did choose the name "Julia"?

[Francesco Bonazzi]

The problem with the name Julia is that you find a lot of other stuff unrelated 
to programming when you look on Google.

Re: [julia-users] UInt should be Uint in documentation

[Pontus Stenetorp]

On 25 November 2014 at 16:16, Ronald L. Rivest  wrote:
>
> You are correct; my apologies for this post.

No worries, we are all happy to help.

> I was reading documentation for development version 0.4.0
> but running 0.3.2.   So, I guess UInt will be the new Uint as of 0.4.0...

Yes, the `Uint` to `UInt` change is a bit of a consistency/clean-up
move and was discussed less than a month ago [1].

Pontus

[1]: https://github.com/JuliaLang/julia/issues/8905

Re: [julia-users] UInt should be Uint in documentation

[Ronald L. Rivest]

You are correct; my apologies for this post.  
I was reading documentation for development version 0.4.0
but running 0.3.2.   So, I guess UInt will be the new Uint as of 0.4.0...

Cheers,
Ron

On Tuesday, November 25, 2014 1:09:29 AM UTC-5, Pontus Stenetorp wrote:
>
> On 25 November 2014 at 15:01, Ronald L. Rivest  > wrote: 
> > 
> > There is an apparent bug (or family of bugs) in the 
> > documentation "julia.pdf". 
> > 
> > The document says unsigned integer is of type UInt, 
> > while the julia system only accepts Uint 
> > (lowercase "i" instead of uppercase "I"). 
> > 
> > This holds for UInt64/Uint64, etc... 
> > 
> > The documentation should be fixed... (or the implementation 
> > made to be in accordance with the documentation...) 
>
> Which version of Julia are you running?  My guess is that you are 
> referring to the development version documentation [1], when what you 
> really want is the 0.3.x release documentation [2] 
>
> > julia0.4 -e 'UInt' 
> > julia0.3 -e 'UInt' 
> ERROR: UInt not defined 
>  in process_options at ./client.jl:213 
>  in _start at ./client.jl:354 
>  in _start_3B_1729 at ~/git/julia0.3/usr/bin/../lib/julia/sys.so 
> > julia0.4 -e 'Uint' 
> > julia0.3 -e 'Uint' 
>
> As seen above, there has been some breaking changes introduced in the 
> development branch. 
>
> Pontus 
>
> [1]: https://media.readthedocs.org/pdf/julia/latest/julia.pdf 
> [2]: https://media.readthedocs.org/pdf/julia/release-0.3/julia.pdf 
>

Re: [julia-users] impressions and questions from a Matlab user

[Uwe Fechner]

You wrote:
"Just to be clear, it takes something like 3 seconds to load Julia, 26 
seconds to load the PyPlot package."

On my Linux Laptop from 2013 (i7-4800MQ CPU @ 2.70GHz, SSD) it takes less 
than 0.3 seconds to load Julia 0.3.2 and 7.3 seconds to load the PyPlot 
package. I am not sure if your performance problems are related to your 
hardware or to Mac OS.

Uwe

On Monday, November 24, 2014 11:37:11 PM UTC+1, Christian Peel wrote:
>
> What I was earlier calling a 'syntax error'  was actually an undefined 
> function; I apologize for not using the correct language.  I believe that 
> syntax errors are caught in the 'include' step, while the 'undefined' error 
> I was actually getting was a compile-time error.   Starting with an 
> undefined variable on the first line of function f in file.jl, here is what 
> i did on the REPL 
> * include file.jl   this takes a fraction of a second
> * run f; after 11ish seconds it notices the undefined variable and returns 
> an error
> * run f again; it immediately complains about the undefined variable
> The number of seconds it takes to compile (the first run of f) are related 
> to the length of the file.
>
> Tim,I guess the fact that I put the undefined function on the first 
> line of the function I was trying to run resolves your question about 
> performance (I am sure there are many things I can do to improve the 
> performance of this toy function).
>
> Elliot, running the command as you requested results in the following on 
> one computer
> julia> filter( x -> contains(x, "sys.dylib"), Sys.dllist())
> 1-element Array{String,1}:
>
>  "/Applications/Julia-0.3.0.app/Contents/Resources/julia/lib/julia/sys.dylib"
> I get similar results on a different computer which uses a homebrew 
> package of Julia 0.3.2.  
>
> Thanks for all your help!
>
> Chris
>
> On Monday, November 24, 2014 2:13:29 AM UTC-8, Tim Holy wrote:
>>
>> When you re-include your script, does it again take 11 seconds to get to 
>> the 
>> error? If so, something is wrong. How much computation is there in your 
>> script? You might have a type inference problem, in which case 
>> performance 
>> will be terrible, and it might simply be taking a long time (much longer 
>> than 
>> Matlab) to get to the point in your computation where the bug is 
>> triggered. 
>>
>> If you haven't already, read at least up through & including the "Tools" 
>> section of 
>> http://docs.julialang.org/en/release-0.3/manual/performance-tips/ 
>> You should definitely profile; that will tell you where the problem is. 
>>
>> --Tim 
>>
>> On Sunday, November 23, 2014 11:35:03 PM Christian Peel wrote: 
>> > >What's your versioninfo? 
>> > 
>> > I used Version 0.3.0   (x86_64-apple-darwin13.3.0) on a 2013 macbook 
>> which 
>> > took about 9.6 seconds to include the function, try to run it, and find 
>> the 
>> > syntax error.   On a 2009 iMac with version 0.3.2 of Julia 
>> > (x86_64-apple-darwin14.0.0) it took 11.3 seconds.Just to be clear, 
>> it 
>> > takes something like 3 seconds to load Julia, 26 seconds to load the 
>> PyPlot 
>> > package (?!?!?) and then an additional 11.3 seconds after that to get 
>> the 
>> > syntax error.   I do not restart Julia every time. 
>> > 
>> > The functions simulate a narrowband multi-antenna fading communications 
>> > channel. To me it feels like a simple and straightforward script, but 
>> it 
>> > may not be so simple for the optimizer. 
>> > 
>> > Chris 
>> > 
>> > On Sunday, November 23, 2014 7:21:36 PM UTC-8, Stefan Karpinski wrote: 
>> > > Ah, yes. That would explain this if you're timing how long it takes 
>> to 
>> > > start Julia from the command prompt. In that case, I can understand 
>> the 
>> > > complaint about the compile-debug-edit cycle, but you probably should 
>> > > consider doing more development at the interactive REPL prompt rather 
>> than 
>> > > restarting Julia every time. 
>> > > 
>> > > On Sun, Nov 23, 2014 at 9:56 PM, Patrick O'Leary <
>> patrick...@gmail.com 
>> > > 
>> > > > wrote: 
>> > >> On Sunday, November 23, 2014 7:55:33 PM UTC-6, Stefan Karpinski 
>> wrote: 
>> > >>> 11 seconds seems like an awfully long time. In the days of the slow 
>> REPL 
>> > >>> when Julia compiled itself upon starting up, that's about how long 
>> it 
>> > >>> took. 
>> > >>> What's your versioninfo? 
>> > >> 
>> > >> Windows doesn't ship with sys.dll, for what it's worth. 
>>
>>

Re: [julia-users] Security problem with unitialized memory

[Ronald L. Rivest]

Sorry; zeros() does not work here instead of new().  My mistake.
Is there a safe alternative to new() that guarantees that all fields
will have a definite fixed value?

Cheers,
Ron

On Tuesday, November 25, 2014 1:05:40 AM UTC-5, Ronald L. Rivest wrote:
>
> The problem also exists for new() (e.g. when initializing a 
> record/object).  zeros() can
> apparently be used here instead.
>
> Cheers,
> Ron
>
> On Tuesday, November 25, 2014 12:29:07 AM UTC-5, Viral Shah wrote:
>>
>> Much has been already said on this topic. 
>>
>> The Array(...) interface was kind of meant to be low-level for the user 
>> of scientific computing, only to be used when they know what they are 
>> doing. You get the raw uninitialized memory as fast as possible.
>>
>> The user-facing interface was always an array constructor - zeros(), 
>> ones(), rand(), etc. Some of this is because of our past experience coming 
>> from a matlab/R-like world. 
>>
>> As Julia has become more popular, we have realized that those not coming 
>> from matlab/R end up using all the possible constructors. While this has 
>> raised a variety of issues, I'd like to say that this will not get sorted 
>> out satisfactorily before the 0.4 release. For a class that may be taught 
>> soon, the thing to do would be to use the zeros/ones/rand constructors to 
>> construct arrays, instead of Array(), which currently is more for a package 
>> developer. I understand that Array() is a much better name as Stefan points 
>> out, but zeros() is not too terrible - it at least clearly tells the user 
>> that they get zeroed out arrays.
>>
>> While we have other "features" that can lead to unsafe code (ccall, 
>> @inbounds), none of these are things one is likely to run into while 
>> learning the language.
>>
>> -viral
>>
>> On Tuesday, November 25, 2014 1:00:10 AM UTC+5:30, Ronald L. Rivest wrote:
>>>
>>> Regarding initialization:
>>>
>>>-- I'm toying with the idea of recommending Julia for an introductory 
>>> programming
>>>   class (rather than Python).  
>>>
>>>-- For this purpose, the language should not have hazards that catch 
>>> the unwary.
>>>
>>>-- Not initializing storage is definitely a hazard.  With 
>>> uninitialized storage, a 
>>>   program may run fine one day, and fail mysteriously the next, 
>>> depending on 
>>>   the contents of memory.  This is about predictability, 
>>> reliability, dependability,
>>>   and correctness.
>>>
>>>-- I would favor a solution like
>>>  A = Array(Int64,n)   -- fills with zeros
>>>  A = Array(Int64,n,fill=1)  -- to fill with ones
>>>  A = Array(Int64,n,fill=None)-- for an uninitialized 
>>> array
>>>so that the *default* is an initialized array, but the speed geeks
>>>can get what they want.
>>>
>>> Cheers,
>>> Ron
>>>
>>> On Monday, November 24, 2014 1:57:14 PM UTC-5, Stefan Karpinski wrote:

 If we can make allocating zeroed arrays faster that's great, but unless 
 we can close the performance gap all the way and eliminate the need to 
 allocated uninitialized arrays altogether, this proposal is just a rename 
 – Unchecked.Array 
 plays the exact same role as the current Array constructor. It's 
 unclear that this would even address the original concern since it still 
 *allows* uninitialized allocation of arrays. This rename would just force 
 people who have used Array correctly in code that cares about being as 
 efficient as possible even for very large arrays to change their code and 
 use Unchecked.Array instead.

 On Nov 24, 2014, at 1:36 PM, Jameson Nash  wrote:

 I think that Rivest’s question may be a good reason to rethink the 
 initialization of structs and offer the explicit guarantee that all 
 unassigned elements will be initialized to 0 (and not just the jl_value_t 
 pointers). I would argue that the current behavior resulted more from a 
 desire to avoid clearing the array twice (if the user is about to call 
 fill, zeros, ones, +, etc.) than an intentional, casual exposure of 
 uninitialized memory.

 A random array of integers is also a security concern if an attacker 
 can extract some other information (with some probability) about the state 
 of the program. Julia is not hardened by design, so you can’t safely run 
 an 
 unknown code fragment, but you still might have an unintended memory 
 exposure in a client-facing app. While zero’ing memory doesn’t prevent the 
 user from simply reusing a memory buffer in a security-unaware fashion 
 (rather than consistently allocating a new one for each use), it’s not 
 clear to me that the performance penalty would be all that noticeable for 
 map Array(X) to zero(X), and only providing an internal constructor for 
 grabbing uninitialized memory (perhaps Base.Unchecked.Array(X) from 
 #8227)

 On Mon Nov 24

Re: [julia-users] UInt should be Uint in documentation

[Pontus Stenetorp]

On 25 November 2014 at 15:01, Ronald L. Rivest  wrote:
>
> There is an apparent bug (or family of bugs) in the
> documentation "julia.pdf".
>
> The document says unsigned integer is of type UInt,
> while the julia system only accepts Uint
> (lowercase "i" instead of uppercase "I").
>
> This holds for UInt64/Uint64, etc...
>
> The documentation should be fixed... (or the implementation
> made to be in accordance with the documentation...)

Which version of Julia are you running?  My guess is that you are
referring to the development version documentation [1], when what you
really want is the 0.3.x release documentation [2]

> julia0.4 -e 'UInt'
> julia0.3 -e 'UInt'
ERROR: UInt not defined
 in process_options at ./client.jl:213
 in _start at ./client.jl:354
 in _start_3B_1729 at ~/git/julia0.3/usr/bin/../lib/julia/sys.so
> julia0.4 -e 'Uint'
> julia0.3 -e 'Uint'

As seen above, there has been some breaking changes introduced in the
development branch.

Pontus

[1]: https://media.readthedocs.org/pdf/julia/latest/julia.pdf
[2]: https://media.readthedocs.org/pdf/julia/release-0.3/julia.pdf

[julia-users] Re: Neuro-Dynamic Programming in Julia

[wildart]

Reinforcement learning (RL) isn't covered much in Julia packages. There is 
a collection of RL algorithms over MDP in package: 
https://github.com/cpritcha/MDP. There is a collection of IJulia notebooks 
from a Stanford course that cover more RL algorithms: 
https://github.com/sisl/aa228-notebook/tree/master

Unfortunately, more advanced function approximation techniques, beyond 
look-up table, that allow to tackle large action-state spaces, are nowhere 
to find.

Couple a month ago, Shane Conway, the guy behind RL-Glue 
, talked about developing 
Julia RL-Glue client. If that happens, it would be quite simple to use 
various advanced RL algorithms, including value function approximators, in 
Julia. 


On Saturday, November 22, 2014 11:12:29 PM UTC-5, Pileas wrote:
>
> Some problems have the so-called curse of dimensionality and curse of 
> modeling. For this reason Bersekas and Tsimtsiklis (at MIT) introduced the 
> so-called Neuro-Dynamic Programing.
>
> Does Julia offer support for the aforementioned and if not, how about the 
> future?
>

Re: [julia-users] Security problem with unitialized memory

[Ronald L. Rivest]

The problem also exists for new() (e.g. when initializing a record/object). 
 zeros() can
apparently be used here instead.

Cheers,
Ron

On Tuesday, November 25, 2014 12:29:07 AM UTC-5, Viral Shah wrote:
>
> Much has been already said on this topic. 
>
> The Array(...) interface was kind of meant to be low-level for the user of 
> scientific computing, only to be used when they know what they are doing. 
> You get the raw uninitialized memory as fast as possible.
>
> The user-facing interface was always an array constructor - zeros(), 
> ones(), rand(), etc. Some of this is because of our past experience coming 
> from a matlab/R-like world. 
>
> As Julia has become more popular, we have realized that those not coming 
> from matlab/R end up using all the possible constructors. While this has 
> raised a variety of issues, I'd like to say that this will not get sorted 
> out satisfactorily before the 0.4 release. For a class that may be taught 
> soon, the thing to do would be to use the zeros/ones/rand constructors to 
> construct arrays, instead of Array(), which currently is more for a package 
> developer. I understand that Array() is a much better name as Stefan points 
> out, but zeros() is not too terrible - it at least clearly tells the user 
> that they get zeroed out arrays.
>
> While we have other "features" that can lead to unsafe code (ccall, 
> @inbounds), none of these are things one is likely to run into while 
> learning the language.
>
> -viral
>
> On Tuesday, November 25, 2014 1:00:10 AM UTC+5:30, Ronald L. Rivest wrote:
>>
>> Regarding initialization:
>>
>>-- I'm toying with the idea of recommending Julia for an introductory 
>> programming
>>   class (rather than Python).  
>>
>>-- For this purpose, the language should not have hazards that catch 
>> the unwary.
>>
>>-- Not initializing storage is definitely a hazard.  With 
>> uninitialized storage, a 
>>   program may run fine one day, and fail mysteriously the next, 
>> depending on 
>>   the contents of memory.  This is about predictability, reliability, 
>> dependability,
>>   and correctness.
>>
>>-- I would favor a solution like
>>  A = Array(Int64,n)   -- fills with zeros
>>  A = Array(Int64,n,fill=1)  -- to fill with ones
>>  A = Array(Int64,n,fill=None)-- for an uninitialized array
>>so that the *default* is an initialized array, but the speed geeks
>>can get what they want.
>>
>> Cheers,
>> Ron
>>
>> On Monday, November 24, 2014 1:57:14 PM UTC-5, Stefan Karpinski wrote:
>>>
>>> If we can make allocating zeroed arrays faster that's great, but unless 
>>> we can close the performance gap all the way and eliminate the need to 
>>> allocated uninitialized arrays altogether, this proposal is just a rename – 
>>> Unchecked.Array 
>>> plays the exact same role as the current Array constructor. It's 
>>> unclear that this would even address the original concern since it still 
>>> *allows* uninitialized allocation of arrays. This rename would just force 
>>> people who have used Array correctly in code that cares about being as 
>>> efficient as possible even for very large arrays to change their code and 
>>> use Unchecked.Array instead.
>>>
>>> On Nov 24, 2014, at 1:36 PM, Jameson Nash  wrote:
>>>
>>> I think that Rivest’s question may be a good reason to rethink the 
>>> initialization of structs and offer the explicit guarantee that all 
>>> unassigned elements will be initialized to 0 (and not just the jl_value_t 
>>> pointers). I would argue that the current behavior resulted more from a 
>>> desire to avoid clearing the array twice (if the user is about to call 
>>> fill, zeros, ones, +, etc.) than an intentional, casual exposure of 
>>> uninitialized memory.
>>>
>>> A random array of integers is also a security concern if an attacker can 
>>> extract some other information (with some probability) about the state of 
>>> the program. Julia is not hardened by design, so you can’t safely run an 
>>> unknown code fragment, but you still might have an unintended memory 
>>> exposure in a client-facing app. While zero’ing memory doesn’t prevent the 
>>> user from simply reusing a memory buffer in a security-unaware fashion 
>>> (rather than consistently allocating a new one for each use), it’s not 
>>> clear to me that the performance penalty would be all that noticeable for 
>>> map Array(X) to zero(X), and only providing an internal constructor for 
>>> grabbing uninitialized memory (perhaps Base.Unchecked.Array(X) from 
>>> #8227)
>>>
>>> On Mon Nov 24 2014 at 12:57:22 PM Stefan Karpinski 
>>> stefan.karpin...@gmail.com  
>>> wrote:
>>>
>>> There are two rather different issues to consider:

 1. Preventing problems due to inadvertent programmer errors.
 2. Preventing malicious security attacks.

 When we initially made this choice, it wasn't clear if 1 woul

[julia-users] UInt should be Uint in documentation

[Ronald L. Rivest]

There is an apparent bug (or family of bugs) in the
documentation "julia.pdf".

The document says unsigned integer is of type UInt,
while the julia system only accepts Uint
(lowercase "i" instead of uppercase "I").

This holds for UInt64/Uint64, etc...

The documentation should be fixed... (or the implementation
made to be in accordance with the documentation...)

(Sorry if this is a repeat, but I searched and didn't find an
early note of this...)

Cheers,
Ron

[julia-users] Re: Julia v0.3.3

[Viral Shah]

The Guardians of the Release Branch are also hard at work on 0.3.4.



Courtesy @staticfloat in https://github.com/JuliaLang/julia/issues/9045

-viral

On Tuesday, November 25, 2014 1:55:53 AM UTC+5:30, Ivar Nesje wrote:
>
> We should also note that this is a recommended upgrade for anyone using 
> any of the previous 0.3.x releases.
>
> We have taken all reasonable measures to avoid breaking existing programs, 
> so it should be pretty hassle free. We would like to get feedback if 
> someone has a correctly working program that doesn't work after this 
> upgrade.
>
> Ivar
>
>
> kl. 20:24:12 UTC+1 mandag 24. november 2014 skrev Elliot Saba følgende:
>>
>> Hello all!  The latest bugfix release of the 0.3.X Julia line has been 
>> released.  Binaries are available from the usual place 
>> , and as usual, please report all 
>> issues to either the issue tracker 
>> , or email this list.
>>
>> As this is a bugfix release, there are no new big-item features to 
>> announce, but if you are interested in the bugs fixed since 0.3.2, this 
>> commit log  
>> should 
>> give you an idea of the effort put in by our team of backporters.
>>
>> Happy Hacking,
>> -E
>>
>

Re: [julia-users] Security problem with unitialized memory

[Viral Shah]

Much has been already said on this topic. 

The Array(...) interface was kind of meant to be low-level for the user of 
scientific computing, only to be used when they know what they are doing. 
You get the raw uninitialized memory as fast as possible.

The user-facing interface was always an array constructor - zeros(), 
ones(), rand(), etc. Some of this is because of our past experience coming 
from a matlab/R-like world. 

As Julia has become more popular, we have realized that those not coming 
from matlab/R end up using all the possible constructors. While this has 
raised a variety of issues, I'd like to say that this will not get sorted 
out satisfactorily before the 0.4 release. For a class that may be taught 
soon, the thing to do would be to use the zeros/ones/rand constructors to 
construct arrays, instead of Array(), which currently is more for a package 
developer. I understand that Array() is a much better name as Stefan points 
out, but zeros() is not too terrible - it at least clearly tells the user 
that they get zeroed out arrays.

While we have other "features" that can lead to unsafe code (ccall, 
@inbounds), none of these are things one is likely to run into while 
learning the language.

-viral

On Tuesday, November 25, 2014 1:00:10 AM UTC+5:30, Ronald L. Rivest wrote:
>
> Regarding initialization:
>
>-- I'm toying with the idea of recommending Julia for an introductory 
> programming
>   class (rather than Python).  
>
>-- For this purpose, the language should not have hazards that catch 
> the unwary.
>
>-- Not initializing storage is definitely a hazard.  With uninitialized 
> storage, a 
>   program may run fine one day, and fail mysteriously the next, 
> depending on 
>   the contents of memory.  This is about predictability, reliability, 
> dependability,
>   and correctness.
>
>-- I would favor a solution like
>  A = Array(Int64,n)   -- fills with zeros
>  A = Array(Int64,n,fill=1)  -- to fill with ones
>  A = Array(Int64,n,fill=None)-- for an uninitialized array
>so that the *default* is an initialized array, but the speed geeks
>can get what they want.
>
> Cheers,
> Ron
>
> On Monday, November 24, 2014 1:57:14 PM UTC-5, Stefan Karpinski wrote:
>>
>> If we can make allocating zeroed arrays faster that's great, but unless 
>> we can close the performance gap all the way and eliminate the need to 
>> allocated uninitialized arrays altogether, this proposal is just a rename – 
>> Unchecked.Array 
>> plays the exact same role as the current Array constructor. It's unclear 
>> that this would even address the original concern since it still *allows* 
>> uninitialized allocation of arrays. This rename would just force people who 
>> have used Array correctly in code that cares about being as efficient as 
>> possible even for very large arrays to change their code and use 
>> Unchecked.Array instead.
>>
>> On Nov 24, 2014, at 1:36 PM, Jameson Nash  wrote:
>>
>> I think that Rivest’s question may be a good reason to rethink the 
>> initialization of structs and offer the explicit guarantee that all 
>> unassigned elements will be initialized to 0 (and not just the jl_value_t 
>> pointers). I would argue that the current behavior resulted more from a 
>> desire to avoid clearing the array twice (if the user is about to call 
>> fill, zeros, ones, +, etc.) than an intentional, casual exposure of 
>> uninitialized memory.
>>
>> A random array of integers is also a security concern if an attacker can 
>> extract some other information (with some probability) about the state of 
>> the program. Julia is not hardened by design, so you can’t safely run an 
>> unknown code fragment, but you still might have an unintended memory 
>> exposure in a client-facing app. While zero’ing memory doesn’t prevent the 
>> user from simply reusing a memory buffer in a security-unaware fashion 
>> (rather than consistently allocating a new one for each use), it’s not 
>> clear to me that the performance penalty would be all that noticeable for 
>> map Array(X) to zero(X), and only providing an internal constructor for 
>> grabbing uninitialized memory (perhaps Base.Unchecked.Array(X) from 
>> #8227)
>>
>> On Mon Nov 24 2014 at 12:57:22 PM Stefan Karpinski 
>> stefan.karpin...@gmail.com  
>> wrote:
>>
>> There are two rather different issues to consider:
>>>
>>> 1. Preventing problems due to inadvertent programmer errors.
>>> 2. Preventing malicious security attacks.
>>>
>>> When we initially made this choice, it wasn't clear if 1 would be a big 
>>> issue but we decided to see how it played out. It hasn't been a problem in 
>>> practice: once people grok that the Array(T, dims...) constructor gives 
>>> uninitialized memory and that the standard usage pattern is to call it and 
>>> then immediately initialize the memory, everything is ok. I can't 
>>> rec

Re: [julia-users] Security problem with unitialized memory

[Erik Schnetter]

On Mon, Nov 24, 2014 at 11:15 PM, Stefan Karpinski  wrote:
> That's not the point - if you already have memory and have to fill it, then
> you're not in any position for the kernel to lazily zero it, so the
> alignment of arbitrary arrays is irrelevant. The point SGJ was making is
> that we want to allocate the memory using something calloc-like so that the
> kernel can do lazy zeroing for us, but we also need that memory to be
> 16-byte aligned, but there is not portable way to get 16-byte-aligned memory
> that the kernel will lazily zero for you. We can have lazy zeroing or
> 16-byte alignment but not both. This makes me wonder if we couldn't just
> allocate 15 bytes more than necessary and return the first address that on a
> 16-byte boundary.

On each system, malloc makes certain guarantees about the alignment of
the pointer it returns. On a 64-bit system, malloc will likely return
memory that is at least aligned to 8 bytes, maybe more. Thus one would
need to allocate 8 additional bytes, not 15.

-erik

-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

Re: [julia-users] Security problem with unitialized memory

[Viral Shah]

Right - we can always use calloc and then provide aligned memory.

Perhaps this is worth benchmarking. It is still likely to be much faster than 
malloc + memset, because it should have significantly better cache behaviour, 
even though the zeroing is not free. The question is, whether this cost is 
small enough.

-viral



> On 25-Nov-2014, at 9:45 am, Stefan Karpinski  wrote:
> 
> That's not the point – if you already have memory and have to fill it, then 
> you're not in any position for the kernel to lazily zero it, so the alignment 
> of arbitrary arrays is irrelevant. The point SGJ was making is that we want 
> to allocate the memory using something calloc-like so that the kernel can do 
> lazy zeroing for us, but we also need that memory to be 16-byte aligned, but 
> there is not portable way to get 16-byte-aligned memory that the kernel will 
> lazily zero for you. We can have lazy zeroing or 16-byte alignment but not 
> both. This makes me wonder if we couldn't just allocate 15 bytes more than 
> necessary and return the first address that on a 16-byte boundary.
> 
> On Mon, Nov 24, 2014 at 11:02 PM, Viral Shah  wrote:
> To add to the point, you can also get non-aligned stuff with subarrays or 
> results from a ccall.
> 
> -viral
> 
> 
> On Tuesday, November 25, 2014 9:24:36 AM UTC+5:30, Simon Kornblith wrote:
> In general, arrays cannot be assumed to be 16-byte aligned because it's 
> always possible to create one that isn't using pointer_to_array. However, 
> from Intel's AVX introduction:
> 
> Intel® AVX has relaxed some memory alignment requirements, so now Intel AVX 
> by default allows unaligned access; however, this access may come at a 
> performance slowdown, so the old rule of designing your data to be memory 
> aligned is still good practice (16-byte aligned for 128-bit access and 
> 32-byte aligned for 256-bit access).
> 
> On Monday, November 24, 2014 10:01:45 PM UTC-5, Erik Schnetter wrote:
> On Mon, Nov 24, 2014 at 9:30 PM, Steven G. Johnson 
>  wrote: 
> > Unfortunately, Julia allocates 16-byte aligned data by default (to help 
> > SIMD 
> > code), and there is no calloc version of posix_memalign as far as I know. 
> 
> The generated machine code I've seen does not make use of this. All 
> the load/store instructions in vectorized or unrolled loops assume 
> unaligned pointers. (Plus, with AVX one should align to 32 bytes 
> instead.) 
> 
> -erik 
> 
> -- 
> Erik Schnetter  
> http://www.perimeterinstitute.ca/personal/eschnetter/ 
> 

Re: [julia-users] Security problem with unitialized memory

[Stefan Karpinski]

That's not the point – if you already have memory and have to fill it, then
you're not in any position for the kernel to lazily zero it, so the
alignment of arbitrary arrays is irrelevant. The point SGJ was making is
that we want to allocate the memory using something calloc-like so that the
kernel can do lazy zeroing for us, but we also need that memory to be
16-byte aligned, but there is not portable way to get 16-byte-aligned
memory that the kernel will lazily zero for you. We can have lazy zeroing
or 16-byte alignment but not both. This makes me wonder if we couldn't just
allocate 15 bytes more than necessary and return the first address that on
a 16-byte boundary.

On Mon, Nov 24, 2014 at 11:02 PM, Viral Shah  wrote:

> To add to the point, you can also get non-aligned stuff with subarrays or
> results from a ccall.
>
> -viral
>
>
> On Tuesday, November 25, 2014 9:24:36 AM UTC+5:30, Simon Kornblith wrote:
>>
>> In general, arrays cannot be assumed to be 16-byte aligned because it's
>> always possible to create one that isn't using pointer_to_array.
>> However, from Intel's AVX introduction
>> 
>> :
>>
>> Intel® AVX has relaxed some memory alignment requirements, so now Intel
>> AVX by default allows unaligned access; however, this access may come at a
>> performance slowdown, so the old rule of designing your data to be memory
>> aligned is still good practice (16-byte aligned for 128-bit access and
>> 32-byte aligned for 256-bit access).
>>
>> On Monday, November 24, 2014 10:01:45 PM UTC-5, Erik Schnetter wrote:
>>>
>>> On Mon, Nov 24, 2014 at 9:30 PM, Steven G. Johnson
>>>  wrote:
>>> > Unfortunately, Julia allocates 16-byte aligned data by default (to
>>> help SIMD
>>> > code), and there is no calloc version of posix_memalign as far as I
>>> know.
>>>
>>> The generated machine code I've seen does not make use of this. All
>>> the load/store instructions in vectorized or unrolled loops assume
>>> unaligned pointers. (Plus, with AVX one should align to 32 bytes
>>> instead.)
>>>
>>> -erik
>>>
>>> --
>>> Erik Schnetter 
>>> http://www.perimeterinstitute.ca/personal/eschnetter/
>>>
>>

Re: [julia-users] Security problem with unitialized memory

[Viral Shah]

To add to the point, you can also get non-aligned stuff with subarrays or 
results from a ccall.

-viral

On Tuesday, November 25, 2014 9:24:36 AM UTC+5:30, Simon Kornblith wrote:
>
> In general, arrays cannot be assumed to be 16-byte aligned because it's 
> always possible to create one that isn't using pointer_to_array. However, 
> from Intel's AVX introduction 
> 
> :
>
> Intel® AVX has relaxed some memory alignment requirements, so now Intel 
> AVX by default allows unaligned access; however, this access may come at a 
> performance slowdown, so the old rule of designing your data to be memory 
> aligned is still good practice (16-byte aligned for 128-bit access and 
> 32-byte aligned for 256-bit access).
>
> On Monday, November 24, 2014 10:01:45 PM UTC-5, Erik Schnetter wrote:
>>
>> On Mon, Nov 24, 2014 at 9:30 PM, Steven G. Johnson 
>>  wrote: 
>> > Unfortunately, Julia allocates 16-byte aligned data by default (to help 
>> SIMD 
>> > code), and there is no calloc version of posix_memalign as far as I 
>> know. 
>>
>> The generated machine code I've seen does not make use of this. All 
>> the load/store instructions in vectorized or unrolled loops assume 
>> unaligned pointers. (Plus, with AVX one should align to 32 bytes 
>> instead.) 
>>
>> -erik 
>>
>> -- 
>> Erik Schnetter  
>> http://www.perimeterinstitute.ca/personal/eschnetter/ 
>>
>

Re: [julia-users] Security problem with unitialized memory

[Simon Kornblith]

In general, arrays cannot be assumed to be 16-byte aligned because it's 
always possible to create one that isn't using pointer_to_array. However, 
from Intel's AVX introduction 

:

Intel® AVX has relaxed some memory alignment requirements, so now Intel AVX 
by default allows unaligned access; however, this access may come at a 
performance slowdown, so the old rule of designing your data to be memory 
aligned is still good practice (16-byte aligned for 128-bit access and 
32-byte aligned for 256-bit access).

On Monday, November 24, 2014 10:01:45 PM UTC-5, Erik Schnetter wrote:
>
> On Mon, Nov 24, 2014 at 9:30 PM, Steven G. Johnson 
> > wrote: 
> > Unfortunately, Julia allocates 16-byte aligned data by default (to help 
> SIMD 
> > code), and there is no calloc version of posix_memalign as far as I 
> know. 
>
> The generated machine code I've seen does not make use of this. All 
> the load/store instructions in vectorized or unrolled loops assume 
> unaligned pointers. (Plus, with AVX one should align to 32 bytes 
> instead.) 
>
> -erik 
>
> -- 
> Erik Schnetter > 
> http://www.perimeterinstitute.ca/personal/eschnetter/ 
>

Re: [julia-users] Security problem with unitialized memory

[Viral Shah]

Filling random garbage is even more time consuming than filling zeros! 

-viral

On Tuesday, November 25, 2014 1:26:29 AM UTC+5:30, Stefan Karpinski wrote:
>
> I guess part of the problem is that calling the `zeros` function may be 
> less obvious as a way of constructing an array to many new programmers than 
> calling the `Array` constructor. Having a Boolean fill keyword argument 
> approach might be reasonable, although calling it `zeroed` might be more 
> accurate since we won't be filling the array with a specific value but 
> rather zeroing the memory first. Alternatively, we could just fill the 
> array with random garbage intentionally so that programmers are made 
> painfully aware that they didn't initialize the array ;-)
>
> On Nov 24, 2014, at 2:39 PM, Tomas Lycken  wrote:
>
> That *is* the default usage in most introductory settings - just don't 
> show them the Array(T,n) constructor, but give them zeros and ones 
> functions instead. (It's perfectly fine to do e.g. A = zeros(10); fill!(A, 
> 5) if you don't care about the extra write...)
>
> If there is a specific setting where the students actually *need* to 
> allocate uninitialized memory (e.g. for speed), they are probably ready to 
> learn that the Array constructor gives them that.
>
> Julia's approach has so far seemed to be that the users are consenting 
> adults. I like that approach.
>
> // T
>
> On Monday, November 24, 2014 8:30:10 PM UTC+1, Ronald L. Rivest wrote:
>>
>> Regarding initialization:
>>
>>-- I'm toying with the idea of recommending Julia for an introductory 
>> programming
>>   class (rather than Python).  
>>
>>-- For this purpose, the language should not have hazards that catch 
>> the unwary.
>>
>>-- Not initializing storage is definitely a hazard.  With 
>> uninitialized storage, a 
>>   program may run fine one day, and fail mysteriously the next, 
>> depending on 
>>   the contents of memory.  This is about predictability, reliability, 
>> dependability,
>>   and correctness.
>>
>>-- I would favor a solution like
>>  A = Array(Int64,n)   -- fills with zeros
>>  A = Array(Int64,n,fill=1)  -- to fill with ones
>>  A = Array(Int64,n,fill=None)-- for an uninitialized array
>>so that the *default* is an initialized array, but the speed geeks
>>can get what they want.
>>
>> Cheers,
>> Ron
>>
>> On Monday, November 24, 2014 1:57:14 PM UTC-5, Stefan Karpinski wrote:
>>>
>>> If we can make allocating zeroed arrays faster that's great, but unless 
>>> we can close the performance gap all the way and eliminate the need to 
>>> allocated uninitialized arrays altogether, this proposal is just a rename – 
>>> Unchecked.Array 
>>> plays the exact same role as the current Array constructor. It's 
>>> unclear that this would even address the original concern since it still 
>>> *allows* uninitialized allocation of arrays. This rename would just force 
>>> people who have used Array correctly in code that cares about being as 
>>> efficient as possible even for very large arrays to change their code and 
>>> use Unchecked.Array instead.
>>>
>>> On Nov 24, 2014, at 1:36 PM, Jameson Nash  wrote:
>>>
>>> I think that Rivest’s question may be a good reason to rethink the 
>>> initialization of structs and offer the explicit guarantee that all 
>>> unassigned elements will be initialized to 0 (and not just the jl_value_t 
>>> pointers). I would argue that the current behavior resulted more from a 
>>> desire to avoid clearing the array twice (if the user is about to call 
>>> fill, zeros, ones, +, etc.) than an intentional, casual exposure of 
>>> uninitialized memory.
>>>
>>> A random array of integers is also a security concern if an attacker can 
>>> extract some other information (with some probability) about the state of 
>>> the program. Julia is not hardened by design, so you can’t safely run an 
>>> unknown code fragment, but you still might have an unintended memory 
>>> exposure in a client-facing app. While zero’ing memory doesn’t prevent the 
>>> user from simply reusing a memory buffer in a security-unaware fashion 
>>> (rather than consistently allocating a new one for each use), it’s not 
>>> clear to me that the performance penalty would be all that noticeable for 
>>> map Array(X) to zero(X), and only providing an internal constructor for 
>>> grabbing uninitialized memory (perhaps Base.Unchecked.Array(X) from 
>>> #8227)
>>>
>>> On Mon Nov 24 2014 at 12:57:22 PM Stefan Karpinski 
>>> stefan.karpin...@gmail.com  
>>> wrote:
>>>
>>> There are two rather different issues to consider:

 1. Preventing problems due to inadvertent programmer errors.
 2. Preventing malicious security attacks.

 When we initially made this choice, it wasn't clear if 1 would be a big 
 issue but we decided to see how it played out. It hasn't been a problem in 
 pr

Re: [julia-users] Security problem with unitialized memory

[Viral Shah]

How much faster is zeros with your PR? IIRC, we used to have something like 
this in the early days.

https://github.com/JuliaLang/julia/pull/9142/

-viral

On Tuesday, November 25, 2014 8:30:00 AM UTC+5:30, Erik Schnetter wrote:
>
> On Mon, Nov 24, 2014 at 7:19 PM, Stefan Karpinski  
> wrote: 
> > Should the comparison actually be more like this: 
> > 
> > julia> @time begin 
> >x = Array(Int,N) 
> >fill!(x,1) 
> >end; 
> > elapsed time: 6.782572096 seconds (800128 bytes allocated) 
> > 
> > julia> @time begin 
> >x = zeros(Int,N) 
> >fill!(x,1) 
> >end; 
> > elapsed time: 14.166256835 seconds (800176 bytes allocated) 
> > 
> > 
> > At least that's the comparison that makes sense for code that allocates 
> and 
> > then initializes an array. I consistently see a 2x slowdown or more. 
>
> My laptop can zero memory at 6+ GByte/sec. The overhead in your case 
> should be about ten times less than what you report. I suspect that 
> what you are measuring is not just the overhead of `zeros` over 
> `Array`, but something else as well. 
>
> Or Julia's `zeros` is not implemented efficiently. Zeroing memory is 
> surprisingly CPU intensive, and one has to call `memset`, or has to 
> ensure that the vectorizer kicks in, and may have to manually unroll 
> the loops. I'm not sure LLVM gets this right by default... 
>
> ... yes, Julia's `zeros` is not as good as it should be. It calls 
> `fill!`, but the generated machine code leaves to be desired. Time to 
> add a special case to `fill!`, delegating things to `memset` in 
> certain cases? 
>
> -erik 
>
> -- 
> Erik Schnetter  
> http://www.perimeterinstitute.ca/personal/eschnetter/ 
>

Re: [julia-users] Error when running parallel Julia with large number of threads

[Kapil]

I tried with the latest 0.3.3 release and with 0.4.0-dev+1061 version, but
both are giving errors.

As Amit has already created a similar issue, should I still open another
issue ?

Regards,
Kapil Agarwal

On Mon, Nov 24, 2014 at 5:23 AM, Tim Holy  wrote:

> That error message sequence reminds me of the one I saw in debugging a
> problem
> on julia 0.4 that was missing a "GC root," meaning that some julia variable
> allocated from the C code was not being correctly protected from garbage-
> collection.
>
> While the bug I fixed can't be present on 0.3 (the relevant code is julia
> 0.4-
> only), it is possible that your bug has been fixed by someone else in the
> meantime---there have been quite a few updates to the 0.3 series. Since
> you're
> running a slightly outdated version, can you update to the latest 0.3
> release?
>
> If updating doesn't fix it, please file an issue with the code needed to
> reproduce the problem.
>
> Best,
> --Tim
>
> On Sunday, November 23, 2014 07:48:08 PM Kapil Agarwal wrote:
> > Hi
> >
> > I am running a parallel matrix transpose using Julia. When I run it on a
> > small matrix with small number of workers, it works fine, but as I
> increase
> > the size and the number of workers, it starts giving MemoryError() and
> > Broken pipe signals.
> >
> > I have put the error stacktrace here :
> >  https://github.com/kapiliitr/JuliaBenchmarks/blob/master/error.txt
> >
> > I checked that my program was not making any bounds errors and that the
> > result is correct. Also, I did not find any errors in the source code
> from
> > the stacktrace.
> > I am basically dealing with large matrices of the order of 1000X1000
> > elements and around 50-200 worker processes.
> >
> > I have a machine with 24 cores and around 24 GB memory, so could this be
> a
> > problem with my system or is there a limit to how many workers Julia can
> > launch and allocate memory ?
> >
> > Thanks
> >
> > Kapil
>
>

Re: [julia-users] Security problem with unitialized memory

[Erik Schnetter]

On Mon, Nov 24, 2014 at 9:30 PM, Steven G. Johnson
 wrote:
> Unfortunately, Julia allocates 16-byte aligned data by default (to help SIMD
> code), and there is no calloc version of posix_memalign as far as I know.

The generated machine code I've seen does not make use of this. All
the load/store instructions in vectorized or unrolled loops assume
unaligned pointers. (Plus, with AVX one should align to 32 bytes
instead.)

-erik

-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

Re: [julia-users] Security problem with unitialized memory

[Erik Schnetter]

On Mon, Nov 24, 2014 at 7:19 PM, Stefan Karpinski  wrote:
> Should the comparison actually be more like this:
>
> julia> @time begin
>x = Array(Int,N)
>fill!(x,1)
>end;
> elapsed time: 6.782572096 seconds (800128 bytes allocated)
>
> julia> @time begin
>x = zeros(Int,N)
>fill!(x,1)
>end;
> elapsed time: 14.166256835 seconds (800176 bytes allocated)
>
>
> At least that's the comparison that makes sense for code that allocates and
> then initializes an array. I consistently see a 2x slowdown or more.

My laptop can zero memory at 6+ GByte/sec. The overhead in your case
should be about ten times less than what you report. I suspect that
what you are measuring is not just the overhead of `zeros` over
`Array`, but something else as well.

Or Julia's `zeros` is not implemented efficiently. Zeroing memory is
surprisingly CPU intensive, and one has to call `memset`, or has to
ensure that the vectorizer kicks in, and may have to manually unroll
the loops. I'm not sure LLVM gets this right by default...

... yes, Julia's `zeros` is not as good as it should be. It calls
`fill!`, but the generated machine code leaves to be desired. Time to
add a special case to `fill!`, delegating things to `memset` in
certain cases?

-erik

-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

Re: [julia-users] Security problem with unitialized memory

[Jameson Nash]

we could also turn on USE_MMAP by default. all systems should be returning
zero pages for new memory. then we only need to worry about manually
zero'ing things that are smaller than a page

On Mon Nov 24 2014 at 9:30:46 PM Steven G. Johnson 
wrote:

>
>
> On Monday, November 24, 2014 3:05:59 PM UTC-5, Stefan Karpinski wrote:
>>
>> Using calloc could reduce this significantly since the kernel can lazily
>> fill zero pages only when you access them.
>>
>
> Even better, on some systems (e.g. Linux) calloc is implemented via
> copy-on-write of a single zero page, so it only needs to fill new zero
> pages when you write.
>
> Unfortunately, Julia allocates 16-byte aligned data by default (to help
> SIMD code), and there is no calloc version of posix_memalign as far as I
> know.
>

Re: [julia-users] Security problem with unitialized memory

[Steven G. Johnson]

On Monday, November 24, 2014 3:05:59 PM UTC-5, Stefan Karpinski wrote:
>
> Using calloc could reduce this significantly since the kernel can lazily 
> fill zero pages only when you access them.
>

Even better, on some systems (e.g. Linux) calloc is implemented via 
copy-on-write of a single zero page, so it only needs to fill new zero 
pages when you write.

Unfortunately, Julia allocates 16-byte aligned data by default (to help 
SIMD code), and there is no calloc version of posix_memalign as far as I 
know.

Re: [julia-users] Security problem with unitialized memory

[Jacob Quinn]

The variance you're seeing is most likely due to the garbage collector
kicking with that much memory being allocated and then abandoned.

On Mon, Nov 24, 2014 at 7:53 PM, David Smith  wrote:

> This is what I was thinking. I just assumed that the fill() time would be
> constant for both and factored that out, not knowing that malloc() was lazy.
>
> I get similar results for Stefan's bench, although the variance is large.
>
> On Monday, November 24, 2014 6:20:21 PM UTC-6, Stefan Karpinski wrote:
>>
>> Should the comparison actually be more like this:
>>
>> julia> @time begin
>>x = Array(Int,N)
>>fill!(x,1)
>>end;
>> elapsed time: 6.782572096 seconds (800128 bytes allocated)
>>
>> julia> @time begin
>>x = zeros(Int,N)
>>fill!(x,1)
>>end;
>> elapsed time: 14.166256835 seconds (800176 bytes allocated)
>>
>>
>> At least that's the comparison that makes sense for code that allocates
>> and then initializes an array. I consistently see a 2x slowdown or more.
>>
>> On Mon, Nov 24, 2014 at 7:09 PM, Jameson Nash  wrote:
>>
>>> > But you initialized it in both cases.
>>>
>>> Yes.
>>>
>>> > Is there a compiler optimization going on here that combines the
>>> zeros() and fill()?
>>>
>>> No.
>>>
>>> But there is a kernel optimization going on that complicates this
>>> measurement. Approximately, the memory requested by `malloc` (& friends) is
>>> not actually allocated until you try to read or write to it. So there are
>>> in fact 3 effects here (roughly speaking, they are malloc, A[1:4096:end],
>>> and fill()), where that second operation is unavoidable, and orders of
>>> magnitude slower than the other two. You measured the speed of 1 vs. 1+2+3.
>>> Whereas I measured the speed of 1+2+3 vs 1+2+3+3.
>>>
>>> On Mon Nov 24 2014 at 6:59:50 PM David Smith  wrote:
>>>
 But you initialized it in both cases.  Is there a compiler optimization
 going on here that combines the zeros() and fill()?


 On Monday, November 24, 2014 5:12:56 PM UTC-6, Jameson wrote:

> yes. the point is to compare the cost of implicitly calling `zero`
> (resulting in the equivalent of calling zero twice) to the cost of not
> initializing the memory before writing to it. I could alternatively have
> done: `@time x=zeros(); @time fill(x, 0)` to measure the same information.
>
> On Mon Nov 24 2014 at 5:57:29 PM David Smith 
> wrote:
>
>> Did you mean to call zeros() in both cases?
>>
>>
>> On Monday, November 24, 2014 3:09:38 PM UTC-6, Jameson wrote:
>>
>>> It appears the fill operation accounts for about 0.15 seconds of the
>>> 6.15 seconds that my OS X laptop takes to create this array:
>>>
>>> $ ./julia -q
>>>
>>> *julia> **N=10^9*
>>>
>>> *10*
>>>
>>>
>>> *julia> **@time begin x=zeros(Int64,N); fill(x,0) end*
>>>
>>> elapsed time: 6.325660691 seconds (8000136616 bytes allocated, 1.71%
>>> gc time)
>>>
>>> *0-element Array{Array{Int64,1},1}*
>>>
>>>
>>> $ ./julia -q
>>>
>>> *julia> **N=10^9*
>>>
>>> *10*
>>>
>>>
>>> *julia> **@time x=zeros(Int64,N)*
>>>
>>> elapsed time: 6.160623835 seconds (814320 bytes allocated, 0.22%
>>> gc time)
>>>
>>>
>>>
>>> On Mon Nov 24 2014 at 3:18:39 PM Erik Schnetter 
>>> wrote:
>>>
 On Mon, Nov 24, 2014 at 3:01 PM, David Smith 
 wrote:
 > To add some data to this conversation, I just timed allocating a
 billion
 > Int64s on my macbook, and I got this (I ran these multiple times
 before this
 > and got similar timings):
 >
 > julia> N=1_000_000_000
 > 10
 >
 > julia> @time x = Array(Int64,N);
 > elapsed time: 0.022577671 seconds (800128 bytes allocated)
 >
 > julia> @time x = zeros(Int64,N);
 > elapsed time: 3.95432248 seconds (800152 bytes allocated)
 >
 > So we are talking adding possibly seconds to a program per large
 array
 > allocation.

 This is not quite right -- the first does not actually map the pages
 into memory; this is only done lazily when they are accessed the
 first
 time. You need to compare "alloc uninitialized; then initialize
 once"
 with "alloc zero-initialized; then initialize again".

 Current high-end system architectures have memory write speeds of
 ten
 or twenty GByte per second; this is what you should see for very
 large
 arrays -- this would be about 0.4 seconds for your case. For smaller
 arrays, the data would reside in the cache, so that the allocation
 overhead should be significantly smaller even.

 -erik

 --

>>> Erik Schnetter 
 http://www

Re: [julia-users] Security problem with unitialized memory

[David Smith]

This is what I was thinking. I just assumed that the fill() time would be 
constant for both and factored that out, not knowing that malloc() was lazy.

I get similar results for Stefan's bench, although the variance is large.

On Monday, November 24, 2014 6:20:21 PM UTC-6, Stefan Karpinski wrote:
>
> Should the comparison actually be more like this:
>
> julia> @time begin
>x = Array(Int,N)
>fill!(x,1)
>end;
> elapsed time: 6.782572096 seconds (800128 bytes allocated)
>
> julia> @time begin
>x = zeros(Int,N)
>fill!(x,1)
>end;
> elapsed time: 14.166256835 seconds (800176 bytes allocated)
>
>
> At least that's the comparison that makes sense for code that allocates 
> and then initializes an array. I consistently see a 2x slowdown or more.
>
> On Mon, Nov 24, 2014 at 7:09 PM, Jameson Nash  > wrote:
>
>> > But you initialized it in both cases. 
>>
>> Yes.
>>
>> > Is there a compiler optimization going on here that combines the 
>> zeros() and fill()?
>>
>> No.
>>
>> But there is a kernel optimization going on that complicates this 
>> measurement. Approximately, the memory requested by `malloc` (& friends) is 
>> not actually allocated until you try to read or write to it. So there are 
>> in fact 3 effects here (roughly speaking, they are malloc, A[1:4096:end], 
>> and fill()), where that second operation is unavoidable, and orders of 
>> magnitude slower than the other two. You measured the speed of 1 vs. 1+2+3. 
>> Whereas I measured the speed of 1+2+3 vs 1+2+3+3.
>>
>> On Mon Nov 24 2014 at 6:59:50 PM David Smith > > wrote:
>>
>>> But you initialized it in both cases.  Is there a compiler optimization 
>>> going on here that combines the zeros() and fill()?
>>>
>>>
>>> On Monday, November 24, 2014 5:12:56 PM UTC-6, Jameson wrote:
>>>
 yes. the point is to compare the cost of implicitly calling `zero` 
 (resulting in the equivalent of calling zero twice) to the cost of not 
 initializing the memory before writing to it. I could alternatively have 
 done: `@time x=zeros(); @time fill(x, 0)` to measure the same information.

 On Mon Nov 24 2014 at 5:57:29 PM David Smith  
 wrote:

> Did you mean to call zeros() in both cases?
>
>
> On Monday, November 24, 2014 3:09:38 PM UTC-6, Jameson wrote:
>
>> It appears the fill operation accounts for about 0.15 seconds of the 
>> 6.15 seconds that my OS X laptop takes to create this array:
>>
>> $ ./julia -q
>>
>> *julia> **N=10^9*
>>
>> *10*
>>
>>
>> *julia> **@time begin x=zeros(Int64,N); fill(x,0) end*
>>
>> elapsed time: 6.325660691 seconds (8000136616 bytes allocated, 1.71% 
>> gc time)
>>
>> *0-element Array{Array{Int64,1},1}*
>>
>>
>> $ ./julia -q
>>
>> *julia> **N=10^9*
>>
>> *10*
>>
>>
>> *julia> **@time x=zeros(Int64,N)*
>>
>> elapsed time: 6.160623835 seconds (814320 bytes allocated, 0.22% 
>> gc time)
>>
>>
>>
>> On Mon Nov 24 2014 at 3:18:39 PM Erik Schnetter  
>> wrote:
>>
>>> On Mon, Nov 24, 2014 at 3:01 PM, David Smith  
>>> wrote:
>>> > To add some data to this conversation, I just timed allocating a 
>>> billion
>>> > Int64s on my macbook, and I got this (I ran these multiple times 
>>> before this
>>> > and got similar timings):
>>> >
>>> > julia> N=1_000_000_000
>>> > 10
>>> >
>>> > julia> @time x = Array(Int64,N);
>>> > elapsed time: 0.022577671 seconds (800128 bytes allocated)
>>> >
>>> > julia> @time x = zeros(Int64,N);
>>> > elapsed time: 3.95432248 seconds (800152 bytes allocated)
>>> >
>>> > So we are talking adding possibly seconds to a program per large 
>>> array
>>> > allocation.
>>>
>>> This is not quite right -- the first does not actually map the pages
>>> into memory; this is only done lazily when they are accessed the 
>>> first
>>> time. You need to compare "alloc uninitialized; then initialize once"
>>> with "alloc zero-initialized; then initialize again".
>>>
>>> Current high-end system architectures have memory write speeds of ten
>>> or twenty GByte per second; this is what you should see for very 
>>> large
>>> arrays -- this would be about 0.4 seconds for your case. For smaller
>>> arrays, the data would reside in the cache, so that the allocation
>>> overhead should be significantly smaller even.
>>>
>>> -erik
>>>
>>> --
>>>
>> Erik Schnetter 
>>> http://www.perimeterinstitute.ca/personal/eschnetter/
>>>
>>  
>

Re: [julia-users] Security problem with unitialized memory

[Stefan Karpinski]

Should the comparison actually be more like this:

julia> @time begin
   x = Array(Int,N)
   fill!(x,1)
   end;
elapsed time: 6.782572096 seconds (800128 bytes allocated)

julia> @time begin
   x = zeros(Int,N)
   fill!(x,1)
   end;
elapsed time: 14.166256835 seconds (800176 bytes allocated)


At least that's the comparison that makes sense for code that allocates and
then initializes an array. I consistently see a 2x slowdown or more.

On Mon, Nov 24, 2014 at 7:09 PM, Jameson Nash  wrote:

> > But you initialized it in both cases.
>
> Yes.
>
> > Is there a compiler optimization going on here that combines the zeros()
> and fill()?
>
> No.
>
> But there is a kernel optimization going on that complicates this
> measurement. Approximately, the memory requested by `malloc` (& friends) is
> not actually allocated until you try to read or write to it. So there are
> in fact 3 effects here (roughly speaking, they are malloc, A[1:4096:end],
> and fill()), where that second operation is unavoidable, and orders of
> magnitude slower than the other two. You measured the speed of 1 vs. 1+2+3.
> Whereas I measured the speed of 1+2+3 vs 1+2+3+3.
>
> On Mon Nov 24 2014 at 6:59:50 PM David Smith 
> wrote:
>
>> But you initialized it in both cases.  Is there a compiler optimization
>> going on here that combines the zeros() and fill()?
>>
>>
>> On Monday, November 24, 2014 5:12:56 PM UTC-6, Jameson wrote:
>>
>>> yes. the point is to compare the cost of implicitly calling `zero`
>>> (resulting in the equivalent of calling zero twice) to the cost of not
>>> initializing the memory before writing to it. I could alternatively have
>>> done: `@time x=zeros(); @time fill(x, 0)` to measure the same information.
>>>
>>> On Mon Nov 24 2014 at 5:57:29 PM David Smith  wrote:
>>>
 Did you mean to call zeros() in both cases?


 On Monday, November 24, 2014 3:09:38 PM UTC-6, Jameson wrote:

> It appears the fill operation accounts for about 0.15 seconds of the
> 6.15 seconds that my OS X laptop takes to create this array:
>
> $ ./julia -q
>
> *julia> **N=10^9*
>
> *10*
>
>
> *julia> **@time begin x=zeros(Int64,N); fill(x,0) end*
>
> elapsed time: 6.325660691 seconds (8000136616 bytes allocated, 1.71%
> gc time)
>
> *0-element Array{Array{Int64,1},1}*
>
>
> $ ./julia -q
>
> *julia> **N=10^9*
>
> *10*
>
>
> *julia> **@time x=zeros(Int64,N)*
>
> elapsed time: 6.160623835 seconds (814320 bytes allocated, 0.22%
> gc time)
>
>
>
> On Mon Nov 24 2014 at 3:18:39 PM Erik Schnetter 
> wrote:
>
>> On Mon, Nov 24, 2014 at 3:01 PM, David Smith 
>> wrote:
>> > To add some data to this conversation, I just timed allocating a
>> billion
>> > Int64s on my macbook, and I got this (I ran these multiple times
>> before this
>> > and got similar timings):
>> >
>> > julia> N=1_000_000_000
>> > 10
>> >
>> > julia> @time x = Array(Int64,N);
>> > elapsed time: 0.022577671 seconds (800128 bytes allocated)
>> >
>> > julia> @time x = zeros(Int64,N);
>> > elapsed time: 3.95432248 seconds (800152 bytes allocated)
>> >
>> > So we are talking adding possibly seconds to a program per large
>> array
>> > allocation.
>>
>> This is not quite right -- the first does not actually map the pages
>> into memory; this is only done lazily when they are accessed the first
>> time. You need to compare "alloc uninitialized; then initialize once"
>> with "alloc zero-initialized; then initialize again".
>>
>> Current high-end system architectures have memory write speeds of ten
>> or twenty GByte per second; this is what you should see for very large
>> arrays -- this would be about 0.4 seconds for your case. For smaller
>> arrays, the data would reside in the cache, so that the allocation
>> overhead should be significantly smaller even.
>>
>> -erik
>>
>> --
>>
> Erik Schnetter 
>> http://www.perimeterinstitute.ca/personal/eschnetter/
>>
>

Re: [julia-users] Security problem with unitialized memory

[Jameson Nash]

> But you initialized it in both cases.

Yes.

> Is there a compiler optimization going on here that combines the zeros()
and fill()?

No.

But there is a kernel optimization going on that complicates this
measurement. Approximately, the memory requested by `malloc` (& friends) is
not actually allocated until you try to read or write to it. So there are
in fact 3 effects here (roughly speaking, they are malloc, A[1:4096:end],
and fill()), where that second operation is unavoidable, and orders of
magnitude slower than the other two. You measured the speed of 1 vs. 1+2+3.
Whereas I measured the speed of 1+2+3 vs 1+2+3+3.

On Mon Nov 24 2014 at 6:59:50 PM David Smith  wrote:

> But you initialized it in both cases.  Is there a compiler optimization
> going on here that combines the zeros() and fill()?
>
>
> On Monday, November 24, 2014 5:12:56 PM UTC-6, Jameson wrote:
>
>> yes. the point is to compare the cost of implicitly calling `zero`
>> (resulting in the equivalent of calling zero twice) to the cost of not
>> initializing the memory before writing to it. I could alternatively have
>> done: `@time x=zeros(); @time fill(x, 0)` to measure the same information.
>>
>> On Mon Nov 24 2014 at 5:57:29 PM David Smith  wrote:
>>
>>> Did you mean to call zeros() in both cases?
>>>
>>>
>>> On Monday, November 24, 2014 3:09:38 PM UTC-6, Jameson wrote:
>>>
 It appears the fill operation accounts for about 0.15 seconds of the
 6.15 seconds that my OS X laptop takes to create this array:

 $ ./julia -q

 *julia> **N=10^9*

 *10*


 *julia> **@time begin x=zeros(Int64,N); fill(x,0) end*

 elapsed time: 6.325660691 seconds (8000136616 bytes allocated, 1.71% gc
 time)

 *0-element Array{Array{Int64,1},1}*


 $ ./julia -q

 *julia> **N=10^9*

 *10*


 *julia> **@time x=zeros(Int64,N)*

 elapsed time: 6.160623835 seconds (814320 bytes allocated, 0.22% gc
 time)



 On Mon Nov 24 2014 at 3:18:39 PM Erik Schnetter 
 wrote:

> On Mon, Nov 24, 2014 at 3:01 PM, David Smith 
> wrote:
> > To add some data to this conversation, I just timed allocating a
> billion
> > Int64s on my macbook, and I got this (I ran these multiple times
> before this
> > and got similar timings):
> >
> > julia> N=1_000_000_000
> > 10
> >
> > julia> @time x = Array(Int64,N);
> > elapsed time: 0.022577671 seconds (800128 bytes allocated)
> >
> > julia> @time x = zeros(Int64,N);
> > elapsed time: 3.95432248 seconds (800152 bytes allocated)
> >
> > So we are talking adding possibly seconds to a program per large
> array
> > allocation.
>
> This is not quite right -- the first does not actually map the pages
> into memory; this is only done lazily when they are accessed the first
> time. You need to compare "alloc uninitialized; then initialize once"
> with "alloc zero-initialized; then initialize again".
>
> Current high-end system architectures have memory write speeds of ten
> or twenty GByte per second; this is what you should see for very large
> arrays -- this would be about 0.4 seconds for your case. For smaller
> arrays, the data would reside in the cache, so that the allocation
> overhead should be significantly smaller even.
>
> -erik
>
> --
>
 Erik Schnetter 
> http://www.perimeterinstitute.ca/personal/eschnetter/
>

Re: [julia-users] Security problem with unitialized memory

[David Smith]

But you initialized it in both cases.  Is there a compiler optimization 
going on here that combines the zeros() and fill()?

On Monday, November 24, 2014 5:12:56 PM UTC-6, Jameson wrote:
>
> yes. the point is to compare the cost of implicitly calling `zero` 
> (resulting in the equivalent of calling zero twice) to the cost of not 
> initializing the memory before writing to it. I could alternatively have 
> done: `@time x=zeros(); @time fill(x, 0)` to measure the same information.
>
> On Mon Nov 24 2014 at 5:57:29 PM David Smith  > wrote:
>
>> Did you mean to call zeros() in both cases?
>>
>>
>> On Monday, November 24, 2014 3:09:38 PM UTC-6, Jameson wrote:
>>
>>> It appears the fill operation accounts for about 0.15 seconds of the 
>>> 6.15 seconds that my OS X laptop takes to create this array:
>>>
>>> $ ./julia -q
>>>
>>> *julia> **N=10^9*
>>>
>>> *10*
>>>
>>>
>>> *julia> **@time begin x=zeros(Int64,N); fill(x,0) end*
>>>
>>> elapsed time: 6.325660691 seconds (8000136616 bytes allocated, 1.71% gc 
>>> time)
>>>
>>> *0-element Array{Array{Int64,1},1}*
>>>
>>>
>>> $ ./julia -q
>>>
>>> *julia> **N=10^9*
>>>
>>> *10*
>>>
>>>
>>> *julia> **@time x=zeros(Int64,N)*
>>>
>>> elapsed time: 6.160623835 seconds (814320 bytes allocated, 0.22% gc 
>>> time)
>>>
>>>
>>>
>>> On Mon Nov 24 2014 at 3:18:39 PM Erik Schnetter  
>>> wrote:
>>>
 On Mon, Nov 24, 2014 at 3:01 PM, David Smith  
 wrote:
 > To add some data to this conversation, I just timed allocating a 
 billion
 > Int64s on my macbook, and I got this (I ran these multiple times 
 before this
 > and got similar timings):
 >
 > julia> N=1_000_000_000
 > 10
 >
 > julia> @time x = Array(Int64,N);
 > elapsed time: 0.022577671 seconds (800128 bytes allocated)
 >
 > julia> @time x = zeros(Int64,N);
 > elapsed time: 3.95432248 seconds (800152 bytes allocated)
 >
 > So we are talking adding possibly seconds to a program per large array
 > allocation.

 This is not quite right -- the first does not actually map the pages
 into memory; this is only done lazily when they are accessed the first
 time. You need to compare "alloc uninitialized; then initialize once"
 with "alloc zero-initialized; then initialize again".

 Current high-end system architectures have memory write speeds of ten
 or twenty GByte per second; this is what you should see for very large
 arrays -- this would be about 0.4 seconds for your case. For smaller
 arrays, the data would reside in the cache, so that the allocation
 overhead should be significantly smaller even.

 -erik

 --

>>> Erik Schnetter 
 http://www.perimeterinstitute.ca/personal/eschnetter/

>>>  

[julia-users] Re: Mutate C struct represented as Julia immutable

[Eric Davies]

On Friday, 21 November 2014 09:44:27 UTC-6, Steven G. Johnson wrote:
>
>
> On Thursday, November 20, 2014 5:37:22 PM UTC-5, Eric Davies wrote:
>>
>> (For context, I'm working on this issue: 
>> https://github.com/JuliaOpt/ECOS.jl/issues/12 and dealing with these 
>> structs: 
>> https://github.com/JuliaOpt/ECOS.jl/blob/master/src/types.jl#L124-L216 )
>>
>> I have a C struct used in a 3rd-party C library mirrored as an immutable 
>> in Julia. A pointer to the C struct is returned in another C struct, and I 
>> get the immutable using pointer_to_array(...)[1]. I want to be able to 
>> modify fields of the struct in-place, but immutables disallow this. How do 
>> I go about this? 
>>
>
> Why not just use pointer_to_array(...)[1] = ...new immutable..., or 
> unsafe_store!(pointer-to-immutable, new immutable) to store a new struct 
> (build from the old struct + modifications) in the old location?
>

I did not know you could do those things, thanks. In general I feel that 
set of functions is not well-explained in the documentation.

I wish there was a way to programmatically create an immutable from another 
immutable and modified field names/values. I think my solution right now is 
iterate over all the fields in order and check if they're to be modified, 
store the resulting modified+unchanged values into a list and splat the 
list in the constructor. Seems like there could be a better way to do it.  

Re: [julia-users] Compose.jl Tutorials for working on Gadfly.jl

[Jiahao Chen]

Here is a quick-and-dirty primer based on my own experience:

1. Let's draw some circles:

compose(context(), circle(0.5, 0.5, 0.3))

Each compose() call takes a context() and zero or more objects which can be
concrete things to draw like circle() or abstract things like fill colors.
The idea is that compose() composites everything in the current context().

compose(context(), circle(0.5, 0.5, 0.3), stroke("blue"), linewidth(5mm),
fill("red"))

The context also lets you do things like change the coordinate system for
the current context().

compose(context(units=UnitBox(-1,-1,4,4)), circle(0.5, 0.5, 0.3))



2. Let's draw some (poly-)lines:

compose(context(), line([(0., 0.), (1., 1.), ]), stroke("black"))

compose(context(), line([(0., 0.), (1., 0.), (1., 1.), ]), stroke("blue"))



3. Here is a short snippet to visualize small matrices as Hinton plots

using Compose
Compose.set_default_graphic_size(4inch, 4inch)

function plothinton{T<:Real}(M::AbstractMatrix{T})
Elements = Any[]
m, n= size(M)
for i=1:m, j=1:n
z = abs(M[i,j])
thecolor = M[i,j]<0 ? "white" : "black"
push!(Elements, compose(context(units=UnitBox(0.5,0.5,m,n)),
rectangle(i-z/2, j-z/2, z, z), stroke(thecolor),
fill(thecolor)))
end
compose(context(), rectangle(0,0,1,1), fill("lavender"), Elements...)
end

#Simple example
x=2rand(25, 25).-1 |> x-> (x+x')/2
plothinton(x)

#end

Notice in this example that there are two different sets of contexts with
different coordinates.

I also like making an array of Compose elements and then splatting it into
a final compose() command.

Thanks,

Jiahao Chen
Staff Research Scientist
MIT Computer Science and Artificial Intelligence Laboratory

On Mon, Nov 24, 2014 at 5:34 PM, Isaiah Norton 
wrote:

> There are some nice Compose examples in the "parallel prefix" notebook,
> here: https://github.com/jiahao/ijulia-notebooks
>
>
> http://nbviewer.ipython.org/github/jiahao/ijulia-notebooks/blob/master/2014-08-06-parallel-prefix.ipynb
>
>
> On Sat, Nov 22, 2014 at 4:08 AM,  wrote:
>
>> Are there any tutorials for Compose.jl other than the documentation page?
>>
>> I want to try to make pie charts and venn diagrams.
>>
>>
>

Re: [julia-users] [Offtopic:] Why you did choose the name "Julia"?

[K Leo]

Very likely Julia was a significant person in that someone's mind.  
Perhaps we should find out who she was.


On 2014年11月25日 06:28, Isaiah Norton wrote:


Alan Edelman told me specifically that it was not named after the
fractal, and in fact that "Julia" doesn't refer to anything in
particular.   Apparently, it just came up in a random conversation
years ago when someone suggested arbitrarily that "Julia" would be
a good name for a programming language.

Re: [julia-users] Security problem with unitialized memory

[Jameson Nash]

yes. the point is to compare the cost of implicitly calling `zero`
(resulting in the equivalent of calling zero twice) to the cost of not
initializing the memory before writing to it. I could alternatively have
done: `@time x=zeros(); @time fill(x, 0)` to measure the same information.

On Mon Nov 24 2014 at 5:57:29 PM David Smith  wrote:

> Did you mean to call zeros() in both cases?
>
>
> On Monday, November 24, 2014 3:09:38 PM UTC-6, Jameson wrote:
>
>> It appears the fill operation accounts for about 0.15 seconds of the 6.15
>> seconds that my OS X laptop takes to create this array:
>>
>> $ ./julia -q
>>
>> *julia> **N=10^9*
>>
>> *10*
>>
>>
>> *julia> **@time begin x=zeros(Int64,N); fill(x,0) end*
>>
>> elapsed time: 6.325660691 seconds (8000136616 bytes allocated, 1.71% gc
>> time)
>>
>> *0-element Array{Array{Int64,1},1}*
>>
>>
>> $ ./julia -q
>>
>> *julia> **N=10^9*
>>
>> *10*
>>
>>
>> *julia> **@time x=zeros(Int64,N)*
>>
>> elapsed time: 6.160623835 seconds (814320 bytes allocated, 0.22% gc
>> time)
>>
>>
>>
>> On Mon Nov 24 2014 at 3:18:39 PM Erik Schnetter 
>> wrote:
>>
>>> On Mon, Nov 24, 2014 at 3:01 PM, David Smith  wrote:
>>> > To add some data to this conversation, I just timed allocating a
>>> billion
>>> > Int64s on my macbook, and I got this (I ran these multiple times
>>> before this
>>> > and got similar timings):
>>> >
>>> > julia> N=1_000_000_000
>>> > 10
>>> >
>>> > julia> @time x = Array(Int64,N);
>>> > elapsed time: 0.022577671 seconds (800128 bytes allocated)
>>> >
>>> > julia> @time x = zeros(Int64,N);
>>> > elapsed time: 3.95432248 seconds (800152 bytes allocated)
>>> >
>>> > So we are talking adding possibly seconds to a program per large array
>>> > allocation.
>>>
>>> This is not quite right -- the first does not actually map the pages
>>> into memory; this is only done lazily when they are accessed the first
>>> time. You need to compare "alloc uninitialized; then initialize once"
>>> with "alloc zero-initialized; then initialize again".
>>>
>>> Current high-end system architectures have memory write speeds of ten
>>> or twenty GByte per second; this is what you should see for very large
>>> arrays -- this would be about 0.4 seconds for your case. For smaller
>>> arrays, the data would reside in the cache, so that the allocation
>>> overhead should be significantly smaller even.
>>>
>>> -erik
>>>
>>> --
>>>
>> Erik Schnetter 
>>> http://www.perimeterinstitute.ca/personal/eschnetter/
>>>
>>

Re: [julia-users] Security problem with unitialized memory

[David Smith]

Did you mean to call zeros() in both cases?

On Monday, November 24, 2014 3:09:38 PM UTC-6, Jameson wrote:
>
> It appears the fill operation accounts for about 0.15 seconds of the 6.15 
> seconds that my OS X laptop takes to create this array:
>
> $ ./julia -q
>
> *julia> **N=10^9*
>
> *10*
>
>
> *julia> **@time begin x=zeros(Int64,N); fill(x,0) end*
>
> elapsed time: 6.325660691 seconds (8000136616 bytes allocated, 1.71% gc 
> time)
>
> *0-element Array{Array{Int64,1},1}*
>
>
> $ ./julia -q
>
> *julia> **N=10^9*
>
> *10*
>
>
> *julia> **@time x=zeros(Int64,N)*
>
> elapsed time: 6.160623835 seconds (814320 bytes allocated, 0.22% gc 
> time)
>
>
>
> On Mon Nov 24 2014 at 3:18:39 PM Erik Schnetter  > wrote:
>
>> On Mon, Nov 24, 2014 at 3:01 PM, David Smith > > wrote:
>> > To add some data to this conversation, I just timed allocating a billion
>> > Int64s on my macbook, and I got this (I ran these multiple times before 
>> this
>> > and got similar timings):
>> >
>> > julia> N=1_000_000_000
>> > 10
>> >
>> > julia> @time x = Array(Int64,N);
>> > elapsed time: 0.022577671 seconds (800128 bytes allocated)
>> >
>> > julia> @time x = zeros(Int64,N);
>> > elapsed time: 3.95432248 seconds (800152 bytes allocated)
>> >
>> > So we are talking adding possibly seconds to a program per large array
>> > allocation.
>>
>> This is not quite right -- the first does not actually map the pages
>> into memory; this is only done lazily when they are accessed the first
>> time. You need to compare "alloc uninitialized; then initialize once"
>> with "alloc zero-initialized; then initialize again".
>>
>> Current high-end system architectures have memory write speeds of ten
>> or twenty GByte per second; this is what you should see for very large
>> arrays -- this would be about 0.4 seconds for your case. For smaller
>> arrays, the data would reside in the cache, so that the allocation
>> overhead should be significantly smaller even.
>>
>> -erik
>>
>> --
>> Erik Schnetter >
>> http://www.perimeterinstitute.ca/personal/eschnetter/
>>
>

Re: [julia-users] [Offtopic:] Why you did choose the name "Julia"?

[Pileas]

I propose this song as the official Julia song:

http://www.youtube.com/watch?v=42j488Ru-Wk

Τη Δευτέρα, 24 Νοεμβρίου 2014 5:48:28 μ.μ. UTC-5, ο χρήστης Pileas έγραψε:
>
> Maybe due to a song they were listening at that time (sub-conscious stuff)!
>
> Haha
>
> Τη Δευτέρα, 24 Νοεμβρίου 2014 5:28:06 μ.μ. UTC-5, ο χρήστης Isaiah έγραψε:
>>
>> Alan Edelman told me specifically that it was not named after the 
>>> fractal, and in fact that "Julia" doesn't refer to anything in 
>>> particular.   Apparently, it just came up in a random conversation years 
>>> ago when someone suggested arbitrarily that "Julia" would be a good name 
>>> for a programming language.
>>>
>>  
>> https://groups.google.com/d/msg/julia-users/6xEWQz_mK10/XrCkLV-7O_sJ
>>
>> This should probably go in a FAQ somewhere.
>>
>> On Mon, Nov 24, 2014 at 5:17 PM, Pileas  wrote:
>>
>>> Is there a specific reason you guys chose this name?
>>>
>>> It kinda reminds me Linux Mint where each distro. has a girl's name ...
>>>
>>
>>

[julia-users] Re: [ANN, x-post julia-stats] Mocha.jl Deep Learning Library for Julia

[David van Leeuwen]

Very cool!

---david

On Friday, November 21, 2014 3:00:42 AM UTC+1, Chiyuan Zhang wrote:
>
> https://github.com/pluskid/Mocha.jl
> Mocha is a Deep Learning framework for Julia , 
> inspired by the C++ Deep Learning framework Caffe 
> . Since this is the first time I post 
> announcement here, change logs of the last two releases are listed:
>
> v0.0.2 2014.11.20
>
>- Infrastructure
>   - Ability to import caffe trained model
>   - Properly release all the allocated resources upon backend shutdown
>- Network
>   - Sigmoid activation function
>   - Power, Split, Element-wise layers
>   - Local Response Normalization layer
>   - Channel Pooling layer
>   - Dropout Layer
>- Documentation
>   - Complete MNIST demo
>   - Complete CIFAR-10 demo
>   - Major part of User's Guide
>
> v0.0.1 
> 2014.11.13
>
>- Backend
>   - Pure Julia CPU
>   - Julia + C++ Extension CPU
>   - CUDA + cuDNN GPU
>- Infrastructure
>   - Evaluate on validation set during training
>   - Automaticly saving and recovering from snapshots
>- Network
>   - Convolution layer, mean and max pooling layer, fully connected 
>   layer, softmax loss layer
>   - ReLU activation function
>   - L2 Regularization
>- Solver
>   - SGD with momentum
>- Documentation
>   - Demo code of LeNet on MNIST
>   - Tutorial document on the MNIST demo (half finished)
>
>
>
> Below is a copy of the README file:
> 
>
> Mocha is a Deep Learning framework for Julia , 
> inspired by the C++ Deep Learning framework Caffe 
> . Mocha support multiple backends:
>
>- Pure Julia CPU Backend: Implemented in pure Julia; Runs out of the 
>box without any external dependency; Reasonably fast on small models 
> thanks 
>to Julia's LLVM-based just-in-time (JIT) compiler and Performance 
>Annotations 
>
> 
>  that 
>eliminate unnecessary bound checkings.
>- CPU Backend with Native Extension: Some bottleneck computations 
>(Convolution and Pooling) have C++ implementations. When compiled and 
>enabled, could be faster than the pure Julia backend.
>- CUDA + cuDNN: An interface to NVidia® cuDNN 
> GPU accelerated deep learning 
>library. When run with CUDA GPU devices, could be much faster depending on 
>the size of the problem (e.g. on MNIST CUDA backend is roughly 20 times 
>faster than the pure Julia backend).
>
> Installation
>
> To install the release version, simply run
>
> Pkg.add("Mocha")
>
> in Julia console. To install the latest development version, run the 
> following command instead:
>
> Pkg.clone("https://github.com/pluskid/Mocha.jl.git";)
>
> Then you can run the built-in unit tests with
>
> Pkg.test("Mocha")
>
> to verify that everything is functioning properly on your machine.
> Hello World
>
> Please refer to the MNIST tutorial 
>  on how 
> prepare the MNIST dataset for the following example. The complete code for 
> this example is located at examples/mnist/mnist.jl 
> . 
> See below for detailed documentation of other tutorials and user's guide.
>
> using Mocha
>
> data  = 
> HDF5DataLayer(name="train-data",source="train-data-list.txt",batch_size=64)
> conv  = 
> ConvolutionLayer(name="conv1",n_filter=20,kernel=(5,5),bottoms=[:data],tops=[:conv])
> pool  = 
> PoolingLayer(name="pool1",kernel=(2,2),stride=(2,2),bottoms=[:conv],tops=[:pool])
> conv2 = 
> ConvolutionLayer(name="conv2",n_filter=50,kernel=(5,5),bottoms=[:pool],tops=[:conv2])
> pool2 = 
> PoolingLayer(name="pool2",kernel=(2,2),stride=(2,2),bottoms=[:conv2],tops=[:pool2])
> fc1   = 
> InnerProductLayer(name="ip1",output_dim=500,neuron=Neurons.ReLU(),bottoms=[:pool2],
>   tops=[:ip1])
> fc2   = InnerProductLayer(name="ip2",output_dim=10,bottoms=[:ip1],tops=[:ip2])
> loss  = SoftmaxLossLayer(name="loss",bottoms=[:ip2,:label])
>
> sys = System(CuDNNBackend())init(sys)
>
> common_layers = [conv, pool, conv2, pool2, fc1, fc2]
> net = Net("MNIST-train", sys, [data, common_layers..., loss])
>
> params = SolverParameters(max_iter=1, regu_coef=0.0005, momentum=0.9,
> lr_policy=LRPolicy.Inv(0.01, 0.0001, 0.75))
> solver = SGD(params)
> # report training progress every 100 iterationsadd_coffee_break(solver, 
> TrainingSummary(), every_n_iter=100)
> # save snapshots every 5000 iterationsadd_coffee_break

Re: [julia-users] [Offtopic:] Why you did choose the name "Julia"?

[Pileas]

Maybe due to a song they were listening at that time (sub-conscious stuff)!

Haha

Τη Δευτέρα, 24 Νοεμβρίου 2014 5:28:06 μ.μ. UTC-5, ο χρήστης Isaiah έγραψε:
>
> Alan Edelman told me specifically that it was not named after the fractal, 
>> and in fact that "Julia" doesn't refer to anything in particular.   
>> Apparently, it just came up in a random conversation years ago when someone 
>> suggested arbitrarily that "Julia" would be a good name for a programming 
>> language.
>>
>  
> https://groups.google.com/d/msg/julia-users/6xEWQz_mK10/XrCkLV-7O_sJ
>
> This should probably go in a FAQ somewhere.
>
> On Mon, Nov 24, 2014 at 5:17 PM, Pileas  > wrote:
>
>> Is there a specific reason you guys chose this name?
>>
>> It kinda reminds me Linux Mint where each distro. has a girl's name ...
>>
>
>

Re: [julia-users] impressions and questions from a Matlab user

[Christian Peel]

What I was earlier calling a 'syntax error'  was actually an undefined 
function; I apologize for not using the correct language.  I believe that 
syntax errors are caught in the 'include' step, while the 'undefined' error 
I was actually getting was a compile-time error.   Starting with an 
undefined variable on the first line of function f in file.jl, here is what 
i did on the REPL 
* include file.jl   this takes a fraction of a second
* run f; after 11ish seconds it notices the undefined variable and returns 
an error
* run f again; it immediately complains about the undefined variable
The number of seconds it takes to compile (the first run of f) are related 
to the length of the file.

Tim,I guess the fact that I put the undefined function on the first 
line of the function I was trying to run resolves your question about 
performance (I am sure there are many things I can do to improve the 
performance of this toy function).

Elliot, running the command as you requested results in the following on 
one computer
julia> filter( x -> contains(x, "sys.dylib"), Sys.dllist())
1-element Array{String,1}:
 "/Applications/Julia-0.3.0.app/Contents/Resources/julia/lib/julia/sys.dylib"
I get similar results on a different computer which uses a homebrew package 
of Julia 0.3.2.  

Thanks for all your help!

Chris

On Monday, November 24, 2014 2:13:29 AM UTC-8, Tim Holy wrote:
>
> When you re-include your script, does it again take 11 seconds to get to 
> the 
> error? If so, something is wrong. How much computation is there in your 
> script? You might have a type inference problem, in which case performance 
> will be terrible, and it might simply be taking a long time (much longer 
> than 
> Matlab) to get to the point in your computation where the bug is 
> triggered. 
>
> If you haven't already, read at least up through & including the "Tools" 
> section of 
> http://docs.julialang.org/en/release-0.3/manual/performance-tips/ 
> You should definitely profile; that will tell you where the problem is. 
>
> --Tim 
>
> On Sunday, November 23, 2014 11:35:03 PM Christian Peel wrote: 
> > >What's your versioninfo? 
> > 
> > I used Version 0.3.0   (x86_64-apple-darwin13.3.0) on a 2013 macbook 
> which 
> > took about 9.6 seconds to include the function, try to run it, and find 
> the 
> > syntax error.   On a 2009 iMac with version 0.3.2 of Julia 
> > (x86_64-apple-darwin14.0.0) it took 11.3 seconds.Just to be clear, 
> it 
> > takes something like 3 seconds to load Julia, 26 seconds to load the 
> PyPlot 
> > package (?!?!?) and then an additional 11.3 seconds after that to get 
> the 
> > syntax error.   I do not restart Julia every time. 
> > 
> > The functions simulate a narrowband multi-antenna fading communications 
> > channel. To me it feels like a simple and straightforward script, but it 
> > may not be so simple for the optimizer. 
> > 
> > Chris 
> > 
> > On Sunday, November 23, 2014 7:21:36 PM UTC-8, Stefan Karpinski wrote: 
> > > Ah, yes. That would explain this if you're timing how long it takes to 
> > > start Julia from the command prompt. In that case, I can understand 
> the 
> > > complaint about the compile-debug-edit cycle, but you probably should 
> > > consider doing more development at the interactive REPL prompt rather 
> than 
> > > restarting Julia every time. 
> > > 
> > > On Sun, Nov 23, 2014 at 9:56 PM, Patrick O'Leary  > > 
> > > > wrote: 
> > >> On Sunday, November 23, 2014 7:55:33 PM UTC-6, Stefan Karpinski 
> wrote: 
> > >>> 11 seconds seems like an awfully long time. In the days of the slow 
> REPL 
> > >>> when Julia compiled itself upon starting up, that's about how long 
> it 
> > >>> took. 
> > >>> What's your versioninfo? 
> > >> 
> > >> Windows doesn't ship with sys.dll, for what it's worth. 
>
>

Re: [julia-users] Compose.jl Tutorials for working on Gadfly.jl

[Isaiah Norton]

There are some nice Compose examples in the "parallel prefix" notebook,
here: https://github.com/jiahao/ijulia-notebooks

http://nbviewer.ipython.org/github/jiahao/ijulia-notebooks/blob/master/2014-08-06-parallel-prefix.ipynb


On Sat, Nov 22, 2014 at 4:08 AM,  wrote:

> Are there any tutorials for Compose.jl other than the documentation page?
>
> I want to try to make pie charts and venn diagrams.
>
>

Re: [julia-users] [Offtopic:] Why you did choose the name "Julia"?

[Isaiah Norton]

>
> Alan Edelman told me specifically that it was not named after the fractal,
> and in fact that "Julia" doesn't refer to anything in particular.
> Apparently, it just came up in a random conversation years ago when someone
> suggested arbitrarily that "Julia" would be a good name for a programming
> language.
>

https://groups.google.com/d/msg/julia-users/6xEWQz_mK10/XrCkLV-7O_sJ

This should probably go in a FAQ somewhere.

On Mon, Nov 24, 2014 at 5:17 PM, Pileas  wrote:

> Is there a specific reason you guys chose this name?
>
> It kinda reminds me Linux Mint where each distro. has a girl's name ...
>

[julia-users] [Offtopic:] Why you did choose the name "Julia"?

[Pileas]

Is there a specific reason you guys chose this name?

It kinda reminds me Linux Mint where each distro. has a girl's name ...

[julia-users] Re: How to generate subplots with 2D and 3D plots?

[Markus Roth]

Thank you Daniel,

I just tried your suggestion. Seems to work fine. However the font sizes 
(of axes ticks) do not automatically turn smaller but have to be adjusted 
manually. I think I will manage to fix this ;).

Best,
Markus

Am Sonntag, 23. November 2014 21:27:22 UTC+1 schrieb Daniel Høegh:

> This works: 
> ax = fig[:add_subplot](2,1,1, projection = "3d") 
> xgrid = repmat(x',n,1) 
> ygrid = repmat(y,1,n) 
> ax[:plot_surface](xgrid, ygrid, z, rstride=2,edgecolors="k", cstride=2, 
> cmap=ColorMap("gray"), alpha=0.8, linewidth=0.25) 
> xlabel("X") 
> ylabel("Y") 
> subplot(212) 
> ax = fig[:add_subplot](2,1,2) 
> cp = ax[:contour](xgrid, ygrid, z, colors="black", linewidth=2.0) 
> ax[:clabel](cp, inline=1, fontsize=10) 
> xlabel("X") 
> ylabel("Y") 
>
> My greatest source of good examples is always: 
> http://matplotlib.org/gallery.html, and there is a good gist that shows 
> how to use PyPlot in julia: https://gist.github.com/gizmaa/7214002

Re: [julia-users] Security problem with unitialized memory

[Jameson Nash]

It appears the fill operation accounts for about 0.15 seconds of the 6.15
seconds that my OS X laptop takes to create this array:

$ ./julia -q

*julia> **N=10^9*

*10*


*julia> **@time begin x=zeros(Int64,N); fill(x,0) end*

elapsed time: 6.325660691 seconds (8000136616 bytes allocated, 1.71% gc
time)

*0-element Array{Array{Int64,1},1}*


$ ./julia -q

*julia> **N=10^9*

*10*


*julia> **@time x=zeros(Int64,N)*

elapsed time: 6.160623835 seconds (814320 bytes allocated, 0.22% gc
time)



On Mon Nov 24 2014 at 3:18:39 PM Erik Schnetter 
wrote:

> On Mon, Nov 24, 2014 at 3:01 PM, David Smith 
> wrote:
> > To add some data to this conversation, I just timed allocating a billion
> > Int64s on my macbook, and I got this (I ran these multiple times before
> this
> > and got similar timings):
> >
> > julia> N=1_000_000_000
> > 10
> >
> > julia> @time x = Array(Int64,N);
> > elapsed time: 0.022577671 seconds (800128 bytes allocated)
> >
> > julia> @time x = zeros(Int64,N);
> > elapsed time: 3.95432248 seconds (800152 bytes allocated)
> >
> > So we are talking adding possibly seconds to a program per large array
> > allocation.
>
> This is not quite right -- the first does not actually map the pages
> into memory; this is only done lazily when they are accessed the first
> time. You need to compare "alloc uninitialized; then initialize once"
> with "alloc zero-initialized; then initialize again".
>
> Current high-end system architectures have memory write speeds of ten
> or twenty GByte per second; this is what you should see for very large
> arrays -- this would be about 0.4 seconds for your case. For smaller
> arrays, the data would reside in the cache, so that the allocation
> overhead should be significantly smaller even.
>
> -erik
>
> --
> Erik Schnetter 
> http://www.perimeterinstitute.ca/personal/eschnetter/
>

[julia-users] Re: Julia v0.3.3

[Ivar Nesje]

We should also note that this is a recommended upgrade for anyone using any 
of the previous 0.3.x releases.

We have taken all reasonable measures to avoid breaking existing programs, 
so it should be pretty hassle free. We would like to get feedback if 
someone has a correctly working program that doesn't work after this 
upgrade.

Ivar


kl. 20:24:12 UTC+1 mandag 24. november 2014 skrev Elliot Saba følgende:
>
> Hello all!  The latest bugfix release of the 0.3.X Julia line has been 
> released.  Binaries are available from the usual place 
> , and as usual, please report all issues 
> to either the issue tracker , 
> or email this list.
>
> As this is a bugfix release, there are no new big-item features to 
> announce, but if you are interested in the bugs fixed since 0.3.2, this 
> commit log  
> should 
> give you an idea of the effort put in by our team of backporters.
>
> Happy Hacking,
> -E
>

Re: [julia-users] Security problem with unitialized memory

[Erik Schnetter]

On Mon, Nov 24, 2014 at 3:01 PM, David Smith  wrote:
> To add some data to this conversation, I just timed allocating a billion
> Int64s on my macbook, and I got this (I ran these multiple times before this
> and got similar timings):
>
> julia> N=1_000_000_000
> 10
>
> julia> @time x = Array(Int64,N);
> elapsed time: 0.022577671 seconds (800128 bytes allocated)
>
> julia> @time x = zeros(Int64,N);
> elapsed time: 3.95432248 seconds (800152 bytes allocated)
>
> So we are talking adding possibly seconds to a program per large array
> allocation.

This is not quite right -- the first does not actually map the pages
into memory; this is only done lazily when they are accessed the first
time. You need to compare "alloc uninitialized; then initialize once"
with "alloc zero-initialized; then initialize again".

Current high-end system architectures have memory write speeds of ten
or twenty GByte per second; this is what you should see for very large
arrays -- this would be about 0.4 seconds for your case. For smaller
arrays, the data would reside in the cache, so that the allocation
overhead should be significantly smaller even.

-erik

-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

Re: [julia-users] Security problem with unitialized memory

[Stefan Karpinski]

Using calloc could reduce this significantly since the kernel can lazily fill 
zero pages only when you access them. 


> On Nov 24, 2014, at 3:01 PM, David Smith  wrote:
> 
> To add some data to this conversation, I just timed allocating a billion 
> Int64s on my macbook, and I got this (I ran these multiple times before this 
> and got similar timings):
> 
> julia> N=1_000_000_000 
> 10 
> 
> julia> @time x = Array(Int64,N); 
> elapsed time: 0.022577671 seconds (800128 bytes allocated) 
> 
> julia> @time x = zeros(Int64,N); 
> elapsed time: 3.95432248 seconds (800152 bytes allocated) 
> 
> So we are talking adding possibly seconds to a program per large array 
> allocation.
> 
> 
> 

Re: [julia-users] Security problem with unitialized memory

[David Smith]

To add some data to this conversation, I just timed allocating a billion 
Int64s on my macbook, and I got this (I ran these multiple times before 
this and got similar timings):

julia> N=1_000_000_000 
10 

julia> @time x = Array(Int64,N); 
elapsed time: 0.022577671 seconds (800128 bytes allocated) 

julia> @time x = zeros(Int64,N); 
elapsed time: 3.95432248 seconds (800152 bytes allocated) 

So we are talking adding possibly seconds to a program per large array 
allocation.

Re: [julia-users] Security problem with unitialized memory

[Stefan Karpinski]

I guess part of the problem is that calling the `zeros` function may be less 
obvious as a way of constructing an array to many new programmers than calling 
the `Array` constructor. Having a Boolean fill keyword argument approach might 
be reasonable, although calling it `zeroed` might be more accurate since we 
won't be filling the array with a specific value but rather zeroing the memory 
first. Alternatively, we could just fill the array with random garbage 
intentionally so that programmers are made painfully aware that they didn't 
initialize the array ;-)

> On Nov 24, 2014, at 2:39 PM, Tomas Lycken  wrote:
> 
> That *is* the default usage in most introductory settings - just don't show 
> them the Array(T,n) constructor, but give them zeros and ones functions 
> instead. (It's perfectly fine to do e.g. A = zeros(10); fill!(A, 5) if you 
> don't care about the extra write...)
> 
> If there is a specific setting where the students actually *need* to allocate 
> uninitialized memory (e.g. for speed), they are probably ready to learn that 
> the Array constructor gives them that.
> 
> Julia's approach has so far seemed to be that the users are consenting 
> adults. I like that approach.
> 
> // T
> 
>> On Monday, November 24, 2014 8:30:10 PM UTC+1, Ronald L. Rivest wrote:
>> Regarding initialization:
>> 
>>-- I'm toying with the idea of recommending Julia for an introductory 
>> programming
>>   class (rather than Python).  
>> 
>>-- For this purpose, the language should not have hazards that catch the 
>> unwary.
>> 
>>-- Not initializing storage is definitely a hazard.  With uninitialized 
>> storage, a 
>>   program may run fine one day, and fail mysteriously the next, 
>> depending on 
>>   the contents of memory.  This is about predictability, reliability, 
>> dependability,
>>   and correctness.
>> 
>>-- I would favor a solution like
>>  A = Array(Int64,n)   -- fills with zeros
>>  A = Array(Int64,n,fill=1)  -- to fill with ones
>>  A = Array(Int64,n,fill=None)-- for an uninitialized array
>>so that the *default* is an initialized array, but the speed geeks
>>can get what they want.
>> 
>> Cheers,
>> Ron
>> 
>>> On Monday, November 24, 2014 1:57:14 PM UTC-5, Stefan Karpinski wrote:
>>> If we can make allocating zeroed arrays faster that's great, but unless we 
>>> can close the performance gap all the way and eliminate the need to 
>>> allocated uninitialized arrays altogether, this proposal is just a rename – 
>>> Unchecked.Array plays the exact same role as the current Array constructor. 
>>> It's unclear that this would even address the original concern since it 
>>> still *allows* uninitialized allocation of arrays. This rename would just 
>>> force people who have used Array correctly in code that cares about being 
>>> as efficient as possible even for very large arrays to change their code 
>>> and use Unchecked.Array instead.
>>> 
 On Nov 24, 2014, at 1:36 PM, Jameson Nash  wrote:
 
 I think that Rivest’s question may be a good reason to rethink the 
 initialization of structs and offer the explicit guarantee that all 
 unassigned elements will be initialized to 0 (and not just the jl_value_t 
 pointers). I would argue that the current behavior resulted more from a 
 desire to avoid clearing the array twice (if the user is about to call 
 fill, zeros, ones, +, etc.) than an intentional, casual exposure of 
 uninitialized memory.
 
 A random array of integers is also a security concern if an attacker can 
 extract some other information (with some probability) about the state of 
 the program. Julia is not hardened by design, so you can’t safely run an 
 unknown code fragment, but you still might have an unintended memory 
 exposure in a client-facing app. While zero’ing memory doesn’t prevent the 
 user from simply reusing a memory buffer in a security-unaware fashion 
 (rather than consistently allocating a new one for each use), it’s not 
 clear to me that the performance penalty would be all that noticeable for 
 map Array(X) to zero(X), and only providing an internal constructor for 
 grabbing uninitialized memory (perhaps Base.Unchecked.Array(X) from #8227)
 
 On Mon Nov 24 2014 at 12:57:22 PM Stefan Karpinski 
 stefan.karpin...@gmail.com wrote:
 
> There are two rather different issues to consider:
> 
> 1. Preventing problems due to inadvertent programmer errors.
> 2. Preventing malicious security attacks.
> 
> When we initially made this choice, it wasn't clear if 1 would be a big 
> issue but we decided to see how it played out. It hasn't been a problem 
> in practice: once people grok that the Array(T, dims...) constructor 
> gives uninitialized memory and that the standard usage pattern is to call 
> it and then immediately initialize 

Re: [julia-users] Security problem with unitialized memory

[Tomas Lycken]

That *is* the default usage in most introductory settings - just don't show 
them the Array(T,n) constructor, but give them zeros and ones functions 
instead. (It's perfectly fine to do e.g. A = zeros(10); fill!(A, 5) if you 
don't care about the extra write...)

If there is a specific setting where the students actually *need* to 
allocate uninitialized memory (e.g. for speed), they are probably ready to 
learn that the Array constructor gives them that.

Julia's approach has so far seemed to be that the users are consenting 
adults. I like that approach.

// T

On Monday, November 24, 2014 8:30:10 PM UTC+1, Ronald L. Rivest wrote:
>
> Regarding initialization:
>
>-- I'm toying with the idea of recommending Julia for an introductory 
> programming
>   class (rather than Python).  
>
>-- For this purpose, the language should not have hazards that catch 
> the unwary.
>
>-- Not initializing storage is definitely a hazard.  With uninitialized 
> storage, a 
>   program may run fine one day, and fail mysteriously the next, 
> depending on 
>   the contents of memory.  This is about predictability, reliability, 
> dependability,
>   and correctness.
>
>-- I would favor a solution like
>  A = Array(Int64,n)   -- fills with zeros
>  A = Array(Int64,n,fill=1)  -- to fill with ones
>  A = Array(Int64,n,fill=None)-- for an uninitialized array
>so that the *default* is an initialized array, but the speed geeks
>can get what they want.
>
> Cheers,
> Ron
>
> On Monday, November 24, 2014 1:57:14 PM UTC-5, Stefan Karpinski wrote:
>>
>> If we can make allocating zeroed arrays faster that's great, but unless 
>> we can close the performance gap all the way and eliminate the need to 
>> allocated uninitialized arrays altogether, this proposal is just a rename – 
>> Unchecked.Array 
>> plays the exact same role as the current Array constructor. It's unclear 
>> that this would even address the original concern since it still *allows* 
>> uninitialized allocation of arrays. This rename would just force people who 
>> have used Array correctly in code that cares about being as efficient as 
>> possible even for very large arrays to change their code and use 
>> Unchecked.Array instead.
>>
>> On Nov 24, 2014, at 1:36 PM, Jameson Nash  wrote:
>>
>> I think that Rivest’s question may be a good reason to rethink the 
>> initialization of structs and offer the explicit guarantee that all 
>> unassigned elements will be initialized to 0 (and not just the jl_value_t 
>> pointers). I would argue that the current behavior resulted more from a 
>> desire to avoid clearing the array twice (if the user is about to call 
>> fill, zeros, ones, +, etc.) than an intentional, casual exposure of 
>> uninitialized memory.
>>
>> A random array of integers is also a security concern if an attacker can 
>> extract some other information (with some probability) about the state of 
>> the program. Julia is not hardened by design, so you can’t safely run an 
>> unknown code fragment, but you still might have an unintended memory 
>> exposure in a client-facing app. While zero’ing memory doesn’t prevent the 
>> user from simply reusing a memory buffer in a security-unaware fashion 
>> (rather than consistently allocating a new one for each use), it’s not 
>> clear to me that the performance penalty would be all that noticeable for 
>> map Array(X) to zero(X), and only providing an internal constructor for 
>> grabbing uninitialized memory (perhaps Base.Unchecked.Array(X) from 
>> #8227)
>>
>> On Mon Nov 24 2014 at 12:57:22 PM Stefan Karpinski 
>> stefan.karpin...@gmail.com  
>> wrote:
>>
>> There are two rather different issues to consider:
>>>
>>> 1. Preventing problems due to inadvertent programmer errors.
>>> 2. Preventing malicious security attacks.
>>>
>>> When we initially made this choice, it wasn't clear if 1 would be a big 
>>> issue but we decided to see how it played out. It hasn't been a problem in 
>>> practice: once people grok that the Array(T, dims...) constructor gives 
>>> uninitialized memory and that the standard usage pattern is to call it and 
>>> then immediately initialize the memory, everything is ok. I can't 
>>> recall a single situation where someone has had some terrible bug due to 
>>> uninitialized int/float arrays.
>>>
>>> Regarding 2, Julia is not intended to be a hardened language for writing 
>>> highly secure software. It allows all sorts of unsafe actions: pointer 
>>> arithmetic, direct memory access, calling arbitrary C functions, etc. The 
>>> future of really secure software seems to be small formally verified 
>>> kernels written in statically typed languages that communicate with larger 
>>> unverified systems over restricted channels. Julia might be appropriate for 
>>> the larger unverified system but certainly not for the trusted kernel. 
>>> Adding enoug

Re: [julia-users] Security problem with unitialized memory

[Ronald L. Rivest]

Regarding initialization:

   -- I'm toying with the idea of recommending Julia for an introductory 
programming
  class (rather than Python).  

   -- For this purpose, the language should not have hazards that catch the 
unwary.

   -- Not initializing storage is definitely a hazard.  With uninitialized 
storage, a 
  program may run fine one day, and fail mysteriously the next, 
depending on 
  the contents of memory.  This is about predictability, reliability, 
dependability,
  and correctness.

   -- I would favor a solution like
 A = Array(Int64,n)   -- fills with zeros
 A = Array(Int64,n,fill=1)  -- to fill with ones
 A = Array(Int64,n,fill=None)-- for an uninitialized array
   so that the *default* is an initialized array, but the speed geeks
   can get what they want.

Cheers,
Ron

On Monday, November 24, 2014 1:57:14 PM UTC-5, Stefan Karpinski wrote:
>
> If we can make allocating zeroed arrays faster that's great, but unless we 
> can close the performance gap all the way and eliminate the need to 
> allocated uninitialized arrays altogether, this proposal is just a rename – 
> Unchecked.Array 
> plays the exact same role as the current Array constructor. It's unclear 
> that this would even address the original concern since it still *allows* 
> uninitialized allocation of arrays. This rename would just force people who 
> have used Array correctly in code that cares about being as efficient as 
> possible even for very large arrays to change their code and use 
> Unchecked.Array instead.
>
> On Nov 24, 2014, at 1:36 PM, Jameson Nash > 
> wrote:
>
> I think that Rivest’s question may be a good reason to rethink the 
> initialization of structs and offer the explicit guarantee that all 
> unassigned elements will be initialized to 0 (and not just the jl_value_t 
> pointers). I would argue that the current behavior resulted more from a 
> desire to avoid clearing the array twice (if the user is about to call 
> fill, zeros, ones, +, etc.) than an intentional, casual exposure of 
> uninitialized memory.
>
> A random array of integers is also a security concern if an attacker can 
> extract some other information (with some probability) about the state of 
> the program. Julia is not hardened by design, so you can’t safely run an 
> unknown code fragment, but you still might have an unintended memory 
> exposure in a client-facing app. While zero’ing memory doesn’t prevent the 
> user from simply reusing a memory buffer in a security-unaware fashion 
> (rather than consistently allocating a new one for each use), it’s not 
> clear to me that the performance penalty would be all that noticeable for 
> map Array(X) to zero(X), and only providing an internal constructor for 
> grabbing uninitialized memory (perhaps Base.Unchecked.Array(X) from #8227)
>
> On Mon Nov 24 2014 at 12:57:22 PM Stefan Karpinski 
> stefan.karpin...@gmail.com  
> wrote:
>
> There are two rather different issues to consider:
>>
>> 1. Preventing problems due to inadvertent programmer errors.
>> 2. Preventing malicious security attacks.
>>
>> When we initially made this choice, it wasn't clear if 1 would be a big 
>> issue but we decided to see how it played out. It hasn't been a problem in 
>> practice: once people grok that the Array(T, dims...) constructor gives 
>> uninitialized memory and that the standard usage pattern is to call it and 
>> then immediately initialize the memory, everything is ok. I can't recall 
>> a single situation where someone has had some terrible bug due to 
>> uninitialized int/float arrays.
>>
>> Regarding 2, Julia is not intended to be a hardened language for writing 
>> highly secure software. It allows all sorts of unsafe actions: pointer 
>> arithmetic, direct memory access, calling arbitrary C functions, etc. The 
>> future of really secure software seems to be small formally verified 
>> kernels written in statically typed languages that communicate with larger 
>> unverified systems over restricted channels. Julia might be appropriate for 
>> the larger unverified system but certainly not for the trusted kernel. 
>> Adding enough verification to Julia to write secure kernels is not 
>> inconceivable, but would be a major research effort. The implementation 
>> would have to check lots of things, including, of course, ensuring that all 
>> arrays are initialized.
>>
>> A couple of other points:
>>
>> Modern OSes protect against data leaking between processes by zeroing 
>> pages before a process first accesses them. Thus any data exposed by 
>> Array(T, dims...) comes from the same process and is not a security leak.
>>
>> An uninitialized array of, say, integers is not in itself a security 
>> concern – the issue is what you do with those integers. The classic 
>> security hole is to use a "random" value from uninitialized memory to 
>> access other memory by using

[julia-users] Julia v0.3.3

[Elliot Saba]

Hello all!  The latest bugfix release of the 0.3.X Julia line has been
released.  Binaries are available from the usual place
, and as usual, please report all issues
to either the issue tracker , or
email this list.

As this is a bugfix release, there are no new big-item features to
announce, but if you are interested in the bugs fixed since 0.3.2, this
commit log  should
give you an idea of the effort put in by our team of backporters.

Happy Hacking,
-E

[julia-users] Date not working in ipython notebook but works on julia REPL

[Ajinkya Kale]

using Dates
dateReported = map((x) -> string(x), df[:DateReported])
df[:DateOccurred] = map((x) -> if 
match(r"^((19|20)\d\d)(0[1-9]|1[012])(0[1-9]|[12][0-9]|3[01])", x)!=nothing 
Date(x, DateFormat("mmdd")) end, dateOccurred)

I am trying to change type of a dataframe column to Date from Int64.

The last statement returns an error

Date not defined
while loading In[18], in expression starting on line 1

 in anonymous at In[18]:1
 in map at /Users/ajkale/.julia/v0.3/DataArrays/src/datavector.jl:117

I am trying this in the ipython julia notebook. This works fine in the REPL 
though.

Asked this on stackoverflow as 
well 
http://stackoverflow.com/questions/27094669/date-not-working-in-ipython-notebook-but-works-on-julia-repl/27112355#27112355

Re: [julia-users] Security problem with unitialized memory

[Stefan Karpinski]

If we can make allocating zeroed arrays faster that's great, but unless we can 
close the performance gap all the way and eliminate the need to allocated 
uninitialized arrays altogether, this proposal is just a rename – 
Unchecked.Array plays the exact same role as the current Array constructor. 
It's unclear that this would even address the original concern since it still 
*allows* uninitialized allocation of arrays. This rename would just force 
people who have used Array correctly in code that cares about being as 
efficient as possible even for very large arrays to change their code and use 
Unchecked.Array instead.

> On Nov 24, 2014, at 1:36 PM, Jameson Nash  wrote:
> 
> I think that Rivest’s question may be a good reason to rethink the 
> initialization of structs and offer the explicit guarantee that all 
> unassigned elements will be initialized to 0 (and not just the jl_value_t 
> pointers). I would argue that the current behavior resulted more from a 
> desire to avoid clearing the array twice (if the user is about to call fill, 
> zeros, ones, +, etc.) than an intentional, casual exposure of uninitialized 
> memory.
> 
> A random array of integers is also a security concern if an attacker can 
> extract some other information (with some probability) about the state of the 
> program. Julia is not hardened by design, so you can’t safely run an unknown 
> code fragment, but you still might have an unintended memory exposure in a 
> client-facing app. While zero’ing memory doesn’t prevent the user from simply 
> reusing a memory buffer in a security-unaware fashion (rather than 
> consistently allocating a new one for each use), it’s not clear to me that 
> the performance penalty would be all that noticeable for map Array(X) to 
> zero(X), and only providing an internal constructor for grabbing 
> uninitialized memory (perhaps Base.Unchecked.Array(X) from #8227)
> 
> On Mon Nov 24 2014 at 12:57:22 PM Stefan Karpinski stefan.karpin...@gmail.com 
> wrote:
> 
>> There are two rather different issues to consider:
>> 
>> 1. Preventing problems due to inadvertent programmer errors.
>> 2. Preventing malicious security attacks.
>> 
>> When we initially made this choice, it wasn't clear if 1 would be a big 
>> issue but we decided to see how it played out. It hasn't been a problem in 
>> practice: once people grok that the Array(T, dims...) constructor gives 
>> uninitialized memory and that the standard usage pattern is to call it and 
>> then immediately initialize the memory, everything is ok. I can't recall a 
>> single situation where someone has had some terrible bug due to 
>> uninitialized int/float arrays.
>> 
>> Regarding 2, Julia is not intended to be a hardened language for writing 
>> highly secure software. It allows all sorts of unsafe actions: pointer 
>> arithmetic, direct memory access, calling arbitrary C functions, etc. The 
>> future of really secure software seems to be small formally verified kernels 
>> written in statically typed languages that communicate with larger 
>> unverified systems over restricted channels. Julia might be appropriate for 
>> the larger unverified system but certainly not for the trusted kernel. 
>> Adding enough verification to Julia to write secure kernels is not 
>> inconceivable, but would be a major research effort. The implementation 
>> would have to check lots of things, including, of course, ensuring that all 
>> arrays are initialized.
>> 
>> A couple of other points:
>> 
>> Modern OSes protect against data leaking between processes by zeroing pages 
>> before a process first accesses them. Thus any data exposed by Array(T, 
>> dims...) comes from the same process and is not a security leak.
>> 
>> An uninitialized array of, say, integers is not in itself a security concern 
>> – the issue is what you do with those integers. The classic security hole is 
>> to use a "random" value from uninitialized memory to access other memory by 
>> using it to index into an array or otherwise convert it to a pointer. In the 
>> presence of bounds checking, however, this isn't actually a big concern 
>> since you will still either get a bounds error or a valid array value – not 
>> a meaningful one, of course, but still just a value.
>> 
>> Writing programs that are secure against malicious attacks is a hard, 
>> unsolved problem. So is doing efficient, productive high-level numerical 
>> programming. Trying to solve both problems at the same time seems like a 
>> recipe for failing at both.
>> 
>>> On Nov 24, 2014, at 11:43 AM, David Smith  wrote:
>>> 
>>> Some ideas:
>>> 
>>> Is there a way to return an error for accesses before at least one 
>>> assignment in bits types?  I.e. when the object is created uninitialized it 
>>> is marked "dirty" and only after assignment of some user values can it be 
>>> "cleanly" accessed?
>>> 
>>> Can Julia provide a thin memory management layer that grabs memory from the 
>>> OS first, zeroes it, and then gives it to the 

Re: [julia-users] Security problem with unitialized memory

[Jameson Nash]

I think that Rivest’s question may be a good reason to rethink the
initialization of structs and offer the explicit guarantee that all
unassigned elements will be initialized to 0 (and not just the jl_value_t
pointers). I would argue that the current behavior resulted more from a
desire to avoid clearing the array twice (if the user is about to call
fill, zeros, ones, +, etc.) than an intentional, casual exposure of
uninitialized memory.

A random array of integers is also a security concern if an attacker can
extract some other information (with some probability) about the state of
the program. Julia is not hardened by design, so you can’t safely run an
unknown code fragment, but you still might have an unintended memory
exposure in a client-facing app. While zero’ing memory doesn’t prevent the
user from simply reusing a memory buffer in a security-unaware fashion
(rather than consistently allocating a new one for each use), it’s not
clear to me that the performance penalty would be all that noticeable for
map Array(X) to zero(X), and only providing an internal constructor for
grabbing uninitialized memory (perhaps Base.Unchecked.Array(X) from #8227)

On Mon Nov 24 2014 at 12:57:22 PM Stefan Karpinski
stefan.karpin...@gmail.com  wrote:

There are two rather different issues to consider:
>
> 1. Preventing problems due to inadvertent programmer errors.
> 2. Preventing malicious security attacks.
>
> When we initially made this choice, it wasn't clear if 1 would be a big
> issue but we decided to see how it played out. It hasn't been a problem in
> practice: once people grok that the Array(T, dims...) constructor gives
> uninitialized memory and that the standard usage pattern is to call it and
> then immediately initialize the memory, everything is ok. I can't recall
> a single situation where someone has had some terrible bug due to
> uninitialized int/float arrays.
>
> Regarding 2, Julia is not intended to be a hardened language for writing
> highly secure software. It allows all sorts of unsafe actions: pointer
> arithmetic, direct memory access, calling arbitrary C functions, etc. The
> future of really secure software seems to be small formally verified
> kernels written in statically typed languages that communicate with larger
> unverified systems over restricted channels. Julia might be appropriate for
> the larger unverified system but certainly not for the trusted kernel.
> Adding enough verification to Julia to write secure kernels is not
> inconceivable, but would be a major research effort. The implementation
> would have to check lots of things, including, of course, ensuring that all
> arrays are initialized.
>
> A couple of other points:
>
> Modern OSes protect against data leaking between processes by zeroing
> pages before a process first accesses them. Thus any data exposed by
> Array(T, dims...) comes from the same process and is not a security leak.
>
> An uninitialized array of, say, integers is not in itself a security
> concern – the issue is what you do with those integers. The classic
> security hole is to use a "random" value from uninitialized memory to
> access other memory by using it to index into an array or otherwise convert
> it to a pointer. In the presence of bounds checking, however, this isn't
> actually a big concern since you will still either get a bounds error or a
> valid array value – not a meaningful one, of course, but still just a value.
>
> Writing programs that are secure against malicious attacks is a hard,
> unsolved problem. So is doing efficient, productive high-level numerical
> programming. Trying to solve both problems at the same time seems like a
> recipe for failing at both.
>
> On Nov 24, 2014, at 11:43 AM, David Smith  wrote:
>
> Some ideas:
>
> Is there a way to return an error for accesses before at least one
> assignment in bits types?  I.e. when the object is created uninitialized it
> is marked "dirty" and only after assignment of some user values can it be
> "cleanly" accessed?
>
> Can Julia provide a thin memory management layer that grabs memory from
> the OS first, zeroes it, and then gives it to the user upon initial
> allocation?  After gc+reallocation it doesn't need to be zeroed again,
> unless the next allocation is larger than anything previous, at which time
> Julia grabs more memory, sanitizes it, and hands it off.
>
> On Monday, November 24, 2014 2:48:05 AM UTC-6, Mauro wrote:
>>
>> Pointer types will initialise to undef and any operation on them fails:
>> julia> a = Array(ASCIIString, 5);
>>
>> julia> a[1]
>> ERROR: access to undefined reference
>>  in getindex at array.jl:246
>>
>> But you're right, for bits-types this is not an error an will just
>> return whatever was there before.  I think the reason this will stay
>> that way is that Julia is a numerics oriented language.  Thus you many
>> wanna create a 1GB array of Float64 and then fill it with something as
>> opposed to first fil

Re: [julia-users] Broadcasting variables

[Blake Johnson]

I use this macro to send variables to remote processes:

macro sendvar(proc, x)
quote
rr = RemoteRef()
put!(rr, $x)
remotecall($proc, (rr)->begin
global $(esc(x))
$(esc(x)) = fetch(rr)
end, rr)
end
end

Though the solution above looks a little simpler.

--Blake

On Sunday, November 23, 2014 1:30:49 AM UTC-5, Amit Murthy wrote:
>
> From the description of Base.localize_vars - 'wrap an expression in "let 
> a=a,b=b,..." for each var it references'
>
> Though that does not seem to the only(?) issue here 
>
> On Sun, Nov 23, 2014 at 11:52 AM, Madeleine Udell  > wrote:
>
>> Thanks! This is extremely helpful. 
>>
>> Can you tell me more about what localize_vars does?
>>
>> On Sat, Nov 22, 2014 at 9:11 PM, Amit Murthy > > wrote:
>>
>>> This works:
>>>
>>> function doparallelstuff(m = 10, n = 20)
>>> # initialize variables
>>> localX = Base.shmem_rand(m; pids=procs())
>>> localY = Base.shmem_rand(n; pids=procs())
>>> localf = [x->i+sum(x) for i=1:m]
>>> localg = [x->i+sum(x) for i=1:n]
>>>
>>> # broadcast variables to all worker processes
>>> @sync begin
>>> for i in procs(localX)
>>> remotecall(i, x->(global X; X=x; nothing), localX)
>>> remotecall(i, x->(global Y; Y=x; nothing), localY)
>>> remotecall(i, x->(global f; f=x; nothing), localf)
>>> remotecall(i, x->(global g; g=x; nothing), localg)
>>> end
>>> end
>>>
>>> # compute
>>> for iteration=1:1
>>> @everywhere for i=localindexes(X)
>>> X[i] = f[i](Y)
>>> end
>>> @everywhere for j=localindexes(Y)
>>> Y[j] = g[j](X)
>>> end
>>> end
>>> end
>>>
>>> doparallelstuff()
>>>
>>> Though I would have expected broadcast of variables to be possible with 
>>> just 
>>> @everywhere X=localX
>>> and so on 
>>>
>>>
>>> Looks like @everywhere does not call localize_vars.  I don't know if 
>>> this is by design or just an oversight. I would have expected it to do so. 
>>> Will file an issue on github.
>>>
>>>
>>>
>>> On Sun, Nov 23, 2014 at 8:24 AM, Madeleine Udell >> > wrote:
>>>
 The code block I posted before works, but throws an error when embedded 
 in a function: "ERROR: X not defined" (in first line of @parallel). Why am 
 I getting this error when I'm *assigning to* X?

 function doparallelstuff(m = 10, n = 20)
 # initialize variables
 localX = Base.shmem_rand(m)
 localY = Base.shmem_rand(n)
 localf = [x->i+sum(x) for i=1:m]
 localg = [x->i+sum(x) for i=1:n]

 # broadcast variables to all worker processes
 @parallel for i=workers()
 global X = localX
 global Y = localY
 global f = localf
 global g = localg
 end
 # give variables same name on master
 X,Y,f,g = localX,localY,localf,localg

 # compute
 for iteration=1:1
 @everywhere for i=localindexes(X)
 X[i] = f[i](Y)
 end
 @everywhere for j=localindexes(Y)
 Y[j] = g[j](X)
 end
 end
 end

 doparallelstuff()

 On Fri, Nov 21, 2014 at 5:13 PM, Madeleine Udell >>> > wrote:

> My experiments with parallelism also occur in focused blocks; I think 
> that's a sign that it's not yet as user friendly as it could be.
>
> Here's a solution to the problem I posed that's simple to use: 
> @parallel + global can be used to broadcast a variable, while @everywhere 
> can be used to do a computation on local data (ie, without resending the 
> data). I'm not sure how to do the variable renaming programmatically, 
> though.
>
> # initialize variables
> m,n = 10,20
> localX = Base.shmem_rand(m)
> localY = Base.shmem_rand(n)
> localf = [x->i+sum(x) for i=1:m]
> localg = [x->i+sum(x) for i=1:n]
>
> # broadcast variables to all worker processes
> @parallel for i=workers()
> global X = localX
> global Y = localY
> global f = localf
> global g = localg
> end
> # give variables same name on master
> X,Y,f,g = localX,localY,localf,localg
>
> # compute
> for iteration=1:10
> @everywhere for i=localindexes(X)
> X[i] = f[i](Y)
> end
> @everywhere for j=localindexes(Y)
> Y[j] = g[j](X)
> end
> end
>
> On Fri, Nov 21, 2014 at 11:14 AM, Tim Holy  > wrote:
>
>> My experiments with parallelism tend to occur in focused blocks, and 
>> I haven't
>> done it in quite a while. So I doubt I can help much. But in general 
>> I suspect
>> you're encountering these problems because much of the IPC goes 
>> through
>> thunks, and so a lot of stuff gets reclaimed when execution is done.
>>
>> If I were experimenting,

Re: [julia-users] Security problem with unitialized memory

[Stefan Karpinski]

There are two rather different issues to consider:

1. Preventing problems due to inadvertent programmer errors.
2. Preventing malicious security attacks.

When we initially made this choice, it wasn't clear if 1 would be a big issue 
but we decided to see how it played out. It hasn't been a problem in practice: 
once people grok that the Array(T, dims...) constructor gives uninitialized 
memory and that the standard usage pattern is to call it and then immediately 
initialize the memory, everything is ok. I can't recall a single situation 
where someone has had some terrible bug due to uninitialized int/float arrays.

Regarding 2, Julia is not intended to be a hardened language for writing highly 
secure software. It allows all sorts of unsafe actions: pointer arithmetic, 
direct memory access, calling arbitrary C functions, etc. The future of really 
secure software seems to be small formally verified kernels written in 
statically typed languages that communicate with larger unverified systems over 
restricted channels. Julia might be appropriate for the larger unverified 
system but certainly not for the trusted kernel. Adding enough verification to 
Julia to write secure kernels is not inconceivable, but would be a major 
research effort. The implementation would have to check lots of things, 
including, of course, ensuring that all arrays are initialized.

A couple of other points:

Modern OSes protect against data leaking between processes by zeroing pages 
before a process first accesses them. Thus any data exposed by Array(T, 
dims...) comes from the same process and is not a security leak.

An uninitialized array of, say, integers is not in itself a security concern – 
the issue is what you do with those integers. The classic security hole is to 
use a "random" value from uninitialized memory to access other memory by using 
it to index into an array or otherwise convert it to a pointer. In the presence 
of bounds checking, however, this isn't actually a big concern since you will 
still either get a bounds error or a valid array value – not a meaningful one, 
of course, but still just a value.

Writing programs that are secure against malicious attacks is a hard, unsolved 
problem. So is doing efficient, productive high-level numerical programming. 
Trying to solve both problems at the same time seems like a recipe for failing 
at both.

> On Nov 24, 2014, at 11:43 AM, David Smith  wrote:
> 
> Some ideas:
> 
> Is there a way to return an error for accesses before at least one assignment 
> in bits types?  I.e. when the object is created uninitialized it is marked 
> "dirty" and only after assignment of some user values can it be "cleanly" 
> accessed?
> 
> Can Julia provide a thin memory management layer that grabs memory from the 
> OS first, zeroes it, and then gives it to the user upon initial allocation?  
> After gc+reallocation it doesn't need to be zeroed again, unless the next 
> allocation is larger than anything previous, at which time Julia grabs more 
> memory, sanitizes it, and hands it off. 
> 
>> On Monday, November 24, 2014 2:48:05 AM UTC-6, Mauro wrote:
>> Pointer types will initialise to undef and any operation on them fails: 
>> julia> a = Array(ASCIIString, 5); 
>> 
>> julia> a[1] 
>> ERROR: access to undefined reference 
>>  in getindex at array.jl:246 
>> 
>> But you're right, for bits-types this is not an error an will just 
>> return whatever was there before.  I think the reason this will stay 
>> that way is that Julia is a numerics oriented language.  Thus you many 
>> wanna create a 1GB array of Float64 and then fill it with something as 
>> opposed to first fill it with zeros and then fill it with something. 
>> See: 
>> 
>> julia> @time b = Array(Float64, 10^9); 
>> elapsed time: 0.029523638 seconds (800144 bytes allocated) 
>> 
>> julia> @time c = zeros(Float64, 10^9); 
>> elapsed time: 0.835062841 seconds (800168 bytes allocated) 
>> 
>> You can argue that the time gain isn't worth the risk but I suspect that 
>> others may feel different. 
>> 
>> On Mon, 2014-11-24 at 09:28, Ronald L. Rivest  wrote: 
>> > I am just learning Julia... 
>> > 
>> > I was quite shocked today to learn that Julia does *not* 
>> > initialize allocated storage (e.g. to 0 or some default value). 
>> > E.g. the code 
>> >  A = Array(Int64,5) 
>> >  println(A[1]) 
>> > has unpredictable behavior, may disclose information from 
>> > other modules, etc. 
>> > 
>> > This is really quite unacceptable in a modern programming 
>> > language; it is as bad as not checking array reads for out-of-bounds 
>> > indices.   
>> > 
>> > Google for "uninitialized security" to find numerous instances 
>> > of security violations and unreliability problems caused by the 
>> > use of uninitialized variables, and numerous security advisories 
>> > warning of problems caused by the (perhaps inadvertent) use 
>> > of uninitialized variables. 
>> > 
>> > You can't design a programming language toda

[julia-users] Re: Solving a Bellman equation with Julia through function iteration: what I do wrong?

[Pileas]

Indeed I have the Matlab code in mind that I want to make it work in Julia.

Re: [julia-users] Security problem with unitialized memory

[David Smith]

Some ideas:

Is there a way to return an error for accesses before at least one 
assignment in bits types?  I.e. when the object is created uninitialized it 
is marked "dirty" and only after assignment of some user values can it be 
"cleanly" accessed?

Can Julia provide a thin memory management layer that grabs memory from the 
OS first, zeroes it, and then gives it to the user upon initial allocation? 
 After gc+reallocation it doesn't need to be zeroed again, unless the next 
allocation is larger than anything previous, at which time Julia grabs more 
memory, sanitizes it, and hands it off. 

On Monday, November 24, 2014 2:48:05 AM UTC-6, Mauro wrote:
>
> Pointer types will initialise to undef and any operation on them fails: 
> julia> a = Array(ASCIIString, 5); 
>
> julia> a[1] 
> ERROR: access to undefined reference 
>  in getindex at array.jl:246 
>
> But you're right, for bits-types this is not an error an will just 
> return whatever was there before.  I think the reason this will stay 
> that way is that Julia is a numerics oriented language.  Thus you many 
> wanna create a 1GB array of Float64 and then fill it with something as 
> opposed to first fill it with zeros and then fill it with something. 
> See: 
>
> julia> @time b = Array(Float64, 10^9); 
> elapsed time: 0.029523638 seconds (800144 bytes allocated) 
>
> julia> @time c = zeros(Float64, 10^9); 
> elapsed time: 0.835062841 seconds (800168 bytes allocated) 
>
> You can argue that the time gain isn't worth the risk but I suspect that 
> others may feel different. 
>
> On Mon, 2014-11-24 at 09:28, Ronald L. Rivest  > wrote: 
> > I am just learning Julia... 
> > 
> > I was quite shocked today to learn that Julia does *not* 
> > initialize allocated storage (e.g. to 0 or some default value). 
> > E.g. the code 
> >  A = Array(Int64,5) 
> >  println(A[1]) 
> > has unpredictable behavior, may disclose information from 
> > other modules, etc. 
> > 
> > This is really quite unacceptable in a modern programming 
> > language; it is as bad as not checking array reads for out-of-bounds 
> > indices.   
> > 
> > Google for "uninitialized security" to find numerous instances 
> > of security violations and unreliability problems caused by the 
> > use of uninitialized variables, and numerous security advisories 
> > warning of problems caused by the (perhaps inadvertent) use 
> > of uninitialized variables. 
> > 
> > You can't design a programming language today under the naive 
> > assumption that code in that language won't be used in highly 
> > critical applications or won't be under adversarial attack. 
> > 
> > You can't reasonably ask all programmers to properly initialize 
> > their allocated storage manually any more than you can ask them 
> > to test all indices before accessing an array manually; these are 
> > things that a high-level language should do for you. 
> > 
> > The default non-initialization of allocated storage is a 
> > mis-feature that should absolutely be fixed. 
> > 
> > There is no efficiency argument here in favor of uninitialized storage 
> > that can outweigh the security and reliability disadvantages... 
> > 
> > Cheers, 
> > Ron Rivest 
>
>

Re: [julia-users] Re: C/MPI/SLURM => Julia/?/SLURM?

[Gabriel Mihalache]

On Mon Nov 24 2014 at 8:37:36 AM Erik Schnetter 
wrote:

> To use MPI from Julia, use the MPI.jl package
>  (`Pkg.add("MPI")`). There are also
> examples in this package.
>

That's great! Somehow I missed the examples when I first saw that package.
If I can't get pmap to work nice across machines, this should be at least
as good.

Thank you for your work on this!

[julia-users] Re: Compose.jl Tutorials for working on Gadfly.jl

[Jason Knight]

For Venn diagrams there is https://github.com/HarlanH/VennEuler.jl

And I started a project with a simpler API 
at https://github.com/binarybana/VennDiagrams.jl (currently a skeleton) 
which is related to the above 
by https://github.com/HarlanH/VennEuler.jl/issues/14

On Saturday, November 22, 2014 3:08:41 AM UTC-6, ccsv.1...@gmail.com wrote:
>
> Are there any tutorials for Compose.jl other than the documentation page?
>
> I want to try to make pie charts and venn diagrams.
>
>

[julia-users] Re: Security problem with unitialized memory

[Alan Edelman]

I would have gone
 A = zeros(Int64,5)
Alan



On Monday, November 24, 2014 3:28:50 AM UTC-5, Ronald L. Rivest wrote:
>
> I am just learning Julia...
>
> I was quite shocked today to learn that Julia does *not*
> initialize allocated storage (e.g. to 0 or some default value).
> E.g. the code
>  A = Array(Int64,5)
>  println(A[1])
> has unpredictable behavior, may disclose information from
> other modules, etc.
>
> This is really quite unacceptable in a modern programming
> language; it is as bad as not checking array reads for out-of-bounds
> indices.  
>
> Google for "uninitialized security" to find numerous instances
> of security violations and unreliability problems caused by the
> use of uninitialized variables, and numerous security advisories
> warning of problems caused by the (perhaps inadvertent) use
> of uninitialized variables.
>
> You can't design a programming language today under the naive
> assumption that code in that language won't be used in highly
> critical applications or won't be under adversarial attack.
>
> You can't reasonably ask all programmers to properly initialize 
> their allocated storage manually any more than you can ask them 
> to test all indices before accessing an array manually; these are
> things that a high-level language should do for you. 
>
> The default non-initialization of allocated storage is a 
> mis-feature that should absolutely be fixed.
>
> There is no efficiency argument here in favor of uninitialized storage
> that can outweigh the security and reliability disadvantages...
>
> Cheers,
> Ron Rivest
>
>

[julia-users] Re: how to do array (or collection) of functions

[Stefan Schwarz]

@Erno If you want to use the list comprehension and not result-type 
Array{Any, 1}, but
Array{Int64, 1} is to define f this way:

f(x)=Int[v[i](x) for i=1:length(v)]

Which yields the exact same result. Just in case you didn't know that. If 
you wanted to display the
inherent difference between the result-type of a *map* and 
*list-comprehension*, I haven't wrote this.

Stefan

On Monday, November 24, 2014 4:06:21 PM UTC+1, Erno Scheiber wrote:
>
> Differently
>
> a(x)=x^2
> b(x)=x^3
> v=cell(2)
> v[1]=a
> v[2]=b
> f(x)=[v[i](x) for i=1:length(v)]
> # from Stefan Schwarz reply
> g(x)=map(y->y(x),[a,b])
> typeof(f(2))
> Array{Any,1}
> typeof(g(2))
> Array{Int64,1}
>
>
> On Monday, November 24, 2014 10:48:26 AM UTC+2, K leo wrote:
>>
>> I have a bunch of functions with same signature.  I would like to invoke 
>> them one by one at run time.  What is the best way of doing this?  Can I 
>> put them all in an array or some other collection and loop through it at 
>> the run time? 
>>
>>

[julia-users] Re: how to do array (or collection) of functions

[Erno Scheiber]

Differently

a(x)=x^2
b(x)=x^3
v=cell(2)
v[1]=a
v[2]=b
f(x)=[v[i](x) for i=1:length(v)]
# from Stefan Schwarz reply
g(x)=map(y->y(x),[a,b])
typeof(f(2))
Array{Any,1}
typeof(g(2))
Array{Int64,1}


On Monday, November 24, 2014 10:48:26 AM UTC+2, K leo wrote:
>
> I have a bunch of functions with same signature.  I would like to invoke 
> them one by one at run time.  What is the best way of doing this?  Can I 
> put them all in an array or some other collection and loop through it at 
> the run time? 
>
>

[julia-users] Re: Solving a Bellman equation with Julia through function iteration: what I do wrong?

[Nils Gudat]

Given that what you've posted looks like you are porting code from Matlab, 
you might want to consider this: 
Julia Docs - Noteworthy Differences from other Languages 

Re: [julia-users] Re: C/MPI/SLURM => Julia/?/SLURM?

[Erik Schnetter]

To use MPI from Julia, use the MPI.jl package
 (`Pkg.add("MPI")`). There are also
examples in this package.

Similar to Python, this MPI wrapper has two API levels: a lower level
that feels much like C, and a higher level where arbitrary Julia
objects can be sent and received. The MPI wrapper is not complete; if
you need an MPI function that isn't wrapped, please speak up.

-erik

On Sun, Nov 23, 2014 at 11:32 PM, Gabriel Mihalache  wrote:
>> 1) It's pretty easy to use MPI from Julia. For some use cases it may make
>> more sense than Julia's built-in approach to parallelism, especially if
>> you're already comfortable with MPI. Though if you're well served by pmap,
>> that's much simpler.
>
>
> Do you mean I should be able to broadcast and allgather memory managed by
> Julia? That sounds scary. Luckily for me these models should be amenable to
> pmap. Is there an example of MPI-from-Julia, just in case?
>
>>
>> 2) It looks like the subproblem you're solving is a 1d optimization. I
>> wouldn't be surprised if you could get a significant speedup by tweaking the
>> optimization algorithm (e.g., using a derivative-based method) and making
>> the function evaluations more efficient (e.g., avoiding temporary
>> allocations).
>
>
> Yes, thank you! This is a toy example for which we have an analytic solution
> to check against. The project I'm working on involves a few more discrete
> and continuous controls and states. I only hope that NLOpt.jl will run in
> reasonable time compared to the C version.
>
>
> Thank you for your reply!
> Gabriel



-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

Re: [julia-users] Re: how to do array (or collection) of functions

[Tim Holy]

I'd also add that there's no reason to be concerned about performance until 
someone complains :). map is a wonderful function, and I use it all the time, 
just not in places where performance really matters.

--Tim

On Monday, November 24, 2014 04:25:15 AM Ivar Nesje wrote:
> > (I expect people here to proclaim that higher order functions like map are
> > slow and raw loops shall be used instead.
> > I disagree and see no point why a higher order function has to be
> > necessarily slower. It's clear and concise syntax.)
> 
> Nobody says higher order functions can't be much faster than they are
> today. The problem is that they currently are slow, because nobody with the
> ability has taken the time to improve the situation yet. Unlike Haskell,
> Julia doesn't have semantics optimized for making higher order functions
> fast, so there are limits to what can be done.

[julia-users] Re: ODE package keyword setting

[Alex]

Hi Erno,

I am afraid that points=:specified is currently not supported by ODE.jl. 
There is a PR for adding ode23s (stiff solver) 
, which supports it. So if you 
need/want it now you might try that fork. As Ivar says we are discussing 
how points=:specified should work in the future (mainly #40 
 and #43 
) and any input/insight is 
welcome!

Best,

Alex.

[julia-users] Re: how to do array (or collection) of functions

[Ivar Nesje]

>
> (I expect people here to proclaim that higher order functions like map are 
> slow and raw loops shall be used instead.
> I disagree and see no point why a higher order function has to be 
> necessarily slower. It's clear and concise syntax.)


Nobody says higher order functions can't be much faster than they are 
today. The problem is that they currently are slow, because nobody with the 
ability has taken the time to improve the situation yet. Unlike Haskell, 
Julia doesn't have semantics optimized for making higher order functions 
fast, so there are limits to what can be done. 

Re: [julia-users] How can I bind C code including cross dependent structs

[Michiaki ARIGA]

Thank you for your answer. I'll try it!

On Mon Nov 24 2014 at 3:37:31 Erik Schnetter  wrote:

> One easy solution is to use `Ptr{Void}` in the structure definition,
> which you then need to convert before accessing the structure.
>
> -erik
>
> On Sun, Nov 23, 2014 at 10:51 AM, Michiaki Ariga  wrote:
> > Hi all,
> >
> > I troubled with binding C code to Julia.
> > C code witch I want to bind have cross dependent structs like following,
> >
> > ```
> > struct node {
> >   struct node *next;
> >   struct path *path;
> >   ...
> > }
> >
> > struct path {
> >   struct node *rnode;
> >   struct path *lnext;
> >   ...
> > }
> > ```
> >
> > I know we can bind if I implement type like C structure, but in this case
> > Julia doesn't know
> > 2 structures depends on each other because one struct doesn't know other
> > struct when first struct is defined.
> >
> > How can I bind such structs?
> >
> > ---
> > Michiaki Ariga
> >
> >
>
>
>
> --
> Erik Schnetter 
> http://www.perimeterinstitute.ca/personal/eschnetter/
>

Re: [julia-users] Security problem with unitialized memory

[Tamas Papp]

On Mon, Nov 24 2014, Ronald L. Rivest  wrote:

> Google for "uninitialized security" to find numerous instances
> of security violations and unreliability problems caused by the
> use of uninitialized variables, and numerous security advisories
> warning of problems caused by the (perhaps inadvertent) use
> of uninitialized variables.

AFAIK Julia was not intended for applications that need to be
secure. Consequently, it may not be the best language for implementing
the next kernel, SSH server, etc. Besides the issue that you brought up,
it may have other vulnerabilities, for example, I suspect that if you
implement crypto naively in Julia it may be vulnerable to timing attacks
because of the dynamic nature of the language.

Given that initialization has a cost (however minor), I think is
reasonable not to initialize when in 99% of the cases the array will be
filled anyway with values that are actually computed using other data.

> You can't reasonably ask all programmers to properly initialize
> their allocated storage manually any more than you can ask them
> to test all indices before accessing an array manually; these are
> things that a high-level language should do for you.

Sorry, I may be missing something, but if you are not filling an array
with values, why did you create it in the first place? If you just need
a bunch of zeroes, you should probably use sparse data structures.

> There is no efficiency argument here in favor of uninitialized storage
> that can outweigh the security and reliability disadvantages...

All these considerations are subjective and application-dependent, I
doubt that there is general rule. Some high-level languages use
uninitialized arrays, Common Lisp is an example.

Best,

Tamas

Re: [julia-users] Error when running parallel Julia with large number of threads

[Tim Holy]

That error message sequence reminds me of the one I saw in debugging a problem 
on julia 0.4 that was missing a "GC root," meaning that some julia variable 
allocated from the C code was not being correctly protected from garbage-
collection.

While the bug I fixed can't be present on 0.3 (the relevant code is julia 0.4-
only), it is possible that your bug has been fixed by someone else in the 
meantime---there have been quite a few updates to the 0.3 series. Since you're 
running a slightly outdated version, can you update to the latest 0.3 release?

If updating doesn't fix it, please file an issue with the code needed to 
reproduce the problem.

Best,
--Tim

On Sunday, November 23, 2014 07:48:08 PM Kapil Agarwal wrote:
> Hi
> 
> I am running a parallel matrix transpose using Julia. When I run it on a
> small matrix with small number of workers, it works fine, but as I increase
> the size and the number of workers, it starts giving MemoryError() and
> Broken pipe signals.
> 
> I have put the error stacktrace here :
>  https://github.com/kapiliitr/JuliaBenchmarks/blob/master/error.txt
> 
> I checked that my program was not making any bounds errors and that the
> result is correct. Also, I did not find any errors in the source code from
> the stacktrace.
> I am basically dealing with large matrices of the order of 1000X1000
> elements and around 50-200 worker processes.
> 
> I have a machine with 24 cores and around 24 GB memory, so could this be a
> problem with my system or is there a limit to how many workers Julia can
> launch and allocate memory ?
> 
> Thanks
> 
> Kapil

Re: [julia-users] impressions and questions from a Matlab user

[Tim Holy]

When you re-include your script, does it again take 11 seconds to get to the 
error? If so, something is wrong. How much computation is there in your 
script? You might have a type inference problem, in which case performance 
will be terrible, and it might simply be taking a long time (much longer than 
Matlab) to get to the point in your computation where the bug is triggered.

If you haven't already, read at least up through & including the "Tools" 
section of
http://docs.julialang.org/en/release-0.3/manual/performance-tips/
You should definitely profile; that will tell you where the problem is.

--Tim

On Sunday, November 23, 2014 11:35:03 PM Christian Peel wrote:
> >What's your versioninfo?
> 
> I used Version 0.3.0   (x86_64-apple-darwin13.3.0) on a 2013 macbook which
> took about 9.6 seconds to include the function, try to run it, and find the
> syntax error.   On a 2009 iMac with version 0.3.2 of Julia
> (x86_64-apple-darwin14.0.0) it took 11.3 seconds.Just to be clear, it
> takes something like 3 seconds to load Julia, 26 seconds to load the PyPlot
> package (?!?!?) and then an additional 11.3 seconds after that to get the
> syntax error.   I do not restart Julia every time.
> 
> The functions simulate a narrowband multi-antenna fading communications
> channel. To me it feels like a simple and straightforward script, but it
> may not be so simple for the optimizer.
> 
> Chris
> 
> On Sunday, November 23, 2014 7:21:36 PM UTC-8, Stefan Karpinski wrote:
> > Ah, yes. That would explain this if you're timing how long it takes to
> > start Julia from the command prompt. In that case, I can understand the
> > complaint about the compile-debug-edit cycle, but you probably should
> > consider doing more development at the interactive REPL prompt rather than
> > restarting Julia every time.
> > 
> > On Sun, Nov 23, 2014 at 9:56 PM, Patrick O'Leary  > 
> > > wrote:
> >> On Sunday, November 23, 2014 7:55:33 PM UTC-6, Stefan Karpinski wrote:
> >>> 11 seconds seems like an awfully long time. In the days of the slow REPL
> >>> when Julia compiled itself upon starting up, that's about how long it
> >>> took.
> >>> What's your versioninfo?
> >> 
> >> Windows doesn't ship with sys.dll, for what it's worth.

[julia-users] Re: how to do array (or collection) of functions

[Stefan Schwarz]

Yes you can. Since a function is introducing a new constant in the current 
namespace 
you can access and list them in an array:

a(x) = x^2
b(x) = x^3

map((x) -> x(2), [a, b])

=> 2-element Array{Int64,1}:
 4
 8

 4
 8


(I expect people here to proclaim that higher order functions like map are 
slow and raw loops shall be used instead.
I disagree and see no point why a higher order function has to be 
necessarily slower. It's clear and concise syntax.)

But this has been discussed well enough and I stop right here.

You can use raw loops as well of course.

Hope this helps.

Stefan

On Monday, November 24, 2014 9:48:26 AM UTC+1, K leo wrote:
>
> I have a bunch of functions with same signature.  I would like to invoke 
> them one by one at run time.  What is the best way of doing this?  Can I 
> put them all in an array or some other collection and loop through it at 
> the run time? 
>
>

Re: [julia-users] Security problem with unitialized memory

[Rafael Fourquet]

Regarding Mauro's example: it is my understanding that your first
example is that much faster mainly for cache reasons; indeed, the
speed difference is less visible with the following two functions:

f1(n) = (b = Array(Float64, n); fill!(b, 0.0))
f2(n) = (b = zeros(Float64, n); fill!(b, 0.0))

Then f2 will be at most twice as slow, but often faster than that.

Rafael

Re: [julia-users] Re: Error when running parallel Julia with large number of threads

[Amit Murthy]

I still suspect the immediate cause to be an out-of-memory issue.

Ran your code with Q set to 6, i.e  36 workers. It took around 5.4 GB of
system memory.
With Q set to 12, i.e  72 workers. It took around 10.0 GB of system memory.

200 workers would mean approximately 30GB would be required.

I have not looked at your code in detail, but
https://github.com/JuliaLang/julia/issues/8912 could be contributing to the
problem.



On Mon, Nov 24, 2014 at 9:54 AM, Kapil  wrote:

> DArray
>
> Here is the code as well
> https://github.com/kapiliitr/JuliaBenchmarks/blob/master/ptrans.jl
>
> Regards,
> Kapil Agarwal
>
> On Sun, Nov 23, 2014 at 11:22 PM, Viral Shah  wrote:
>
>> Are you using SharedArray or DArray?
>>
>> -viral
>>
>>
>> On Monday, November 24, 2014 9:18:08 AM UTC+5:30, Kapil Agarwal wrote:
>>>
>>> Hi
>>>
>>> I am running a parallel matrix transpose using Julia. When I run it on a
>>> small matrix with small number of workers, it works fine, but as I increase
>>> the size and the number of workers, it starts giving MemoryError() and
>>> Broken pipe signals.
>>>
>>> I have put the error stacktrace here :  https://github.com/kapiliitr/
>>> JuliaBenchmarks/blob/master/error.txt
>>>
>>> I checked that my program was not making any bounds errors and that the
>>> result is correct. Also, I did not find any errors in the source code from
>>> the stacktrace.
>>> I am basically dealing with large matrices of the order of 1000X1000
>>> elements and around 50-200 worker processes.
>>>
>>> I have a machine with 24 cores and around 24 GB memory, so could this be
>>> a problem with my system or is there a limit to how many workers Julia can
>>> launch and allocate memory ?
>>>
>>> Thanks
>>>
>>> Kapil
>>>
>>
>

[julia-users] how to do array (or collection) of functions

[K Leo]

I have a bunch of functions with same signature.  I would like to invoke 
them one by one at run time.  What is the best way of doing this?  Can I 
put them all in an array or some other collection and loop through it at 
the run time?

Re: [julia-users] Security problem with unitialized memory

[Mauro]

Pointer types will initialise to undef and any operation on them fails:
julia> a = Array(ASCIIString, 5);

julia> a[1]
ERROR: access to undefined reference
 in getindex at array.jl:246

But you're right, for bits-types this is not an error an will just
return whatever was there before.  I think the reason this will stay
that way is that Julia is a numerics oriented language.  Thus you many
wanna create a 1GB array of Float64 and then fill it with something as
opposed to first fill it with zeros and then fill it with something.
See:

julia> @time b = Array(Float64, 10^9);
elapsed time: 0.029523638 seconds (800144 bytes allocated)

julia> @time c = zeros(Float64, 10^9);
elapsed time: 0.835062841 seconds (800168 bytes allocated)

You can argue that the time gain isn't worth the risk but I suspect that
others may feel different.

On Mon, 2014-11-24 at 09:28, Ronald L. Rivest  wrote:
> I am just learning Julia...
>
> I was quite shocked today to learn that Julia does *not*
> initialize allocated storage (e.g. to 0 or some default value).
> E.g. the code
>  A = Array(Int64,5)
>  println(A[1])
> has unpredictable behavior, may disclose information from
> other modules, etc.
>
> This is really quite unacceptable in a modern programming
> language; it is as bad as not checking array reads for out-of-bounds
> indices.  
>
> Google for "uninitialized security" to find numerous instances
> of security violations and unreliability problems caused by the
> use of uninitialized variables, and numerous security advisories
> warning of problems caused by the (perhaps inadvertent) use
> of uninitialized variables.
>
> You can't design a programming language today under the naive
> assumption that code in that language won't be used in highly
> critical applications or won't be under adversarial attack.
>
> You can't reasonably ask all programmers to properly initialize 
> their allocated storage manually any more than you can ask them 
> to test all indices before accessing an array manually; these are
> things that a high-level language should do for you. 
>
> The default non-initialization of allocated storage is a 
> mis-feature that should absolutely be fixed.
>
> There is no efficiency argument here in favor of uninitialized storage
> that can outweigh the security and reliability disadvantages...
>
> Cheers,
> Ron Rivest

[julia-users] Security problem with unitialized memory

[Ronald L. Rivest]

I am just learning Julia...

I was quite shocked today to learn that Julia does *not*
initialize allocated storage (e.g. to 0 or some default value).
E.g. the code
 A = Array(Int64,5)
 println(A[1])
has unpredictable behavior, may disclose information from
other modules, etc.

This is really quite unacceptable in a modern programming
language; it is as bad as not checking array reads for out-of-bounds
indices.  

Google for "uninitialized security" to find numerous instances
of security violations and unreliability problems caused by the
use of uninitialized variables, and numerous security advisories
warning of problems caused by the (perhaps inadvertent) use
of uninitialized variables.

You can't design a programming language today under the naive
assumption that code in that language won't be used in highly
critical applications or won't be under adversarial attack.

You can't reasonably ask all programmers to properly initialize 
their allocated storage manually any more than you can ask them 
to test all indices before accessing an array manually; these are
things that a high-level language should do for you. 

The default non-initialization of allocated storage is a 
mis-feature that should absolutely be fixed.

There is no efficiency argument here in favor of uninitialized storage
that can outweigh the security and reliability disadvantages...

Cheers,
Ron Rivest

[julia-users] Re: ODE package keyword setting

[Ivar Nesje]

In principle it should just be to write something like

odefun(...,points==:specified)

But I'm not sure how many (if any) of the solvers implements that part of 
the API. There was a lot of interest and discussion at some point, but it 
seem to have slowed down recently. 

kl. 08:39:59 UTC+1 mandag 24. november 2014 skrev Erno Scheiber følgende:
>
> In order to use a function of the ODE package, how to set the value 
> "specified" to the keyword "points"?
>