[julia-users] Re: How to set JULIA_NUM_THREADS for remote workers?

[Lyndon White]

On a per machine basis, some `profile` initialisations files run even when 
your login shell is used (ie even if a command is specified in the SSH 
parameter, i.e. even when starting julia workers),
where as others only run if you start a shell script or an interactive 
shell (rc) 
If your default shell (on the remote machines) is bash,
then setting the environment variable JULIA_NUM_THREADS in your 
`~./bash_profile` rather than in `~./.bashrc` will make it work.

See
http://stackoverflow.com/questions/415403/whats-the-difference-between-bashrc-bash-profile-and-environment

Another hack, if you really want to specify it from julia would be to make 
a function that reads your `~/.ssh/enviroment` file, and then appends 
`export JULIA_NUM_THREADS=$n`,
then calls addproc, and then when does replaced the file with its original 
content.
(Use `tryfinally`)

But these are just hacks til the issue is done.

---



On Wednesday, 17 August 2016 18:52:33 UTC+8, Oliver Schulz wrote:
>
> Sure, but *how* do I preset JULIA_NUM_THREADS in the environment of the 
> workers? It'll still be application dependent, so it can be in a .profile 
> or so (which probably won't be run anyway for the worker processes).
>
> On Wednesday, August 17, 2016 at 12:00:16 PM UTC+2, Jeffrey Sarnoff wrote:
>>
>> Hi Oliver,
>>
>> I omitted two letters "the environment" should have been "their 
>> environment":
>> "and in [each of] *their* [remote] enviroment[s] JULIA_NUM_THREADS had 
>> been preset [in each remote environment before each remote Julia had been 
>> started]..
>>
>> from in the REPL ?addprocs
>> " Note that workers do not run a .juliarc.jl startup script, nor do they
>>   synchronize their global state (such as global variables, new method 
>> definitions,
>>   and loaded modules) with any of the other running processes."
>>
>>  so it seems environment variables are one of those nonsynced things
>>
>> -- Jeffrey
>>
>>
>> On Wednesday, August 17, 2016 at 5:17:53 AM UTC-4, Oliver Schulz wrote:
>>>
>>> Hi Jeff,
>>>
>>> > If your remote workers are remotely local invocations of Julia and in 
>>> the environment JULIA_NUM_THREADS has been preset, then the remote workers 
>>> will be using that many threads
>>>
>>> I tried it, and at least when the workers are started via SSH (using 
>>> addprocs([host1, ...])), that doesn't seem to be the case, 
>>> JULIA_NUM_THREADS 
>>> doesn't seem to be passed on. It would actually be very helpful to be 
>>> able to forward (or explicitly set) environment variables for remote 
>>> workers.
>>>
>>> Cheers,
>>>
>>> Oliver
>>>
>>>
>>> On Wednesday, August 17, 2016 at 12:16:11 AM UTC+2, Jeffrey Sarnoff 
>>> wrote:

 Hi Oliver,
 As I understand it:
 JULIA_NUM_THREADS is an environment variable read by the local 
 invocation of Julia.  It is not a run-time passable value. If your remote 
 workers are remotely local invocations of Julia and in the environment 
 JULIA_NUM_THREADS has been preset, then the remote workers will be using 
 that many threads (if the have the cores).


 On Tuesday, August 16, 2016 at 11:52:20 AM UTC-4, Oliver Schulz wrote:
>
> I guess the answer is "no", then?
>
> On Monday, August 1, 2016 at 3:26:17 PM UTC+2, Oliver Schulz wrote:
>>
>> Is it possible to pass on or explicitly set JULIA_NUM_THREADS for 
>> remote workers started via
>>
>> addprocs([host1, ...])
>>
>> ?
>>
>>

[julia-users] Re: Does Julia always need internet access when loading a package?

[Jan Hlavacek]

On Wednesday, August 17, 2016 at 12:37:58 AM UTC-4, Andreas Lobinger wrote:
>
>
> http://docs.julialang.org/en/release-0.4/stdlib/pkg/#Base.Pkg.dir 
>
> For improving the documentation it would be interesting, what information 
> you (or your admin, the 'they') had used for the setup.
>

One thing that is not clear to me is: what happens when I have several 
package directories:  a readonly global one where only root can install new 
packages, and a local user one where the user can install.  Which one 
should Pkg.dir() return?  How will Julia know about the other one? 

I know now about Base.LOAD_PATH and Base.LOAD_CACHE_PATH, and know that 
each of them can have  multiple paths in them.  Their contents does not 
seem to have any effect at all on Pkg.dir(), though.

Is it possible to have some packages in a system wide global directory, say 
/usr/local/share/julia/site/... and some user installed packages in a user 
directory, say ~/.julia/...?  How would such setup be achieved?

[julia-users] Re: Help needed with excessive memory allocation for Tuples

[vavasis]

Just so you know, a similar issue exists in some of the Base types in 
Julia.  For example "sub" (to extract a subarray from an array that does 
not copy the elements but rather provides a view of the original array) and 
"Ref" (to create a pointer that can be passed to a C subroutine) both 
allocate small reference objects on the heap rather than the stack.  This 
means that there is a significant loss of performance, e.g., if a code 
repeatedly uses "sub" to extract a small subarray in an inner loop. In this 
scenario, it is faster to copy the subarray into temporary preallocated 
storage and then copy it back after modification.  I believe that the core 
Julia developers are working on optimizations so that functions like this 
will allocate on the stack instead of the heap if the compiler can prove to 
itself that the "sub" or "Ref" object will never escape from an inner 
loop.  I don't know what is the state of these optimizations.

Stack allocation in the general case (i.e., in the case when the object may 
escape from a loop) does not seem feasible for "sub" objects, "Ref" objects 
or tuples similar to your example.  The reason is that the garbage 
collector needs to know whether there is a live reference to a mutable data 
structure, and if references to mutable data structures were copied around 
like bits objects, the run-time system could not keep track of this.

-- Steve Vavasis


 


On Wednesday, August 17, 2016 at 9:09:31 AM UTC-4, Bill Hart wrote:
>
> We have a (time critical function) with the following prototype:
>
> heapinsert!{N}(xs::Array{Tuple{NTuple{N, UInt}, heap_s}, 1}, exp::NTuple{N
> , UInt}, x::heap_s)
>
>
> However, one line of code in the implementation is allocating gigabytes of 
> memory, which is causing a massive performance deficit in our program:
>
> xs[1] = (exp, x)
>
> Note heap_s is just an ordinary Julia type here, which contains a couple 
> of Ints and another heap_s. As far as I can see it is irrelevant. It should 
> just be a pointer at the machine level.
>
> It seems that Julia allocates space for the tuple (exp, x) and then writes 
> it into the array xs.
>
> Does anyone have any ideas how I can stop it from making this unnecessary 
> allocation? It's slowing down multivariate polynomial multiplication by a 
> factor of at least 2 (and possibly 10) and causing a massive blowout in 
> memory allocation (this is really performance critical code).
>
> I've already tried making using an immutable rather than a tuple. Same 
> problem. I'm tracing this using julia --track-allocation=all.
>
> Bill.
>

Re: [julia-users] Fast random access to Dict

[Erik Schnetter]

For the insertion stage, the Dict is likely ideal.

For the second stage, a Dict that is based on hashing is a good data
structure. Another data structure worth examining would be a sorted vector
of `(Int64, Float64)` pairs (sorted by the `Int64` keys). Interpolation
search  can then have a
complexity of `O(log log N)`. My personal guess is that a well-optimized
Dict (i.e. with both hash function and hash table size "optimized") will be
faster if we assume a close to random access, as it will have fewer memory
accesses.

Julia's Dict implementation (see base/dict.jl) has constants that you could
tune (read: play with). There is also a function `Base.rehash!` that you
can call to increase the size of the hash table, which might increase
performance by avoiding hash collisions, if you have sufficient memory.

-erik


On Wed, Aug 17, 2016 at 7:58 PM, 'Greg Plowman' via julia-users <
julia-users@googlegroups.com> wrote:

> I need fast random access to a Dict{Int64,Float64}.
> My application has a first phase in which the Dict is populated, and a
> second phase where it accessed randomly (with no further additions or
> deletions).
> There are about 50,000 to 100,000 entries, with keys in the range 10^9 to
> 10^14.
>
> Firstly is a Dict the most optimal data structure? (presumably Dict is
> faster than SparseVector for random lookup?)
>
> Secondly, is there any "preconditioning" I can do after creating the Dict
> in phase 1 but before phase 2, to optimise the Dict for random access.
> (e.g. sizehint, sorting keys)
>
> I do appreciate access might already be fast and optimal and I should just
> benchmark, but I'm just looking for some theoretical
> insight before benchmarking.
>
>
>


-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

[julia-users] Fast random access to Dict

['Greg Plowman' via julia-users]

I need fast random access to a Dict{Int64,Float64}.
My application has a first phase in which the Dict is populated, and a 
second phase where it accessed randomly (with no further additions or 
deletions).
There are about 50,000 to 100,000 entries, with keys in the range 10^9 to 
10^14.

Firstly is a Dict the most optimal data structure? (presumably Dict is 
faster than SparseVector for random lookup?)

Secondly, is there any "preconditioning" I can do after creating the Dict 
in phase 1 but before phase 2, to optimise the Dict for random access. 
(e.g. sizehint, sorting keys)

I do appreciate access might already be fast and optimal and I should just 
benchmark, but I'm just looking for some theoretical 
insight before benchmarking.

[julia-users] Re: Error in `/usr/local/bin/julia': malloc(): smallbin double linked list corrupted

[Jeffrey Sarnoff]

moving this to be an issue #18098 


On Wednesday, August 17, 2016 at 7:13:33 PM UTC-4, Jeffrey Sarnoff wrote:
>
> My code uses a stable C library's initialization and freeing (as a 
> finalizer) routines via ccall.  I do not do anything with Julia internals 
> overtly.
>
> On Wednesday, August 17, 2016 at 6:43:36 PM UTC-4, Jeffrey Sarnoff wrote:
>>
>> Is this the fault of my code or does the fault lie elsewhere?
>>
>> v0.5rc2, the log is available: error.txt 
>> 
>>  
>>
>>

[julia-users] Re: Error in `/usr/local/bin/julia': malloc(): smallbin double linked list corrupted

[Jeffrey Sarnoff]

My code uses a stable C library's initialization and freeing (as a 
finalizer) routines via ccall.  I do not do anything with Julia internals 
overtly.

On Wednesday, August 17, 2016 at 6:43:36 PM UTC-4, Jeffrey Sarnoff wrote:
>
> Is this the fault of my code or does the fault lie elsewhere?
>
> v0.5rc2, the log is available: error.txt 
>  
>
>

[julia-users] Error in `/usr/local/bin/julia': malloc(): smallbin double linked list corrupted

[Jeffrey Sarnoff]

Is this the fault of my code or does the fault lie elsewhere?

v0.5rc2, the log is available: error.txt 
 

[julia-users] Re: Wrapping an array from C

[Kiran Pamnany]

On Wednesday, August 17, 2016 at 11:49:14 AM UTC-7, Steven G. Johnson wrote:
>
> Will simply overwrite the Bar that's at array[i]?
>>
>
> Yes. 
>
>
I verified this. Fantastic, thanks!
 

> Is there a way to assert that a type is immutable?
>>
>
> You declare it as
>
>  immutable MyType
>  
>  end
>

No, I mean when I am passed a type as a function parameter, is there a way 
to check if it has been declared as an immutable type?
 

Re: [julia-users] Could I start multiple workers for parallel processing from the program without starting from terminal ?

[Stefan Karpinski]

http://docs.julialang.org/en/release-0.4/stdlib/parallel/#Base.addprocs

On Wed, Aug 17, 2016 at 3:50 PM, Rishabh Raghunath 
wrote:

> Hello Julia Users !
> I am a beginner in Julia ..
> I was wondering if there is a way to start multiple workers for parallel
> processing from the Julia program rather than starting Julia from terminal
> with " julia -p n " ? Is there a way that I can increase the number of
> workers right from the program without actually restarting Julia with the
> above command ?
> I am asking this because.. I have a Programming competition coming up.. so
> was thinking about using parallel processing for faster code execution..
> But the only option I have is to upload the " *.jl " file in the contest
> page and they will tell you if the program was correctly executed or not..
> And occasionally We get "Time Limit Exceeded"..
> So when I upload my code and they run it on their server, they probably
> didn't start Julia to work with multiple workers to execute my program..
> Lets Assume they started Julia as usual with only 1 worker process.. Is
> there a way that my code can add more workers and use parallel processing
>  (I'm guessing its not possible) ?
> Or would they have started Julia with multiple workers because of the
> large volume of code they receive for execution ? What do you guys think ?
> And hacks to get this working ?
>

[julia-users] Could I start multiple workers for parallel processing from the program without starting from terminal ?

[Rishabh Raghunath]

Hello Julia Users !
I am a beginner in Julia ..  
I was wondering if there is a way to start multiple workers for parallel 
processing from the Julia program rather than starting Julia from terminal 
with " julia -p n " ? Is there a way that I can increase the number of 
workers right from the program without actually restarting Julia with the 
above command ?
I am asking this because.. I have a Programming competition coming up.. so 
was thinking about using parallel processing for faster code execution..
But the only option I have is to upload the " *.jl " file in the contest 
page and they will tell you if the program was correctly executed or not.. 
And occasionally We get "Time Limit Exceeded".. 
So when I upload my code and they run it on their server, they probably 
didn't start Julia to work with multiple workers to execute my program.. 
Lets Assume they started Julia as usual with only 1 worker process.. Is 
there a way that my code can add more workers and use parallel processing 
 (I'm guessing its not possible) ?
Or would they have started Julia with multiple workers because of the large 
volume of code they receive for execution ? What do you guys think ?
And hacks to get this working ?

Re: [julia-users] How to Manipulate each character in of a string using a for loop in Julia ?

[jonathan . bieler]

Note also that using a range to loop through a string is not a good idea if 
it might contain unicode characters.

s = "a°sd2β2"
for i=1:length(s)
print(s[i])  
end

LoadError: UnicodeError: invalid character index


Last time I checked there was still some functions in Base like findlast that 
had such issue.


The correct way to do it is to use start/done/next. That said it can become a 
bit tedious, when you have to reverse the string order first or so.

I found that using a vector of characters is sometimes an easy solution.

Re: [julia-users] Help needed with excessive memory allocation for Tuples

['Bill Hart' via julia-users]

I created [1] to make specific note of this issue.

Bill.

[1] https://github.com/JuliaLang/julia/issues/18084

[julia-users] Code coverage and multi-line expressions

[Brandon Taylor]

I like using chaining macros a lot, but it tanks my code coverage. For 
example, if I did something like this, the three lines in the middle would 
be marked as not covered. Is this a bug? Should I raise an issue somewhere? 
This seems like it's a general issue with multi-line expressions (for 
example, quote blocks cause the same problem).

```julia
Lazy.@> begin
1
+(2)
-(3)
end
```

Re: [julia-users] Parent type of a parametric type

[Cedric St-Jean]

Thank you, that works. I suppose I will have to use a generated function to
make this type-stable.

On Wed, Aug 17, 2016 at 1:02 AM, Mauro  wrote:

>
> On Wed, 2016-08-17 at 05:04, Cedric St-Jean 
> wrote:
> > Hi, I'm writing a function to recursively traverse a heterogeneous tree
> of
> > immutables and replace certain elements of it. So far I have:
> >
> > subst(x::Number, assoc::Associative) = x
> > subst(x::Variable, assoc::Associative) = get(assoc, x, x)
> > subst{T}(x::T, assoc::Associative) = T([subst(getfield(x, f), assoc) for
> f
> > in fieldnames(x)]...)
> >
> > I.e. numbers are kept as is, variables are possible replaced, and
> composite
> > structures are recursively traversed. This doesn't look very
> > compiler-friendly, so question 1 is if there's a better way of writing
> > this, short of manually writing one `subst` for each type.
> >
> > My problem is that the above fails when the composite structure is
> > parametric, since a Variable might be replaced by a number. Eg. `Pair(2,
> > Variable(:x))` will become `Pair(2, 5)`, and that's a different type, so
> > the T([ ]...) line fails because T is Pair{Int, Variable}.
> >
> > Question 2 is: given T = Pair{Int, Int}, is there any way to recover
> Pair?
> > Pair{Int, Int}.name returns Pair, but as a TypeName.
>
> julia> Pair{Int, Int}.name.primary
> Pair{A,B}
>
> But it's not type stable:
>
> julia> f(z) = z.name.primary
> f (generic function with 1 method)
>
> julia> @code_warntype f(Pair{Int,Int})
> Variables:
>   #self#::#f
>   z::Type{Pair{Int64,Int64}}
>
> Body:
>   begin
>   return (Core.getfield)((Core.getfield)(z::Type{Pair{Int64,
> Int64}},:name)::TypeName,:primary)::Type{T}
>   end::Type{T}
>

[julia-users] Re: What exactly triggers a package recompile?

[Steven G. Johnson]

On Wednesday, August 17, 2016 at 12:01:31 PM UTC-4, Jan Hlavacek wrote:
>
> I am working on a system that has a number of packages installed system 
> wide in /usr/local/share/julia/site/.  The problem is, when trying to load 
> a package, Julia tries to recompile it, and cannot do so since 
> /usr/local/share/julia/site/lib/v0.4/ 
> is not writable. 
>

Just make sure your system-wide cache-file location is added at the *end* 
of Base.LOAD_CACHE_PATH, not the beginning, and any cache files that need 
to be recompiled will be put into the user's .julia directory, not in the 
read-only system-wide location.   See 
https://github.com/JuliaLang/julia/pull/14369 (which is in Julia 0.4.3 or 
later)
 

> I would not expect the recompile, since the cached ji files should have 
> the same version as the installed jl files, so as far as I can tell, no 
> recompilation should be necessary.  I am trying to figure out if there is 
> any version mismatch, or if there is any other reason for the recompile.  
> Also, is there any way to find out the version of an installed package and 
> the version of the cache?
>

 The cache files are recompiled whenever 

:

1) the Julia version changes
2) the module path changes
3) the timestamp on a source file (that was used to create the cache file) 
changes
4) a dependency module changes (i.e. a dependency was recompiled)

I'm guessing that you are bumping into condition (2).  Instead of compiling 
the modules in your home .julia directory and then copying them into 
/usr/local, what I would suggest would be:

a) remove the module and cache from your .julia directory
b) install the module sources into /usr/local/share/julia/site/v0.4
c) (in sudo julia) add /usr/local/share/julia/site/lib/v0.4 to the 
*beginning* of Base.LOAD_CACHE_PATH and import the module (this will 
recompile the cache file directly into /usr/local)
d) have the users (in their .juliarc) add /usr/local/share/julia/site/lib/v0.4 
to the *end* of their Base.LOAD_CACHE_PATH

Re: [julia-users] Help needed with excessive memory allocation for Tuples

['Bill Hart' via julia-users]

On Wednesday, 17 August 2016 20:22:56 UTC+2, Kristoffer Carlsson wrote:
>
> See pr 8134. I dont think you have any worse cache locality than with your 
> original tuple.


You are right. I just have more memory usage. The heap will in fact have 
better cache locality. I'll just have exp appearing in every object in the 
"chained" heap, rather than per chain.

I was trying hard to keep memory usage down, since some of the large 
benchmarks we want to do will barely fit on the machine, otherwise.

I looked at the PR, and it seems relevant, but I'm not sure it directly 
deals with this issue.

The problem with immutables indeed seems to be that you can't make 
immutables that contain pointers stack allocated since they contain 
pointers. But the standard way of dealing with this is to have the GC 
conservatively scan the stack. Jeff seems to make this point.

On the other hand, he seems to suggest that such a mechanism already exists 
but earlier seems to suggest that you should only stack allocate something 
if it doesn't contain pointers. So I'm totally confused after reading that.

I suppose if the compiler is clever enough to elide the creation of tuples 
that are being written directly into heap allocated resources (such as 
arrays), then Jeff can get his way and problems such as the one I have wont 
occur. But I'm very sceptical that this would be a complete solution.

Bill.

[julia-users] Re: Wrapping an array from C

[Steven G. Johnson]

On Wednesday, August 17, 2016 at 12:08:12 PM UTC-4, Kiran Pamnany wrote:
>
> On Wednesday, August 17, 2016 at 5:46:34 AM UTC-7, Steven G. Johnson wrote:
>>
>> An array of immutables is just an array of structs.  Replacing an element 
>> just overwrites that memory in an array, it doesn't involve any allocation.
>>
>
> So the line of code you posted:
>
> array[i] = Bar(x, y, ...)
>
> Will simply overwrite the Bar that's at array[i]?
>

Yes. 

Is there a way to assert that a type is immutable?
>

You declare it as

 immutable MyType
 
 end

Re: [julia-users] Julia eval(text) function

[Steven G. Johnson]

On Wednesday, August 17, 2016 at 12:59:54 PM UTC-4, Erik Schnetter wrote:
>
> In Julia, you would not map a string, but map a function instead:
>

Even in Matlab these days you should really pass functions around.  Matlab 
has perfectly okay lexically scoped anonymous functions @(args...) expr.

Higher-order functions (passing functions as arguments) and anonymous 
functions (defining functions on the fly) are two of those things that 
should be on more people's "must learn" list in programming, but which are 
often pushed to the sidelines.

Re: [julia-users] How to Manipulate each character in of a string using a for loop in Julia ?

[Steven G. Johnson]

On Wednesday, August 17, 2016 at 1:52:47 PM UTC-4, Rishabh Raghunath wrote:
>
> For Example: Writing a Encrypting program requires me to modify the 
> characters in the string..
>

This is a pretty bad example, because encryption almost always (a) doesn't 
work in-place and (b) only looks at raw bytes (ignoring whether they are 
"characters").

Similarly lot of the very low-level C-like operations that you might want 
to do actually involve raw bytes, and so you are better off working with a 
byte array, which you can obtain by Vector{UInt8}(some string) and can 
subsequently convert to a string via String(some byte array).

If you are creating a string incrementally, you can do that (roughly) 
in-place with an IOBuffer() as Jacob mentioned.

A lot of things that you might *think* are do-able in-place, like 
converting a string to upper-case, are actually *not* possible to do 
in-place if you have non-ASCII data, because strings are typically stored 
in a variable-length encoding (UTF-8 is the default in Julia).

 

Re: [julia-users] How to Manipulate each character in of a string using a for loop in Julia ?

['Bill Hart' via julia-users]

Since strings are immutable in Julia, and immutable means you can't change 
them, you can't change the characters in a string whilst keeping them of 
type string.

If you want to be able to change the characters directly, you need to use 
something other than a string, e.g. an array of characters as explained 
above.

Here is an example using byte arrays:

a = b"abcd"

for i in 1:length(a) # <--- note I corrected an error in your original code
a[i] += 1
end

println(String(a))

Technically, the right way to deal with immutable objects is to just create 
a new one every time you want to change something. The compiler can then 
recognise that all you are doing is changing a string and it can decide to 
actually do the mutation behind the scenes automatically. On the other 
hand, I wouldn't necessarily expect this to be the case with Julia 
currently. 

As an example:

a = "abcd"

for i in 1:length(a)
   a ="$(a[1:i-1])$(a[i] + 1)$(a[i + 1:end])"
end

println(a)

Here I've used "string interpolation" and "ranges" to generate a new string 
from parts of the existing one.

I'm not suggesting this is a good way of doing it. I'm just demonstrating 
that this is a "functional" way of doing things, using immutables instead 
of the mutable strings we are used to in C.

Bill.

On Wednesday, 17 August 2016 19:52:47 UTC+2, Rishabh Raghunath wrote:
>
> Thank you for the immense support from this amazing group !!.. I am a 
> beginner in Julia and I am more familiar with the C language.. Now.. I 
> understand strings are immutable..
> I am actually learning Julia for use in a programming contest and love the 
> experience so far..
> There are often questions that require you to manipulate the characters 
> comprising the string. Its easy to do in C as strings are nothing but an 
> array of Characters and you can play around with it.. C is the online 
> language I know well.. So I do not know about other languages..
> But How do I such manipulation with the individual characters of a string 
> while still maintaining the type as a string.. You said its considered a 
> bad Idea to take this route...
> Please tell me how I should go about manipulating characters in Julia the 
> right way while the type is string..?
> For Example: Writing a Encrypting program requires me to modify the 
> characters in the string..
> I tried using Character Array but .. It prints out as separate characters 
> instead of a string
>
> On Wed, Aug 17, 2016 at 10:48 PM, g  
> wrote:
>
>> Isn't it a red herring to say that strings lack setindex because they are 
>> immutable? I think strings don't have setindex! because it is considered to 
>> be a bad API choice, at least partially because you can't do O(1) indexing 
>> for many string encodings.
>>
>> I can easily define a setindex! method* that does what the OP is 
>> expecting, so how can this be related to strings being immutable? 
>>
>> *julia> **Base.setindex!(s::ASCIIString, v,i) = s.data[i]=v*
>>
>> *setindex! (generic function with 58 methods)*
>>
>> *julia> **a="abc"*
>>
>> *"abc"*
>>
>> *julia> **a[1]='7';a*
>>
>> *"7bc"*
>>
>> *julia> **a[1]+=1;a*
>>
>> *"8bc"*
>>
>> *This works in 0.4, and I'm lead to believe this is considered a bad idea.
>>
>> On Wednesday, August 17, 2016 at 12:40:36 AM UTC-6, Jacob Quinn wrote:
>>>
>>> Strings are immutable (similar to other languages). There are several 
>>> different ways to get what you want, but I tend to utilize IOBuffer a lot:
>>>
>>> a = "abcd"
>>> io = IOBuffer()
>>>
>>> for char in a
>>> write(io, a + 1)
>>> end
>>>
>>> println(takebuf_string(io))
>>>
>>> -Jacob
>>>
>>> On Wed, Aug 17, 2016 at 12:30 AM, Rishabh Raghunath >> > wrote:
>>>

 Hello fellow Julia Users!!

 How do you manipulate the individual characters comprising a string in 
 Julia using a for loop ?
 For example:
 ###

 a = "abcd"

   for i in length(a)
a[i]+=1
  end

 print(a)

 
  I am expecting to get my EXPECTED OUTPUT as" bcde  "

  BUT I get the following error:  
 ##

  ERROR: MethodError: `setindex!` has no method matching 
 setindex!(::ASCIIString, ::Char, ::Int64)
  [inlined code] from ./none:2
  in anonymous at ./no file:4294967295

 ##
 I also tried using:

 for i in eachindex(a) instead of the for loop in the above program .. 
 And I get the same error..

 Please tell me what i should do to get my desired output ..
 Please respond ASAP..
 Thanks..

>>>
>>>
>

Re: [julia-users] Help needed with excessive memory allocation for Tuples

[Kristoffer Carlsson]

See pr 8134. I dont think you have any worse cache locality than with your 
original tuple.

Re: [julia-users] Re: Symbols as Dict keys - efficiency / memory allocation?

[Erik Schnetter]

Since the pointer to the symbol's value does not change over time, you can
simply store the pointer (i.e. essentially the object id) as an integer
value. There is no need to choose an offset.

The advantage of storing the pointer instead of the object id is that you
can use the pointer to re-create the symbol. This is not directly possible
with the object id.

-erik

On Wed, Aug 17, 2016 at 1:00 PM, Jameson  wrote:

> We could change the implementation of Symbol to be an opaque offset
> instead of being and opaque value pointer, but then passing around Symbols
> in generic contexts would require boxing that offset. So six of one, half
> dozen of the other. Fixing the issue that all immutable objects can't be
> allocated inline would probably be the better fix for eliminating the
> problem.
>
>
> On Wednesday, August 17, 2016 at 12:44:21 PM UTC-4, Erik Schnetter wrote:
>>
>> See  for what I had in mind.
>>
>> -erik
>>
>> On Wed, Aug 17, 2016 at 10:32 AM, Yichao Yu  wrote:
>>
>>>
>>>
>>> On Wed, Aug 17, 2016 at 10:19 PM, Oliver Schulz <
>>> oliver.sch...@tu-dortmund.de> wrote:
>>>
 Hi Yichao

 I get that - I was just curious why (and I guess there's likely a very
 good reason for it). My intuition would be that a non-GCed interned string
 type could be represented by a bitstype (i.e. a pointer or index to the
 string table). So I was surprised that Symbol is an object reference and
 wanted to understand this a bit better.

>>>
>>> It's a pointer to managed memory with tag that's all what's needed for
>>> it to be a !pointerfree type. There are way more things that we don't GC
>>> and it has nothing to do with whether it's a isbits type or not.
>>>
>>>


 Cheers,

 Oliver


 On Wednesday, August 17, 2016 at 3:35:41 PM UTC+2, Yichao Yu wrote:
>
>
> > PS: Yes, symbols in Julia should be bitstypes.
>
> No, it's a object reference and isn't a `isbits()` type.
>
>
>>
>> On Wed, Aug 17, 2016 at 8:26 AM, Cedric St-Jean 
>> wrote:
>>
>>> Good points.
>>>
>>> Given that symbols have a name which is a string, I don't see how
>>> they could be a bits-type unless they just became numbers to index into 
>>> a
>>> global array somewhere.. i.e. a pointer. Stack-allocating 
>>> non-bits-type
>>> immutables is on the roadmap to 1.0, so that should solve your problems.
>>>
>>> Maybe you can solve the CategorizedPoint problem by using an @enum.
>>>
>>> On Wed, Aug 17, 2016 at 6:48 AM, Oliver Schulz <
>>> oliver...@tu-dortmund.de> wrote:
>>>
 Hello Andreas,

 consider iterating over a Dict{Symbol,Int64}:

 d = Dict(:foo => 42, :bar => 77)
 for x in d println("$(typeof(x)), $(isbits(x))") end


 During iteration, a lot of stack allocated "Pair{Symbol,Int64}"
 objects are created. That is very bad for performance.

 immutable CategorizedPoint
 x::Float64
 y::Float64
 category::Symbol
 end


 Also, consider a (hypothetical) data type for categorized points
 (with a finite number of categories - but extensible, so that Symbol 
 is a
 better fit than using an enum):

 immutable CategorizedPoint
 x::Float64
 y::Float64
 category::Symbol
 end

 I would definitely want this to be a bits-type, and not have every
 instance heap-allocated.


 Cheers,

 Oliver


 On Wednesday, August 17, 2016 at 11:44:13 AM UTC+2, Andreas
 Lobinger wrote:
>
> Hello colleague,
>
> On Wednesday, August 17, 2016 at 11:12:51 AM UTC+2, Oliver Schulz
> wrote:
>>
>> Sure -  I was assuming that as Symbol (as an interned string) is
>> basically just a pointer. So comparisons are O(1), etc. What I'd 
>> like to
>> understand is, why can't it be a bitstype? Currently, it's not, so 
>> Symbol
>> cannot be used in a lightweight (i.e. stack-allocated) immutable 
>> type. I
>> assume there's a good reason for it, I just want to understand why.
>>
>
> Could you make an example how you would like to use Symbol in
> lightweight types?
>
>
>>>
>>
>>
>> --
>> Erik Schnetter  http://www.perimeterinstitute.
>> ca/personal/eschnetter/
>>
>
>
>>>
>>
>>
>> --
>> Erik Schnetter  http://www.perimeterinstitute.
>> ca/personal/eschnetter/
>>
>


-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

Re: [julia-users] Determining L1 cache size?

[Erik Schnetter]

On Wed, Aug 17, 2016 at 1:31 PM, Páll Haraldsson 
wrote:

> On Tuesday, August 16, 2016 at 1:41:54 PM UTC, Erik Schnetter wrote:
>>
>> ```Julia
>> Pkg.add("Hwloc")
>> using Hwloc
>> topo = load_topology()
>> print(topo)
>> ```
>>
>> I just see that Hwloc has problems with Julia 0.5; I'll tag a new release.
>>
>
> Great to know about the hwloc package and its wrapper (I was going to post
> some links on dynamically finding out.. or statically for Linux) .
>
> I did notice however:
>
> "Remove non-functional Windows support
> "
> (just in the wrapper, the base library supports Windows, so you could add
> suport.. and that commit helps knowing how).
>
> [I didn't check FreeBSD (or OS X) support, is it compatible with Linux?]
>

Yes, it works on all Unix systems I tried. The code to determine the L1
cache size is the same on Windows as on Unix. The problem is just that I
don't know how to build the hwloc library on Windows, or how to install a
pre-built one. This is not a difficult problem, I just didn't have a need
for this.

I was looking into this myself a while back (and getting cache-line size).
> I guess a default fallback of at least 4 KB, possibly 32 KB, might be ok
> (and 16-byte cache lines, probably save and lowest/most common size) on
> case your library finds nothing, e.g. on Windows. [BLAS is compiled, I
> think with cache knowledge, maybe there's a way of knowing dynamically with
> what options? I guess not good to rely on, think it's stipped out in
> Julia-lite branch.]
>

Current (and many former) Intel CPUs use an L1 data cache size with 32
kByte, with cache lines of 64 Bytes.

Depending on your application, it is safer to over-estimate the cache line
size, for example if you need to prevent accessing the same cache line from
multiple threads.

-erik

[I was also thinking ahead to AOT compiled Julia.. be aware of that also..
> when cross-compiling Julia and C++ would seem to have to be conservative,
> one reason Julia seems better than C++ (even without AOT).]
>
>
> See here for save lowest-L1 size you want to support (I've never heard of
> lower than 4 KB data, all shared code+data gone and also where 4 KB):
>
> https://en.wikipedia.org/wiki/List_of_ARM_microarchitectures
>
>
> >is there a (cross-platform) way to get the CPU L1 cache size
>
> You assume you have a [L1] cache..? :) My/the first ARM had none.. than
> came 4 KB (shared, later split and more than L1). Yes, all computers in
> decades have a cache, except for newest fastest TOP500 supercomputer
> (scratchpad, Adapteva's CPU similar) and Tera had just many threads..
>
> --
> Palli.
>
>
>
>>
>> -erik
>>
>>
>> On Tue, Aug 16, 2016 at 5:12 AM, Oliver Schulz 
>> wrote:
>>
>>> Hi,
>>>
>>> is there a (cross-platform) way to get the CPU L1 cache size from Julia?
>>> I'd like to determine the optimal block size(s) for an array algorithm.
>>>
>>>
>>> Cheers,
>>>
>>> Oliver
>>>
>>>
>>
>>
>> --
>> Erik Schnetter  http://www.perimeterinstitute.
>> ca/personal/eschnetter/
>>
>


-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

Re: [julia-users] How to Manipulate each character in of a string using a for loop in Julia ?

[Rishabh Raghunath]

Thank you for the immense support from this amazing group !!.. I am a
beginner in Julia and I am more familiar with the C language.. Now.. I
understand strings are immutable..
I am actually learning Julia for use in a programming contest and love the
experience so far..
There are often questions that require you to manipulate the characters
comprising the string. Its easy to do in C as strings are nothing but an
array of Characters and you can play around with it.. C is the online
language I know well.. So I do not know about other languages..
But How do I such manipulation with the individual characters of a string
while still maintaining the type as a string.. You said its considered a
bad Idea to take this route...
Please tell me how I should go about manipulating characters in Julia the
right way while the type is string..?
For Example: Writing a Encrypting program requires me to modify the
characters in the string..
I tried using Character Array but .. It prints out as separate characters
instead of a string

On Wed, Aug 17, 2016 at 10:48 PM, g  wrote:

> Isn't it a red herring to say that strings lack setindex because they are
> immutable? I think strings don't have setindex! because it is considered to
> be a bad API choice, at least partially because you can't do O(1) indexing
> for many string encodings.
>
> I can easily define a setindex! method* that does what the OP is
> expecting, so how can this be related to strings being immutable?
>
> *julia> **Base.setindex!(s::ASCIIString, v,i) = s.data[i]=v*
>
> *setindex! (generic function with 58 methods)*
>
> *julia> **a="abc"*
>
> *"abc"*
>
> *julia> **a[1]='7';a*
>
> *"7bc"*
>
> *julia> **a[1]+=1;a*
>
> *"8bc"*
>
> *This works in 0.4, and I'm lead to believe this is considered a bad idea.
>
> On Wednesday, August 17, 2016 at 12:40:36 AM UTC-6, Jacob Quinn wrote:
>>
>> Strings are immutable (similar to other languages). There are several
>> different ways to get what you want, but I tend to utilize IOBuffer a lot:
>>
>> a = "abcd"
>> io = IOBuffer()
>>
>> for char in a
>> write(io, a + 1)
>> end
>>
>> println(takebuf_string(io))
>>
>> -Jacob
>>
>> On Wed, Aug 17, 2016 at 12:30 AM, Rishabh Raghunath 
>> wrote:
>>
>>>
>>> Hello fellow Julia Users!!
>>>
>>> How do you manipulate the individual characters comprising a string in
>>> Julia using a for loop ?
>>> For example:
>>> ###
>>>
>>> a = "abcd"
>>>
>>>   for i in length(a)
>>>a[i]+=1
>>>  end
>>>
>>> print(a)
>>>
>>> 
>>>  I am expecting to get my EXPECTED OUTPUT as" bcde  "
>>>
>>>  BUT I get the following error:
>>> ##
>>>
>>>  ERROR: MethodError: `setindex!` has no method matching
>>> setindex!(::ASCIIString, ::Char, ::Int64)
>>>  [inlined code] from ./none:2
>>>  in anonymous at ./no file:4294967295
>>>
>>> ##
>>> I also tried using:
>>>
>>> for i in eachindex(a) instead of the for loop in the above program ..
>>> And I get the same error..
>>>
>>> Please tell me what i should do to get my desired output ..
>>> Please respond ASAP..
>>> Thanks..
>>>
>>
>>

[julia-users] Re: Determining L1 cache size?

[Páll Haraldsson]

On Tuesday, August 16, 2016 at 9:12:51 AM UTC, Oliver Schulz wrote:
>
> I'd like to determine the optimal block size(s) for an array algorithm.
>

That is done, as I said, e.g. with BLAS. But is it still relevant? Are 
cache-oblivious algorithms supposed to not need to know sizes and details 
and still be fast? They are all divide-and-conquer/recursive (a sufficient 
condition, there are some more if I recall (or not?)..):

http://supertech.csail.mit.edu/papers/Prokop99.pdf

"*Conventional wisdom says that recursive procedures should be converted 
into iterative loops in order to improve performance [8]. While this 
strategy was effective ten years ago, many recursive programs now actually 
run faster than their iterative counterparts.* So far most of the work by 
architects and compiler writers is concentrated on loop-based iterative 
programs."

I think this applies for last-level size, e.g. you need not know it. I'm 
still thinking you can explain knowing L1 size, and if I recall, this paper 
goes not into details such as cache-line size, that I think you can also 
exploit.

I guess the base-case for your recursion can stop at the L1 size and do 
something different.

--
Palli

Re: [julia-users] Help needed with excessive memory allocation for Tuples

['Bill Hart' via julia-users]

I basically extended the heap_s struct to also include the exp field. This
uses twice the memory, but gets rid of the massive number of heap
allocations. Unfortunately I lose the cache efficiency of the original
solution, so it's still about 20-50% slower than it should be. It'll have
to do for now.

I've been looking around for a specific ticket for this tuple/immutable
issue, but haven't managed to find one.

I see "temporary tuple elision" on Jeff's compiler optimization tracker
marked as done, but I don't know what the details are. There's a bunch of
tickets around the place, mostly about fixed sized arrays and so on, which
seem to depend on efficient tuples. But this is all stuff that went up
before the tupocalypse (which I thought made tuples efficient).

I'd really like to make sure this issue has a ticket somewhere since it is
so fundamental. Do you by chance know where you saw this already? If not,
I'll make another ticket for it so it can be tracked explicitly.

The best I can find is [1]. But I don't see any links to open tickets
there. There is a pull request, but it's not totally clear to me what this
fixes (though it might explain why this hasn't been fixed already). I don't
understand how the Julia GC can work at all without marking roots in the
stack, so I'm not sure how to understand that PR.

Bill.

[1] https://groups.google.com/forum/#!topic/julia-users/LthfABeDN50



On 17 August 2016 at 19:11, Kristoffer Carlsson 
wrote:

> Just make a new type instead of the tuple and directly set the fields of
> it?

Re: [julia-users] How to Manipulate each character in of a string using a for loop in Julia ?

[Stefan Karpinski]

Indeed. Don't do this – your code will definitely break in the future if
you do this.

On Wed, Aug 17, 2016 at 1:18 PM, g  wrote:

> Isn't it a red herring to say that strings lack setindex because they are
> immutable? I think strings don't have setindex! because it is considered to
> be a bad API choice, at least partially because you can't do O(1) indexing
> for many string encodings.
>
> I can easily define a setindex! method* that does what the OP is
> expecting, so how can this be related to strings being immutable?
>
> *julia> **Base.setindex!(s::ASCIIString, v,i) = s.data[i]=v*
>
> *setindex! (generic function with 58 methods)*
>
> *julia> **a="abc"*
>
> *"abc"*
>
> *julia> **a[1]='7';a*
>
> *"7bc"*
>
> *julia> **a[1]+=1;a*
>
> *"8bc"*
>
> *This works in 0.4, and I'm lead to believe this is considered a bad idea.
>
> On Wednesday, August 17, 2016 at 12:40:36 AM UTC-6, Jacob Quinn wrote:
>>
>> Strings are immutable (similar to other languages). There are several
>> different ways to get what you want, but I tend to utilize IOBuffer a lot:
>>
>> a = "abcd"
>> io = IOBuffer()
>>
>> for char in a
>> write(io, a + 1)
>> end
>>
>> println(takebuf_string(io))
>>
>> -Jacob
>>
>> On Wed, Aug 17, 2016 at 12:30 AM, Rishabh Raghunath 
>> wrote:
>>
>>>
>>> Hello fellow Julia Users!!
>>>
>>> How do you manipulate the individual characters comprising a string in
>>> Julia using a for loop ?
>>> For example:
>>> ###
>>>
>>> a = "abcd"
>>>
>>>   for i in length(a)
>>>a[i]+=1
>>>  end
>>>
>>> print(a)
>>>
>>> 
>>>  I am expecting to get my EXPECTED OUTPUT as" bcde  "
>>>
>>>  BUT I get the following error:
>>> ##
>>>
>>>  ERROR: MethodError: `setindex!` has no method matching
>>> setindex!(::ASCIIString, ::Char, ::Int64)
>>>  [inlined code] from ./none:2
>>>  in anonymous at ./no file:4294967295
>>>
>>> ##
>>> I also tried using:
>>>
>>> for i in eachindex(a) instead of the for loop in the above program ..
>>> And I get the same error..
>>>
>>> Please tell me what i should do to get my desired output ..
>>> Please respond ASAP..
>>> Thanks..
>>>
>>
>>

Re: [julia-users] Julia eval(text) function

[Stefan Karpinski]

See also the mapslices

function which is very close to your function but more general.

On Wed, Aug 17, 2016 at 12:59 PM, Erik Schnetter 
wrote:

> In Julia, you would not map a string, but map a function instead:
>
> ```Julia
> function mapcol(f, p0)
> M,N = size(p0)
> y = similar(p0)
> for i in 1:N
> x = view(p0, :,i)
> y[:,i] = f(x)
> end
> y
> end
> ```
>
> You would call it e.g. as `mapcol(v -> v+1, p0)`.
>
> Parsing and evaluating a string is also possible, of course, but is
> usually not necessary. Look at the documentation for `parse` if you are
> interested.
>
> -erik
>
>
> On Wed, Aug 17, 2016 at 3:34 AM, Nicholas Hausch 
> wrote:
>
>> Hello all. I am attempting to translate a MATLAB file that evaluates a
>> string column by column. The function is simple, pasted below:
>>
>> function y = mapcol(text,p0,p1,p2)
>> %MAPCOL
>> % mapcol(STRING,A,B,C)  will execute the command
>> %   contained in STRING over the columns of the matrix A,
>> %   creating one column of the output matrix at a time.
>> %   The additional arguments B and C are available to pass parameters.
>> % EX:  mapcol('x.*p1',A,b) will do a point-wise
>> %  multiplication of each column of A by the vector b.
>> %
>> [M,N] = size(p0);
>> y = [];
>> for i=1:N
>>x = p0(:,i);
>>y(:,i) = eval(text);
>> end
>>
>> In trying to write the equivalent function in Julia, I ran into issues
>> regarding eval() only having access to variables at the global level. Is
>> this possible to translate?/ How would I go about doing so (preferably
>> without too much added complexity, which I noticed in other threads
>> regarding the scope of eval)? I realize that this function may not appear
>> to be very useful on the surface, but for me it would be worth having as a
>> tool to use in other functions.
>>
>> Any feedback is appreciated. Thanks.
>>
>
>
>
> --
> Erik Schnetter  http://www.perimeterinstitute.
> ca/personal/eschnetter/
>

Re: [julia-users] Determining L1 cache size?

[Páll Haraldsson]

On Tuesday, August 16, 2016 at 1:41:54 PM UTC, Erik Schnetter wrote:
>
> ```Julia
> Pkg.add("Hwloc")
> using Hwloc
> topo = load_topology()
> print(topo)
> ```
>
> I just see that Hwloc has problems with Julia 0.5; I'll tag a new release.
>

Great to know about the hwloc package and its wrapper (I was going to post 
some links on dynamically finding out.. or statically for Linux) .

I did notice however:

"Remove non-functional Windows support 
"
 
(just in the wrapper, the base library supports Windows, so you could add 
suport.. and that commit helps knowing how).

[I didn't check FreeBSD (or OS X) support, is it compatible with Linux?]


I was looking into this myself a while back (and getting cache-line size). 
I guess a default fallback of at least 4 KB, possibly 32 KB, might be ok 
(and 16-byte cache lines, probably save and lowest/most common size) on 
case your library finds nothing, e.g. on Windows. [BLAS is compiled, I 
think with cache knowledge, maybe there's a way of knowing dynamically with 
what options? I guess not good to rely on, think it's stipped out in 
Julia-lite branch.]

[I was also thinking ahead to AOT compiled Julia.. be aware of that also.. 
when cross-compiling Julia and C++ would seem to have to be conservative, 
one reason Julia seems better than C++ (even without AOT).]


See here for save lowest-L1 size you want to support (I've never heard of 
lower than 4 KB data, all shared code+data gone and also where 4 KB):

https://en.wikipedia.org/wiki/List_of_ARM_microarchitectures


>is there a (cross-platform) way to get the CPU L1 cache size

You assume you have a [L1] cache..? :) My/the first ARM had none.. than 
came 4 KB (shared, later split and more than L1). Yes, all computers in 
decades have a cache, except for newest fastest TOP500 supercomputer 
(scratchpad, Adapteva's CPU similar) and Tera had just many threads..

-- 
Palli.

 

>
> -erik
>
>
> On Tue, Aug 16, 2016 at 5:12 AM, Oliver Schulz  > wrote:
>
>> Hi,
>>
>> is there a (cross-platform) way to get the CPU L1 cache size from Julia? 
>> I'd like to determine the optimal block size(s) for an array algorithm.
>>
>>
>> Cheers,
>>
>> Oliver
>>
>>
>
>
> -- 
> Erik Schnetter  
> http://www.perimeterinstitute.ca/personal/eschnetter/
>

Re: [julia-users] How to Manipulate each character in of a string using a for loop in Julia ?

[ggggg]

Isn't it a red herring to say that strings lack setindex because they are 
immutable? I think strings don't have setindex! because it is considered to 
be a bad API choice, at least partially because you can't do O(1) indexing 
for many string encodings.

I can easily define a setindex! method* that does what the OP is expecting, 
so how can this be related to strings being immutable? 

*julia> **Base.setindex!(s::ASCIIString, v,i) = s.data[i]=v*

*setindex! (generic function with 58 methods)*

*julia> **a="abc"*

*"abc"*

*julia> **a[1]='7';a*

*"7bc"*

*julia> **a[1]+=1;a*

*"8bc"*

*This works in 0.4, and I'm lead to believe this is considered a bad idea.

On Wednesday, August 17, 2016 at 12:40:36 AM UTC-6, Jacob Quinn wrote:
>
> Strings are immutable (similar to other languages). There are several 
> different ways to get what you want, but I tend to utilize IOBuffer a lot:
>
> a = "abcd"
> io = IOBuffer()
>
> for char in a
> write(io, a + 1)
> end
>
> println(takebuf_string(io))
>
> -Jacob
>
> On Wed, Aug 17, 2016 at 12:30 AM, Rishabh Raghunath  > wrote:
>
>>
>> Hello fellow Julia Users!!
>>
>> How do you manipulate the individual characters comprising a string in 
>> Julia using a for loop ?
>> For example:
>> ###
>>
>> a = "abcd"
>>
>>   for i in length(a)
>>a[i]+=1
>>  end
>>
>> print(a)
>>
>> 
>>  I am expecting to get my EXPECTED OUTPUT as" bcde  "
>>
>>  BUT I get the following error:  
>> ##
>>
>>  ERROR: MethodError: `setindex!` has no method matching 
>> setindex!(::ASCIIString, ::Char, ::Int64)
>>  [inlined code] from ./none:2
>>  in anonymous at ./no file:4294967295
>>
>> ##
>> I also tried using:
>>
>> for i in eachindex(a) instead of the for loop in the above program .. And 
>> I get the same error..
>>
>> Please tell me what i should do to get my desired output ..
>> Please respond ASAP..
>> Thanks..
>>
>
>

Re: [julia-users] Help needed with excessive memory allocation for Tuples

[Kristoffer Carlsson]

Just make a new type instead of the tuple and directly set the fields of it?

Re: [julia-users] Help needed with excessive memory allocation for Tuples

['Bill Hart' via julia-users]

Wowsers! That's one hell of a serious issue.

On 17 August 2016 at 18:33, Kristoffer Carlsson 
wrote:

> Immutables and tuples that contain references to heap allocated objects
> are currently themselves allocated on the heap. There is an issue about it.
>
>

Re: [julia-users] Re: Symbols as Dict keys - efficiency / memory allocation?

[Jameson]

We could change the implementation of Symbol to be an opaque offset instead 
of being and opaque value pointer, but then passing around Symbols in 
generic contexts would require boxing that offset. So six of one, half 
dozen of the other. Fixing the issue that all immutable objects can't be 
allocated inline would probably be the better fix for eliminating the 
problem.


On Wednesday, August 17, 2016 at 12:44:21 PM UTC-4, Erik Schnetter wrote:
>
> See  for what I had in mind.
>
> -erik
>
> On Wed, Aug 17, 2016 at 10:32 AM, Yichao Yu  wrote:
>
>>
>>
>> On Wed, Aug 17, 2016 at 10:19 PM, Oliver Schulz <
>> oliver.sch...@tu-dortmund.de> wrote:
>>
>>> Hi Yichao
>>>
>>> I get that - I was just curious why (and I guess there's likely a very 
>>> good reason for it). My intuition would be that a non-GCed interned string 
>>> type could be represented by a bitstype (i.e. a pointer or index to the 
>>> string table). So I was surprised that Symbol is an object reference and 
>>> wanted to understand this a bit better.
>>>
>>
>> It's a pointer to managed memory with tag that's all what's needed for it 
>> to be a !pointerfree type. There are way more things that we don't GC and 
>> it has nothing to do with whether it's a isbits type or not.
>>  
>>
>>>
>>>
>>> Cheers,
>>>
>>> Oliver
>>>
>>>
>>> On Wednesday, August 17, 2016 at 3:35:41 PM UTC+2, Yichao Yu wrote:


 > PS: Yes, symbols in Julia should be bitstypes.

 No, it's a object reference and isn't a `isbits()` type.


>
> On Wed, Aug 17, 2016 at 8:26 AM, Cedric St-Jean  
> wrote:
>
>> Good points. 
>>
>> Given that symbols have a name which is a string, I don't see how 
>> they could be a bits-type unless they just became numbers to index into 
>> a 
>> global array somewhere.. i.e. a pointer. Stack-allocating 
>> non-bits-type 
>> immutables is on the roadmap to 1.0, so that should solve your problems.
>>
>> Maybe you can solve the CategorizedPoint problem by using an @enum.
>>
>> On Wed, Aug 17, 2016 at 6:48 AM, Oliver Schulz <
>> oliver...@tu-dortmund.de> wrote:
>>
>>> Hello Andreas,
>>>
>>> consider iterating over a Dict{Symbol,Int64}:
>>>
>>> d = Dict(:foo => 42, :bar => 77)
>>> for x in d println("$(typeof(x)), $(isbits(x))") end
>>>
>>>
>>> During iteration, a lot of stack allocated "Pair{Symbol,Int64}" 
>>> objects are created. That is very bad for performance.
>>>
>>> immutable CategorizedPoint
>>> x::Float64
>>> y::Float64
>>> category::Symbol
>>> end
>>>
>>>
>>> Also, consider a (hypothetical) data type for categorized points 
>>> (with a finite number of categories - but extensible, so that Symbol is 
>>> a 
>>> better fit than using an enum):
>>>
>>> immutable CategorizedPoint
>>> x::Float64
>>> y::Float64
>>> category::Symbol
>>> end
>>>
>>> I would definitely want this to be a bits-type, and not have every 
>>> instance heap-allocated.
>>>
>>>
>>> Cheers,
>>>
>>> Oliver
>>>
>>>
>>> On Wednesday, August 17, 2016 at 11:44:13 AM UTC+2, Andreas Lobinger 
>>> wrote:

 Hello colleague,

 On Wednesday, August 17, 2016 at 11:12:51 AM UTC+2, Oliver Schulz 
 wrote:
>
> Sure -  I was assuming that as Symbol (as an interned string) is 
> basically just a pointer. So comparisons are O(1), etc. What I'd like 
> to 
> understand is, why can't it be a bitstype? Currently, it's not, so 
> Symbol 
> cannot be used in a lightweight (i.e. stack-allocated) immutable 
> type. I 
> assume there's a good reason for it, I just want to understand why.
>

 Could you make an example how you would like to use Symbol in 
 lightweight types?


>>
>
>
> -- 
> Erik Schnetter  
> http://www.perimeterinstitute.ca/personal/eschnetter/
>


>>
>
>
> -- 
> Erik Schnetter  
> http://www.perimeterinstitute.ca/personal/eschnetter/
>

Re: [julia-users] Julia eval(text) function

[Erik Schnetter]

In Julia, you would not map a string, but map a function instead:

```Julia
function mapcol(f, p0)
M,N = size(p0)
y = similar(p0)
for i in 1:N
x = view(p0, :,i)
y[:,i] = f(x)
end
y
end
```

You would call it e.g. as `mapcol(v -> v+1, p0)`.

Parsing and evaluating a string is also possible, of course, but is usually
not necessary. Look at the documentation for `parse` if you are interested.

-erik


On Wed, Aug 17, 2016 at 3:34 AM, Nicholas Hausch  wrote:

> Hello all. I am attempting to translate a MATLAB file that evaluates a
> string column by column. The function is simple, pasted below:
>
> function y = mapcol(text,p0,p1,p2)
> %MAPCOL
> % mapcol(STRING,A,B,C)  will execute the command
> %   contained in STRING over the columns of the matrix A,
> %   creating one column of the output matrix at a time.
> %   The additional arguments B and C are available to pass parameters.
> % EX:  mapcol('x.*p1',A,b) will do a point-wise
> %  multiplication of each column of A by the vector b.
> %
> [M,N] = size(p0);
> y = [];
> for i=1:N
>x = p0(:,i);
>y(:,i) = eval(text);
> end
>
> In trying to write the equivalent function in Julia, I ran into issues
> regarding eval() only having access to variables at the global level. Is
> this possible to translate?/ How would I go about doing so (preferably
> without too much added complexity, which I noticed in other threads
> regarding the scope of eval)? I realize that this function may not appear
> to be very useful on the surface, but for me it would be worth having as a
> tool to use in other functions.
>
> Any feedback is appreciated. Thanks.
>



-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

Re: [julia-users] Re: How to Manipulate each character in of a string using a for loop in Julia ?

[Erik Schnetter]

Alternatively, convert the string to an array of characters, which is
mutable:

```Julia
a = Vector{Char}("abcd")
for i in 1:length(a)
   a[i] += 1
end
```

-erik


On Wed, Aug 17, 2016 at 3:59 AM, 'Greg Plowman' via julia-users <
julia-users@googlegroups.com> wrote:

> I think Jacob meant:
>
> for char in a
>write(io, char + 1)
> end
>



-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

Re: [julia-users] Re: Symbols as Dict keys - efficiency / memory allocation?

[Erik Schnetter]

See  for what I had in mind.

-erik

On Wed, Aug 17, 2016 at 10:32 AM, Yichao Yu  wrote:

>
>
> On Wed, Aug 17, 2016 at 10:19 PM, Oliver Schulz <
> oliver.sch...@tu-dortmund.de> wrote:
>
>> Hi Yichao
>>
>> I get that - I was just curious why (and I guess there's likely a very
>> good reason for it). My intuition would be that a non-GCed interned string
>> type could be represented by a bitstype (i.e. a pointer or index to the
>> string table). So I was surprised that Symbol is an object reference and
>> wanted to understand this a bit better.
>>
>
> It's a pointer to managed memory with tag that's all what's needed for it
> to be a !pointerfree type. There are way more things that we don't GC and
> it has nothing to do with whether it's a isbits type or not.
>
>
>>
>>
>> Cheers,
>>
>> Oliver
>>
>>
>> On Wednesday, August 17, 2016 at 3:35:41 PM UTC+2, Yichao Yu wrote:
>>>
>>>
>>> > PS: Yes, symbols in Julia should be bitstypes.
>>>
>>> No, it's a object reference and isn't a `isbits()` type.
>>>
>>>

 On Wed, Aug 17, 2016 at 8:26 AM, Cedric St-Jean 
 wrote:

> Good points.
>
> Given that symbols have a name which is a string, I don't see how they
> could be a bits-type unless they just became numbers to index into a 
> global
> array somewhere.. i.e. a pointer. Stack-allocating non-bits-type
> immutables is on the roadmap to 1.0, so that should solve your problems.
>
> Maybe you can solve the CategorizedPoint problem by using an @enum.
>
> On Wed, Aug 17, 2016 at 6:48 AM, Oliver Schulz <
> oliver...@tu-dortmund.de> wrote:
>
>> Hello Andreas,
>>
>> consider iterating over a Dict{Symbol,Int64}:
>>
>> d = Dict(:foo => 42, :bar => 77)
>> for x in d println("$(typeof(x)), $(isbits(x))") end
>>
>>
>> During iteration, a lot of stack allocated "Pair{Symbol,Int64}"
>> objects are created. That is very bad for performance.
>>
>> immutable CategorizedPoint
>> x::Float64
>> y::Float64
>> category::Symbol
>> end
>>
>>
>> Also, consider a (hypothetical) data type for categorized points
>> (with a finite number of categories - but extensible, so that Symbol is a
>> better fit than using an enum):
>>
>> immutable CategorizedPoint
>> x::Float64
>> y::Float64
>> category::Symbol
>> end
>>
>> I would definitely want this to be a bits-type, and not have every
>> instance heap-allocated.
>>
>>
>> Cheers,
>>
>> Oliver
>>
>>
>> On Wednesday, August 17, 2016 at 11:44:13 AM UTC+2, Andreas Lobinger
>> wrote:
>>>
>>> Hello colleague,
>>>
>>> On Wednesday, August 17, 2016 at 11:12:51 AM UTC+2, Oliver Schulz
>>> wrote:

 Sure -  I was assuming that as Symbol (as an interned string) is
 basically just a pointer. So comparisons are O(1), etc. What I'd like 
 to
 understand is, why can't it be a bitstype? Currently, it's not, so 
 Symbol
 cannot be used in a lightweight (i.e. stack-allocated) immutable type. 
 I
 assume there's a good reason for it, I just want to understand why.

>>>
>>> Could you make an example how you would like to use Symbol in
>>> lightweight types?
>>>
>>>
>


 --
 Erik Schnetter  http://www.perimeterinstitute.
 ca/personal/eschnetter/

>>>
>>>
>


-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

[julia-users] Julia eval(text) function

[Nicholas Hausch]

Hello all. I am attempting to translate a MATLAB file that evaluates a 
string column by column. The function is simple, pasted below:

function y = mapcol(text,p0,p1,p2)
%MAPCOL 
% mapcol(STRING,A,B,C)  will execute the command
%   contained in STRING over the columns of the matrix A,
%   creating one column of the output matrix at a time.
%   The additional arguments B and C are available to pass parameters.
% EX:  mapcol('x.*p1',A,b) will do a point-wise
%  multiplication of each column of A by the vector b.
%
[M,N] = size(p0);
y = [];
for i=1:N
   x = p0(:,i);
   y(:,i) = eval(text);
end

In trying to write the equivalent function in Julia, I ran into issues 
regarding eval() only having access to variables at the global level. Is 
this possible to translate?/ How would I go about doing so (preferably 
without too much added complexity, which I noticed in other threads 
regarding the scope of eval)? I realize that this function may not appear 
to be very useful on the surface, but for me it would be worth having as a 
tool to use in other functions.

Any feedback is appreciated. Thanks.

[julia-users] Memory allocations when accessing tuple fields

[Kristoffer Carlsson]

Dont benchmark in global scope.

[julia-users] Memory allocations when accessing tuple fields

[Uri Patish]

Hi,

Consider the following code, 

type TestType
a::Int
b::Array{Int, 1}
c::Tuple{Int, Int}
end

a = 1
b = [2,3]
c = (4,5)

t = TestType(a, b, c)

println(@allocated a)
println(@allocated t.a)
println(@allocated b)
println(@allocated t.b)
println(@allocated c)
println(@allocated t.c)

Output:

0
0
0
0
0
32

I'm puzzled by the allocation that is happening when accessing the tuple 
field, that is the c field. This doesn't happen with the oher two fields, 
nor when referencing the variables directly. 

Any ideas?

Uri

[julia-users] Re: Performance tips for network data transfer?

[Matthew Pearce]

Thanks for the thoughts people, much appreciated, gives me some ideas to 
work with.

I'm going to play around with pure Julia solutions first as my prior 
experience trying to get MPI.jl running on my cluster in a REPL was 
painful. This could be the wrong attitude and I may have to change it.

Workers will be in the low tens as I only need one per compute node.

Re: [julia-users] Help needed with excessive memory allocation for Tuples

[Kristoffer Carlsson]

Immutables and tuples that contain references to heap allocated objects are 
currently themselves allocated on the heap. There is an issue about it.

Re: [julia-users] Wrapping an array from C

[Kiran Pamnany]

On Wednesday, August 17, 2016 at 12:49:58 AM UTC-7, Bart Janssens wrote:
>
> It looks like ptr is actually an array of pointers? If so, you could 
> access the first element using:
> unsafe_load(convert(Ptr{atyp}, p[1]))
>
>
ptr is a pointer to the beginning of an array of structs. Steve has 
explained that there is really no such thing in Julia, except by using 
immutable types.

Re: [julia-users] Help needed with excessive memory allocation for Tuples

['Bill Hart' via julia-users]

Wait, I understand why heap_s would be allocated on the heap. But why would
a tuple involving a heap_s need to be allocated on the heap assuming the
heap_s already exists? The tuple should just have the NTuple and one
pointer (to a heap_s) at the machine level.

This is clearly the problem, but I don't understand why this would happen
(or how to work around it).

Bill.

On 17 August 2016 at 18:21, Kristoffer Carlsson 
wrote:

> Unless heap_s is a bitstype then the tuple will be allocated on the heap.
> But you are saying it is getting allocated twice?
>

Re: [julia-users] Help needed with excessive memory allocation for Tuples

[Kristoffer Carlsson]

Unless heap_s is a bitstype then the tuple will be allocated on the heap. But 
you are saying it is getting allocated twice?

[julia-users] Re: Wrapping an array from C

[Kiran Pamnany]

On Wednesday, August 17, 2016 at 5:46:34 AM UTC-7, Steven G. Johnson wrote:
>
> An array of immutables is just an array of structs.  Replacing an element 
> just overwrites that memory in an array, it doesn't involve any allocation.
>

So the line of code you posted:

array[i] = Bar(x, y, ...)

Will simply overwrite the Bar that's at array[i]?

Is there a way to assert that a type is immutable?

Re: [julia-users] Help needed with excessive memory allocation for Tuples

['Bill Hart' via julia-users]

It's very, very hard to produce working code that illustrates a problem
when working on a very large system like ours. There's nearly 1000 lines of
code just to set up the data!

If you want to take a look, it's here, though I'm not sure how enlightening
it will be:

https://github.com/wbhart/Nemo.jl/blob/mpoly/src/generic/MPoly.jl

Specifically the issue is at lines 578 and 584.

To run this in Nemo, you would need to check out the mpoly branch, build
Nemo and use the following lines of code:

using Nemo

R, x, y, z, t = PolynomialRing(ZZ, ["x", "y", "z", "t"])

f = 1 + x + y + z + t
p = f^20;

q = Nemo.mul(p, p + 1);

It is this final line of code that in turn invokes Nemo.mul_johnson which
in turn invokes heapinsert!.

I am quite certain the line of code that I gave is at issue here. Instead
of simply writing the tuple directly into the array, it first allocates a
tuple on the heap, then copies that into the array. That obviously
shouldn't happen.

Bill.



On 17 August 2016 at 17:54, Kristoffer Carlsson 
wrote:

> As always, it is very helpful if there is some code that is actually
> runnable in the post that contains the problem. It is easier to run Julia
> code in Julia than in your brain :)

[julia-users] What exactly triggers a package recompile?

[Jan Hlavacek]

I am working on a system that has a number of packages installed system 
wide in /usr/local/share/julia/site/.  The problem is, when trying to load 
a package, Julia tries to recompile it, and cannot do so since 
/usr/local/share/julia/site/lib/v0.4/ 
is not writable. I would not expect the recompile, since the cached ji 
files should have the same version as the installed jl files, so as far as 
I can tell, no recompilation should be necessary.  I am trying to figure 
out if there is any version mismatch, or if there is any other reason for 
the recompile.  Also, is there any way to find out the version of an 
installed package and the version of the cache?

[julia-users] Re: Hang generating sys-all.o

[Jared Crean]

Done: https://github.com/JuliaLang/julia/issues/18080

On Wednesday, August 17, 2016 at 12:28:25 AM UTC-4, Jameson wrote:
>
> Please open a bug on Github. That counter often has had issues with 
> over-counting when other code around it changes, but usually its a minor 
> fix to get it to converge.
>
>
> On Saturday, August 13, 2016 at 9:04:56 PM UTC-4, Jared Crean wrote:
>>
>> I am playing around with Julia's static compilation capabilities 
>> described here: 
>> http://juliacomputing.com/blog/2016/02/09/static-julia.html, but part of 
>> the process hangs. When I try to generate a system image containing 
>> compiled code for all functions:
>>
>> julia5l --output-o sys-all.o --sysimage /mnt/external/build/julia_0.
>> 5latest/usr/lib/julia/sys.so --startup-file=no --compile=all --eval 
>> nothing
>>
>> I get output:
>>
>> found 24662 uncompiled methods for compile-all
>> ... a counter that counts up to 24662...
>> found 115 uncompiled methods for compile-all
>> ... a counter that counts up to 115 ...
>>
>>
>> The last two lines repeat a large number of times until either the 
>> counter stops on some particular value (the value is different each time I 
>> try this) and memory usage steadily increases until I have to kill the 
>> process, or the last two lines repeat infinitely while memory usage 
>> steadily increases, although more slowly than if the counter stops.
>>
>> Has anyone else seen this or know the cause?  I'm running the latest 
>> julia (commit e4b5233, Sat Aug 13 12:10:46 2016 -0400) on Ubuntu.
>>
>>
>> Jared Crean
>>
>

[julia-users] Help needed with excessive memory allocation for Tuples

[Kristoffer Carlsson]

As always, it is very helpful if there is some code that is actually runnable 
in the post that contains the problem. It is easier to run Julia code in Julia 
than in your brain :)

Re: [julia-users] Help needed with excessive memory allocation for Tuples

['Bill Hart' via julia-users]

N = 1 in the case I'm profiling/timing.

It's typically < 10 in applications, usually much less.

Bil.

On 17 August 2016 at 17:50, Kristoffer Carlsson 
wrote:

> What is N typically?

[julia-users] Re: Does Julia always need internet access when loading a package?

[Jan Hlavacek]

On Wednesday, August 17, 2016 at 12:37:58 AM UTC-4, Andreas Lobinger wrote:
>
>
> http://docs.julialang.org/en/release-0.4/stdlib/pkg/#Base.Pkg.dir 
>
> For improving the documentation it would be interesting, what information 
> you (or your admin, the 'they') had used for the setup.
>
>
The documentation seems good, it seems that the problem was caused by 
missing JULIA_PKGDIR environment variable.  Thanks for your help! 

[julia-users] Help needed with excessive memory allocation for Tuples

[Kristoffer Carlsson]

What is N typically?

[julia-users] Re: unable to connect to FTP using IP address

[Samuel Massinon]

Hi Yared,

Okay, since it worked with you and not me I won't be able to tests my 
solutions.

You don't specify that you want to download in binary when you make the 
connection. I'm not sure what version of FTPClient.jl you are using, but 
binary downloads did not work on the first version of FTPClient.jl.

If you have a correct version, you can do the following.
using FTPClient
ftp_init()
ftp = FTP(host="192.168.251.200")
file = download(ftp, "dataOnFTP.bin", "C:\Users\xyz\test.bin")
close(ftp)
ftp_cleanup()

Binary is longer a function in the latest version of FTPClient.jl and it is 
currently assumes that downloads are in binary. So there's no need to 
specify it.

Let me know if this does or doesn't work.

On Wednesday, August 17, 2016 at 9:22:53 AM UTC-5, Yared Melese wrote:
>
> Hi Samuel, 
>
> Thanks 
>
> The following worked
> ftp = FTP(host="192.168.255.69")
> However, if I want the transfer to be in binary mode then it won't work 
>
>
> Thanks 
> Yared
> On Tuesday, August 9, 2016 at 2:26:30 PM UTC-5, Samuel Massinon wrote:
>
>> Hi Yared
>>
>> Sorry, I got side tracked by other things and forgot about this. 
>>
>> Can you show what's in "host", "username", and "password"? Is it 
>> just "192.168.251.200","anonymous", and "" respectively?
>>
>> I tried 
>> ftp('192.168.251.200', 'anonymous', '')
>> But got an error saying that I was unable to reach it. But that might 
>> make sense if I don't have access to it, is that the case here?
>>
>> Other things that might help is that are you sure your ftp connection is 
>> over ssl? Is it implicit? Maybe playing around with `implt=true, ssl=true` 
>> in 
>> ftp = FTP(host="192.168.251.200", implt=true, ssl=true, user="anonymous", 
>> pswd="")
>> might work.
>>
>> Sam
>>
>> On Friday, August 5, 2016 at 9:22:15 AM UTC-5, Yared Melese wrote:
>>>
>>> Hi Samuel, 
>>>
>>> Would you please let me know if the information I provided is enough? 
>>>
>>> Thanks 
>>> Yared
>>>
>>> On Wednesday, July 27, 2016 at 12:23:56 PM UTC-5, Yared Melese wrote:

 Hi Samuel, 

 Here is how it is done in matlab
 % connect to ftp 
 f = ftp(host, username, password)
 mget(f,'dataOnFTP.bin')

 % read data from file 
 readData = fopen('dataOnFTP.bin', 'r', 'ieee-le') 

 %delete file from server 
 delete(f, 'dataOnFTP.bin');

  Here one thing not included is I would like to find the latest file 
 from FTP server because there might be similar names with different 
 suffix. 

 Thanks 
 Yared




 On Tuesday, July 26, 2016 at 1:29:36 PM UTC-5, Samuel Massinon wrote:

> No worries, can you show me how you get the file with another program?
>
> On Tuesday, July 26, 2016 at 10:09:06 AM UTC-5, Yared Melese wrote:
>>
>> Hi Samuel, 
>>
>> Thanks for your inputs 
>>
>> I am trying to connect to a file server with IP address and get the 
>> latest file with certain extension then process the data and save it on 
>> local drive, then delete the file from the server. 
>>
>> Sorry, I don't know how to get files from Curl, it seems it is mainly 
>> designed for Http 
>>
>> Thanks 
>> Yared
>>
>>
>>
>>
>>
>>
>> On Thursday, July 21, 2016 at 1:56:01 PM UTC-5, Samuel Massinon wrote:
>>
>>> Hi Yared,
>>>
>>> The error you are getting is something LibCURL is erring on, as 
>>> described here. https://curl.haxx.se/libcurl/c/libcurl-errors.html
>>>
>>> If I try using curl with your settings, I get
>>> ~ $ curl -u anonymous '192.168.251.200/dataOnFTP.bin 
>>> 
>>> '
>>> Enter host password for user 'anonymous':
>>> curl: (7) Failed to connect to 192.168.251.200 port 80: Network is 
>>> unreachable
>>>
>>> The FTPClient.jl uses the same library as curl and if you could post 
>>> how to get the file with curl, I might be able to better serve you.
>>>
>>> On Wednesday, July 20, 2016 at 4:17:29 PM UTC-5, Yared Melese wrote:



 Hello 

 Would you please let me know if I missed anything, I am using 
 FTPClient and using IP address as a host but not able to connect 
 Here are my commands 

 using FTPClient
 ftp_init()
 ftp = FTP(host="192.168.251.200", implt=true, ssl=true, 
 user="anonymous", pswd="")
 binary(ftp)
 file = download(ftp, "dataOnFTP.bin", "C:\Users\xyz\test.bin")
 close(ftp)
 ftp_cleanup()

 when sending  " using FTPClient" there are bunch of warnings as 
 shown below partially
 WARNING: Base.String is deprecated, use AbstractString instead.
   likely near C:\Users\melese\.julia\v0.4\FTPClient\src\FTPC.jl:35
 WARNING: Base.String is 

Re: [julia-users] Re: Symbols as Dict keys - efficiency / memory allocation?

[Oliver Schulz]

Thanks for the explanation, Yichao!

On Wednesday, August 17, 2016 at 4:33:43 PM UTC+2, Yichao Yu wrote:
>
>
>
> On Wed, Aug 17, 2016 at 10:19 PM, Oliver Schulz  > wrote:
>
>> Hi Yichao
>>
>> I get that - I was just curious why (and I guess there's likely a very 
>> good reason for it). My intuition would be that a non-GCed interned string 
>> type could be represented by a bitstype (i.e. a pointer or index to the 
>> string table). So I was surprised that Symbol is an object reference and 
>> wanted to understand this a bit better.
>>
>
> It's a pointer to managed memory with tag that's all what's needed for it 
> to be a !pointerfree type. There are way more things that we don't GC and 
> it has nothing to do with whether it's a isbits type or not.
>  
>
>>
>>
>> Cheers,
>>
>> Oliver
>>
>>
>> On Wednesday, August 17, 2016 at 3:35:41 PM UTC+2, Yichao Yu wrote:
>>>
>>>
>>> > PS: Yes, symbols in Julia should be bitstypes.
>>>
>>> No, it's a object reference and isn't a `isbits()` type.
>>>
>>>

 On Wed, Aug 17, 2016 at 8:26 AM, Cedric St-Jean  
 wrote:

> Good points. 
>
> Given that symbols have a name which is a string, I don't see how they 
> could be a bits-type unless they just became numbers to index into a 
> global 
> array somewhere.. i.e. a pointer. Stack-allocating non-bits-type 
> immutables is on the roadmap to 1.0, so that should solve your problems.
>
> Maybe you can solve the CategorizedPoint problem by using an @enum.
>
> On Wed, Aug 17, 2016 at 6:48 AM, Oliver Schulz <
> oliver...@tu-dortmund.de> wrote:
>
>> Hello Andreas,
>>
>> consider iterating over a Dict{Symbol,Int64}:
>>
>> d = Dict(:foo => 42, :bar => 77)
>> for x in d println("$(typeof(x)), $(isbits(x))") end
>>
>>
>> During iteration, a lot of stack allocated "Pair{Symbol,Int64}" 
>> objects are created. That is very bad for performance.
>>
>> immutable CategorizedPoint
>> x::Float64
>> y::Float64
>> category::Symbol
>> end
>>
>>
>> Also, consider a (hypothetical) data type for categorized points 
>> (with a finite number of categories - but extensible, so that Symbol is 
>> a 
>> better fit than using an enum):
>>
>> immutable CategorizedPoint
>> x::Float64
>> y::Float64
>> category::Symbol
>> end
>>
>> I would definitely want this to be a bits-type, and not have every 
>> instance heap-allocated.
>>
>>
>> Cheers,
>>
>> Oliver
>>
>>
>> On Wednesday, August 17, 2016 at 11:44:13 AM UTC+2, Andreas Lobinger 
>> wrote:
>>>
>>> Hello colleague,
>>>
>>> On Wednesday, August 17, 2016 at 11:12:51 AM UTC+2, Oliver Schulz 
>>> wrote:

 Sure -  I was assuming that as Symbol (as an interned string) is 
 basically just a pointer. So comparisons are O(1), etc. What I'd like 
 to 
 understand is, why can't it be a bitstype? Currently, it's not, so 
 Symbol 
 cannot be used in a lightweight (i.e. stack-allocated) immutable type. 
 I 
 assume there's a good reason for it, I just want to understand why.

>>>
>>> Could you make an example how you would like to use Symbol in 
>>> lightweight types?
>>>
>>>
>


 -- 
 Erik Schnetter  
 http://www.perimeterinstitute.ca/personal/eschnetter/

>>>
>>>
>

Re: [julia-users] Re: Symbols as Dict keys - efficiency / memory allocation?

[Yichao Yu]

On Wed, Aug 17, 2016 at 10:19 PM, Oliver Schulz <
oliver.sch...@tu-dortmund.de> wrote:

> Hi Yichao
>
> I get that - I was just curious why (and I guess there's likely a very
> good reason for it). My intuition would be that a non-GCed interned string
> type could be represented by a bitstype (i.e. a pointer or index to the
> string table). So I was surprised that Symbol is an object reference and
> wanted to understand this a bit better.
>

It's a pointer to managed memory with tag that's all what's needed for it
to be a !pointerfree type. There are way more things that we don't GC and
it has nothing to do with whether it's a isbits type or not.


>
>
> Cheers,
>
> Oliver
>
>
> On Wednesday, August 17, 2016 at 3:35:41 PM UTC+2, Yichao Yu wrote:
>>
>>
>> > PS: Yes, symbols in Julia should be bitstypes.
>>
>> No, it's a object reference and isn't a `isbits()` type.
>>
>>
>>>
>>> On Wed, Aug 17, 2016 at 8:26 AM, Cedric St-Jean 
>>> wrote:
>>>
 Good points.

 Given that symbols have a name which is a string, I don't see how they
 could be a bits-type unless they just became numbers to index into a global
 array somewhere.. i.e. a pointer. Stack-allocating non-bits-type
 immutables is on the roadmap to 1.0, so that should solve your problems.

 Maybe you can solve the CategorizedPoint problem by using an @enum.

 On Wed, Aug 17, 2016 at 6:48 AM, Oliver Schulz <
 oliver...@tu-dortmund.de> wrote:

> Hello Andreas,
>
> consider iterating over a Dict{Symbol,Int64}:
>
> d = Dict(:foo => 42, :bar => 77)
> for x in d println("$(typeof(x)), $(isbits(x))") end
>
>
> During iteration, a lot of stack allocated "Pair{Symbol,Int64}"
> objects are created. That is very bad for performance.
>
> immutable CategorizedPoint
> x::Float64
> y::Float64
> category::Symbol
> end
>
>
> Also, consider a (hypothetical) data type for categorized points (with
> a finite number of categories - but extensible, so that Symbol is a better
> fit than using an enum):
>
> immutable CategorizedPoint
> x::Float64
> y::Float64
> category::Symbol
> end
>
> I would definitely want this to be a bits-type, and not have every
> instance heap-allocated.
>
>
> Cheers,
>
> Oliver
>
>
> On Wednesday, August 17, 2016 at 11:44:13 AM UTC+2, Andreas Lobinger
> wrote:
>>
>> Hello colleague,
>>
>> On Wednesday, August 17, 2016 at 11:12:51 AM UTC+2, Oliver Schulz
>> wrote:
>>>
>>> Sure -  I was assuming that as Symbol (as an interned string) is
>>> basically just a pointer. So comparisons are O(1), etc. What I'd like to
>>> understand is, why can't it be a bitstype? Currently, it's not, so 
>>> Symbol
>>> cannot be used in a lightweight (i.e. stack-allocated) immutable type. I
>>> assume there's a good reason for it, I just want to understand why.
>>>
>>
>> Could you make an example how you would like to use Symbol in
>> lightweight types?
>>
>>

>>>
>>>
>>> --
>>> Erik Schnetter  http://www.perimeterinstitute.
>>> ca/personal/eschnetter/
>>>
>>
>>

[julia-users] Re: unable to connect to FTP using IP address

[Yared Melese]

Hi Samuel, 

Thanks 

The following worked
ftp = FTP(host="192.168.255.69")
However, if I want the transfer to be in binary mode then it won't work 


Thanks 
Yared
On Tuesday, August 9, 2016 at 2:26:30 PM UTC-5, Samuel Massinon wrote:

> Hi Yared
>
> Sorry, I got side tracked by other things and forgot about this. 
>
> Can you show what's in "host", "username", and "password"? Is it 
> just "192.168.251.200","anonymous", and "" respectively?
>
> I tried 
> ftp('192.168.251.200', 'anonymous', '')
> But got an error saying that I was unable to reach it. But that might make 
> sense if I don't have access to it, is that the case here?
>
> Other things that might help is that are you sure your ftp connection is 
> over ssl? Is it implicit? Maybe playing around with `implt=true, ssl=true` 
> in 
> ftp = FTP(host="192.168.251.200", implt=true, ssl=true, user="anonymous", 
> pswd="")
> might work.
>
> Sam
>
> On Friday, August 5, 2016 at 9:22:15 AM UTC-5, Yared Melese wrote:
>>
>> Hi Samuel, 
>>
>> Would you please let me know if the information I provided is enough? 
>>
>> Thanks 
>> Yared
>>
>> On Wednesday, July 27, 2016 at 12:23:56 PM UTC-5, Yared Melese wrote:
>>>
>>> Hi Samuel, 
>>>
>>> Here is how it is done in matlab
>>> % connect to ftp 
>>> f = ftp(host, username, password)
>>> mget(f,'dataOnFTP.bin')
>>>
>>> % read data from file 
>>> readData = fopen('dataOnFTP.bin', 'r', 'ieee-le') 
>>>
>>> %delete file from server 
>>> delete(f, 'dataOnFTP.bin');
>>>
>>>  Here one thing not included is I would like to find the latest file 
>>> from FTP server because there might be similar names with different suffix. 
>>>
>>> Thanks 
>>> Yared
>>>
>>>
>>>
>>>
>>> On Tuesday, July 26, 2016 at 1:29:36 PM UTC-5, Samuel Massinon wrote:
>>>
 No worries, can you show me how you get the file with another program?

 On Tuesday, July 26, 2016 at 10:09:06 AM UTC-5, Yared Melese wrote:
>
> Hi Samuel, 
>
> Thanks for your inputs 
>
> I am trying to connect to a file server with IP address and get the 
> latest file with certain extension then process the data and save it on 
> local drive, then delete the file from the server. 
>
> Sorry, I don't know how to get files from Curl, it seems it is mainly 
> designed for Http 
>
> Thanks 
> Yared
>
>
>
>
>
>
> On Thursday, July 21, 2016 at 1:56:01 PM UTC-5, Samuel Massinon wrote:
>
>> Hi Yared,
>>
>> The error you are getting is something LibCURL is erring on, as 
>> described here. https://curl.haxx.se/libcurl/c/libcurl-errors.html
>>
>> If I try using curl with your settings, I get
>> ~ $ curl -u anonymous '192.168.251.200/dataOnFTP.bin 
>> 
>> '
>> Enter host password for user 'anonymous':
>> curl: (7) Failed to connect to 192.168.251.200 port 80: Network is 
>> unreachable
>>
>> The FTPClient.jl uses the same library as curl and if you could post 
>> how to get the file with curl, I might be able to better serve you.
>>
>> On Wednesday, July 20, 2016 at 4:17:29 PM UTC-5, Yared Melese wrote:
>>>
>>>
>>>
>>> Hello 
>>>
>>> Would you please let me know if I missed anything, I am using 
>>> FTPClient and using IP address as a host but not able to connect 
>>> Here are my commands 
>>>
>>> using FTPClient
>>> ftp_init()
>>> ftp = FTP(host="192.168.251.200", implt=true, ssl=true, 
>>> user="anonymous", pswd="")
>>> binary(ftp)
>>> file = download(ftp, "dataOnFTP.bin", "C:\Users\xyz\test.bin")
>>> close(ftp)
>>> ftp_cleanup()
>>>
>>> when sending  " using FTPClient" there are bunch of warnings as 
>>> shown below partially
>>> WARNING: Base.String is deprecated, use AbstractString instead.
>>>   likely near C:\Users\melese\.julia\v0.4\FTPClient\src\FTPC.jl:35
>>> WARNING: Base.String is deprecated, use AbstractString instead.
>>>   likely near C:\Users\melese\.julia\v0.4\FTPClient\src\FTPC.jl:67
>>> WARNING: Base.Uint8 is deprecated, use UInt8 instead.
>>>   likely near C:\Users\melese\.julia\v0.4\FTPClient\src\FTPC.jl:81
>>> ..
>>> .
>>> .and at the end I am getting the following error 
>>>
>>> ERROR: Failed to connect. :: LibCURL error #7
>>>  [inlined code] from 
>>> C:\Users\melese\.julia\v0.4\FTPClient\src\FTPC.jl:138
>>>  in ftp_command at 
>>> C:\Users\melese\.julia\v0.4\FTPClient\src\FTPC.jl:454
>>>  in ftp_connect at 
>>> C:\Users\melese\.julia\v0.4\FTPClient\src\FTPC.jl:493
>>>  in call at C:\Users\melese\.julia\v0.4\FTPClient\src\FTPObject.jl:23
>>>
>>> Thanks 
>>> Yared
>>>
>>

Re: [julia-users] Re: Symbols as Dict keys - efficiency / memory allocation?

[Oliver Schulz]

Hi Yichao

I get that - I was just curious why (and I guess there's likely a very good 
reason for it). My intuition would be that a non-GCed interned string type 
could be represented by a bitstype (i.e. a pointer or index to the string 
table). So I was surprised that Symbol is an object reference and wanted to 
understand this a bit better.


Cheers,

Oliver


On Wednesday, August 17, 2016 at 3:35:41 PM UTC+2, Yichao Yu wrote:
>
>
> > PS: Yes, symbols in Julia should be bitstypes.
>
> No, it's a object reference and isn't a `isbits()` type.
>
>
>>
>> On Wed, Aug 17, 2016 at 8:26 AM, Cedric St-Jean > > wrote:
>>
>>> Good points. 
>>>
>>> Given that symbols have a name which is a string, I don't see how they 
>>> could be a bits-type unless they just became numbers to index into a global 
>>> array somewhere.. i.e. a pointer. Stack-allocating non-bits-type 
>>> immutables is on the roadmap to 1.0, so that should solve your problems.
>>>
>>> Maybe you can solve the CategorizedPoint problem by using an @enum.
>>>
>>> On Wed, Aug 17, 2016 at 6:48 AM, Oliver Schulz >> > wrote:
>>>
 Hello Andreas,

 consider iterating over a Dict{Symbol,Int64}:

 d = Dict(:foo => 42, :bar => 77)
 for x in d println("$(typeof(x)), $(isbits(x))") end


 During iteration, a lot of stack allocated "Pair{Symbol,Int64}" objects 
 are created. That is very bad for performance.

 immutable CategorizedPoint
 x::Float64
 y::Float64
 category::Symbol
 end


 Also, consider a (hypothetical) data type for categorized points (with 
 a finite number of categories - but extensible, so that Symbol is a better 
 fit than using an enum):

 immutable CategorizedPoint
 x::Float64
 y::Float64
 category::Symbol
 end

 I would definitely want this to be a bits-type, and not have every 
 instance heap-allocated.


 Cheers,

 Oliver


 On Wednesday, August 17, 2016 at 11:44:13 AM UTC+2, Andreas Lobinger 
 wrote:
>
> Hello colleague,
>
> On Wednesday, August 17, 2016 at 11:12:51 AM UTC+2, Oliver Schulz 
> wrote:
>>
>> Sure -  I was assuming that as Symbol (as an interned string) is 
>> basically just a pointer. So comparisons are O(1), etc. What I'd like to 
>> understand is, why can't it be a bitstype? Currently, it's not, so 
>> Symbol 
>> cannot be used in a lightweight (i.e. stack-allocated) immutable type. I 
>> assume there's a good reason for it, I just want to understand why.
>>
>
> Could you make an example how you would like to use Symbol in 
> lightweight types?
>
>
>>>
>>
>>
>> -- 
>> Erik Schnetter  
>> http://www.perimeterinstitute.ca/personal/eschnetter/
>>
>
>

[julia-users] Re: ANN: Julia 0.5.0-rc2 now available

[Henri Girard]

Ok, I did it again :
0.5
julia> tic(); using PyPlot; toc();
elapsed time: 8.14015376 seconds

julia> tic(); using PyPlot; toc();
elapsed time: 0.000152792 seconds

julia> 
0.4.5
julia> tic(); using PyPlot; toc();
elapsed time: 5.030407122 seconds

julia> tic(); using PyPlot; toc();
elapsed time: 0.00018798 seconds

julia> 

Le vendredi 12 août 2016 14:38:20 UTC+2, Tony Kelman a écrit :
>
> I have just tagged and uploaded release candidate 2 for Julia version 
> 0.5.0. Binaries are available from
>
>
> https://s3.amazonaws.com/julialang/bin/linux/x64/0.5/julia-0.5.0-rc2-linux-x86_64.tar.gz
>  
>
> https://s3.amazonaws.com/julialang/bin/linux/x86/0.5/julia-0.5.0-rc2-linux-i686.tar.gz
>  
>
> https://s3.amazonaws.com/julialang/bin/osx/x64/0.5/julia-0.5.0-rc2-osx10.7+.dmg
>  
>
> https://s3.amazonaws.com/julialang/bin/winnt/x64/0.5/julia-0.5.0-rc2-win64.exe
>  
>
> https://s3.amazonaws.com/julialang/bin/winnt/x86/0.5/julia-0.5.0-rc2-win32.exe
>  
> https://s3.amazonaws.com/julialang/bin/checksums/julia-0.5.0-rc2.sha256 
> https://s3.amazonaws.com/julialang/bin/checksums/julia-0.5.0-rc2.md5 
> For gpg signatures (with this key http://julialang.org/juliareleases.asc) 
> of the Linux tar.gz binaries, append .asc to the filename.
>
> (arm binaries are taking a while to build, I will upload them later - we 
> will also put links to this release candidate on the web site soon)
>
> The primary thing this does not yet include that we do plan on getting 
> into the final 0.5.0 release is proxy support for the package manager. A 
> preliminary version of that has been merged to master but still has some 
> build system issues that need to be worked out. We will put out a release 
> candidate 3 next week that will hopefully have this resolved, along with 
> any other major bug fixes that happen by then. If all goes well and no 
> major blocking issues come up after that, RC3 could possibly be the last 
> release candidate and promoted to final, but we will see how it goes next 
> week. Follow the progress at 
> https://github.com/JuliaLang/julia/issues/17418 and please report any 
> issues you find.
>
> -Tony
>
>

Re: [julia-users] Re: Symbols as Dict keys - efficiency / memory allocation?

[Andreas Lobinger]

Hello colleague,

On Wednesday, August 17, 2016 at 3:01:37 PM UTC+2, Oliver Schulz wrote:
>
> > unless they just became numbers to index into a global array somewhere
>
> I had assumed that this is basically what Symbols (being interned strings) 
> are.
>

I might write something stupid here, but for me Symbols are constants at 
compile time. So this "intered strings" behaviour is part of generating 
code, not running it. I looked at a similar problem - strings as keys in 
Dicts with frequent access - and think a workaround is somewhere in the 
area of hash and object_id. As last resort i thought about implementing a 
fast string->Int64 hash...

Re: [julia-users] Determining L1 cache size?

[Erik Schnetter]

On Wed, Aug 17, 2016 at 5:43 AM, Oliver Schulz  wrote:

> Great - thanks, Eric!
>
> Would it be possible to enable precompilation for SIMD.jl (so that another
> precompiled packages can have it as a dependency)?
>

Sure. Of course, you probably won't see any speed benefit, since most
functions are generated at run time as needed.

Do you want to submit a pull request?

Uh, and can I ask another question regarding SIMD.jl? Is it possible to
> shift SIMD vectors by one element? Ideally, I'd
> like to shift one element out and another element in (for a filter
> algorithm).
>

Yes! This is possible since yesterday using "vector shuffles" (see the
readme), thanks to @GunnarFarneback.

-erik


> On Tuesday, August 16, 2016 at 8:02:30 PM UTC+2, Erik Schnetter wrote:
>>
>> Ooh! Yes, I need to register it. Thanks for the reminder!
>>
>> -erik
>>
>> On Tue, Aug 16, 2016 at 12:01 PM, Oliver Schulz > > wrote:
>>
>>> Thanks, Eric!
>>>
>>> I should have known you had something up your sleeve - I was actually
>>> thinking about adding SIMD.jl to the mix, too. :-)
>>>
>>> Can I ask - do you plan to register SIMD? Or is it more like a prototype?
>>>
>>>
>>> Cheers,
>>>
>>> Oliver
>>>
>>>
>>> On Tuesday, August 16, 2016 at 3:41:54 PM UTC+2, Erik Schnetter wrote:

 ```Julia
 Pkg.add("Hwloc")
 using Hwloc
 topo = load_topology()
 print(topo)
 ```

 I just see that Hwloc has problems with Julia 0.5; I'll tag a new
 release.

 -erik


 On Tue, Aug 16, 2016 at 5:12 AM, Oliver Schulz <
 oliver...@tu-dortmund.de> wrote:

> Hi,
>
> is there a (cross-platform) way to get the CPU L1 cache size from
> Julia? I'd like to determine the optimal block size(s) for an array
> algorithm.
>
>
> Cheers,
>
> Oliver
>
>


 --
 Erik Schnetter  http://www.perimeterinstitute.
 ca/personal/eschnetter/

>>>
>>
>>
>> --
>> Erik Schnetter  http://www.perimeterinstitute.
>> ca/personal/eschnetter/
>>
>


-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

Re: [julia-users] Re: Symbols as Dict keys - efficiency / memory allocation?

[Yichao Yu]

> PS: Yes, symbols in Julia should be bitstypes.

No, it's a object reference and isn't a `isbits()` type.


>
> On Wed, Aug 17, 2016 at 8:26 AM, Cedric St-Jean 
> wrote:
>
>> Good points.
>>
>> Given that symbols have a name which is a string, I don't see how they
>> could be a bits-type unless they just became numbers to index into a global
>> array somewhere.. i.e. a pointer. Stack-allocating non-bits-type
>> immutables is on the roadmap to 1.0, so that should solve your problems.
>>
>> Maybe you can solve the CategorizedPoint problem by using an @enum.
>>
>> On Wed, Aug 17, 2016 at 6:48 AM, Oliver Schulz <
>> oliver.sch...@tu-dortmund.de> wrote:
>>
>>> Hello Andreas,
>>>
>>> consider iterating over a Dict{Symbol,Int64}:
>>>
>>> d = Dict(:foo => 42, :bar => 77)
>>> for x in d println("$(typeof(x)), $(isbits(x))") end
>>>
>>>
>>> During iteration, a lot of stack allocated "Pair{Symbol,Int64}" objects
>>> are created. That is very bad for performance.
>>>
>>> immutable CategorizedPoint
>>> x::Float64
>>> y::Float64
>>> category::Symbol
>>> end
>>>
>>>
>>> Also, consider a (hypothetical) data type for categorized points (with a
>>> finite number of categories - but extensible, so that Symbol is a better
>>> fit than using an enum):
>>>
>>> immutable CategorizedPoint
>>> x::Float64
>>> y::Float64
>>> category::Symbol
>>> end
>>>
>>> I would definitely want this to be a bits-type, and not have every
>>> instance heap-allocated.
>>>
>>>
>>> Cheers,
>>>
>>> Oliver
>>>
>>>
>>> On Wednesday, August 17, 2016 at 11:44:13 AM UTC+2, Andreas Lobinger
>>> wrote:

 Hello colleague,

 On Wednesday, August 17, 2016 at 11:12:51 AM UTC+2, Oliver Schulz wrote:
>
> Sure -  I was assuming that as Symbol (as an interned string) is
> basically just a pointer. So comparisons are O(1), etc. What I'd like to
> understand is, why can't it be a bitstype? Currently, it's not, so Symbol
> cannot be used in a lightweight (i.e. stack-allocated) immutable type. I
> assume there's a good reason for it, I just want to understand why.
>

 Could you make an example how you would like to use Symbol in
 lightweight types?


>>
>
>
> --
> Erik Schnetter  http://www.perimeterinstitute.
> ca/personal/eschnetter/
>

Re: [julia-users] Re: Symbols as Dict keys - efficiency / memory allocation?

[Erik Schnetter]

You could create your own immutable type that holds a `Ptr`, and then
define respective `unsafe` functions that convert from `Symbol` to `Ptr`
and back:

```Julia
# Take a symbol, convert to a bitstype:
sym = :car
ptr = Cstring(sym)
isbits(ptr)

# Convert back to a symbol
sym2 = Symbol(unsafe_wrap(String, ptr))
sym2 === sym
```

The call to the `Symbol` constructor is probably expensive, as Julia will
have to traverse its internal set of existing symbols to determine whether
this symbol is new.

I assume that "hiding" symbols in pointers in this way will break e.g.
serialization. If you define your own serialization format (where you
serialize the pointer instead of the bitstype pointer) you should be fine.
There might be other gotchas like this as well.

-erik

PS: Yes, symbols in Julia should be bitstypes.


On Wed, Aug 17, 2016 at 8:26 AM, Cedric St-Jean 
wrote:

> Good points.
>
> Given that symbols have a name which is a string, I don't see how they
> could be a bits-type unless they just became numbers to index into a global
> array somewhere.. i.e. a pointer. Stack-allocating non-bits-type
> immutables is on the roadmap to 1.0, so that should solve your problems.
>
> Maybe you can solve the CategorizedPoint problem by using an @enum.
>
> On Wed, Aug 17, 2016 at 6:48 AM, Oliver Schulz <
> oliver.sch...@tu-dortmund.de> wrote:
>
>> Hello Andreas,
>>
>> consider iterating over a Dict{Symbol,Int64}:
>>
>> d = Dict(:foo => 42, :bar => 77)
>> for x in d println("$(typeof(x)), $(isbits(x))") end
>>
>>
>> During iteration, a lot of stack allocated "Pair{Symbol,Int64}" objects
>> are created. That is very bad for performance.
>>
>> immutable CategorizedPoint
>> x::Float64
>> y::Float64
>> category::Symbol
>> end
>>
>>
>> Also, consider a (hypothetical) data type for categorized points (with a
>> finite number of categories - but extensible, so that Symbol is a better
>> fit than using an enum):
>>
>> immutable CategorizedPoint
>> x::Float64
>> y::Float64
>> category::Symbol
>> end
>>
>> I would definitely want this to be a bits-type, and not have every
>> instance heap-allocated.
>>
>>
>> Cheers,
>>
>> Oliver
>>
>>
>> On Wednesday, August 17, 2016 at 11:44:13 AM UTC+2, Andreas Lobinger
>> wrote:
>>>
>>> Hello colleague,
>>>
>>> On Wednesday, August 17, 2016 at 11:12:51 AM UTC+2, Oliver Schulz wrote:

 Sure -  I was assuming that as Symbol (as an interned string) is
 basically just a pointer. So comparisons are O(1), etc. What I'd like to
 understand is, why can't it be a bitstype? Currently, it's not, so Symbol
 cannot be used in a lightweight (i.e. stack-allocated) immutable type. I
 assume there's a good reason for it, I just want to understand why.

>>>
>>> Could you make an example how you would like to use Symbol in
>>> lightweight types?
>>>
>>>
>


-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

Re: [julia-users] Re: Symbols as Dict keys - efficiency / memory allocation?

[Oliver Schulz]

> unless they just became numbers to index into a global array somewhere

I had assumed that this is basically what Symbols (being interned strings) 
are.

> Maybe you can solve the CategorizedPoint problem by using an @enum.

For a fixed set of categories, of course, but not otherwise.

> Stack-allocating non-bits-type immutables is on the roadmap to 1.0

I know - but what if I want to put the CategorizedPoint values into an 
array? :-)

On Wednesday, August 17, 2016 at 2:27:09 PM UTC+2, Cedric St-Jean wrote:
>
> Good points. 
>
> Given that symbols have a name which is a string, I don't see how they 
> could be a bits-type unless they just became numbers to index into a global 
> array somewhere.. i.e. a pointer. Stack-allocating non-bits-type 
> immutables is on the roadmap to 1.0, so that should solve your problems.
>
> Maybe you can solve the CategorizedPoint problem by using an @enum.
>
> On Wed, Aug 17, 2016 at 6:48 AM, Oliver Schulz  > wrote:
>
>> Hello Andreas,
>>
>> consider iterating over a Dict{Symbol,Int64}:
>>
>> d = Dict(:foo => 42, :bar => 77)
>> for x in d println("$(typeof(x)), $(isbits(x))") end
>>
>>
>> During iteration, a lot of stack allocated "Pair{Symbol,Int64}" objects 
>> are created. That is very bad for performance.
>>
>> immutable CategorizedPoint
>> x::Float64
>> y::Float64
>> category::Symbol
>> end
>>
>>
>> Also, consider a (hypothetical) data type for categorized points (with a 
>> finite number of categories - but extensible, so that Symbol is a better 
>> fit than using an enum):
>>
>> immutable CategorizedPoint
>> x::Float64
>> y::Float64
>> category::Symbol
>> end
>>
>> I would definitely want this to be a bits-type, and not have every 
>> instance heap-allocated.
>>
>>
>> Cheers,
>>
>> Oliver
>>
>>
>> On Wednesday, August 17, 2016 at 11:44:13 AM UTC+2, Andreas Lobinger 
>> wrote:
>>>
>>> Hello colleague,
>>>
>>> On Wednesday, August 17, 2016 at 11:12:51 AM UTC+2, Oliver Schulz wrote:

 Sure -  I was assuming that as Symbol (as an interned string) is 
 basically just a pointer. So comparisons are O(1), etc. What I'd like to 
 understand is, why can't it be a bitstype? Currently, it's not, so Symbol 
 cannot be used in a lightweight (i.e. stack-allocated) immutable type. I 
 assume there's a good reason for it, I just want to understand why.

>>>
>>> Could you make an example how you would like to use Symbol in 
>>> lightweight types?
>>>
>>>
>

[julia-users] Re: Wrapping an array from C

[Steven G. Johnson]

On Wednesday, August 17, 2016 at 12:39:51 AM UTC-4, Kiran Pamnany wrote:
>
> The memory underlying this array needs to be managed by the C library (it 
> must be pinned and its location published to allow remote memory access). 
> I'd have to provide a specialized allocator, but even that wouldn't work 
> because I cannot fragment the array. So, making the type immutable isn't 
> possible for this purpose.
>

 This is fine.  You can have an array of immutables whose memory is 
allocated in C.

An array of immutables is just an array of structs.  Replacing an element 
just overwrites that memory in an array, it doesn't involve any allocation.

[julia-users] Re: ANN: Julia 0.5.0-rc2 now available

[Uwe Fechner]

Well, your test with 0.4.5 includes the time of the precompilation, your 
test with 0.5 rc2 not, so this is not a valid comparison.
I compared only the time without precompilation, the second time of using 
the module, but after restarting julia.

Uwe

On Wednesday, August 17, 2016 at 2:32:42 PM UTC+2, Henri Girard wrote:
>
> Thank you... The first one is my mistake when I copy it I started by the 
> second line for the second one I will look further in the doc.
>
> (By the way I did your regression test : First is first time runing in 
> julia second is straight after
>
> 0.4.5 
>
> julia> tic();using PyPlot;toc() 
> INFO: Precompiling module PyPlot... 
> elapsed time: 85.385106597 seconds 
> 85.385106597 
>
> julia> tic();using PyPlot;toc() 
> elapsed time: 0.000178282 seconds 
> 0.000178282 
>
> 0.5 rc2 
>
> julia> tic(); using PyPlot; toc(); 
> elapsed time: 8.056058961 seconds 
>
> julia> tic(); using PyPlot; toc(); 
> elapsed time: 0.000141473 seconds 
>
>
>
> Le vendredi 12 août 2016 14:38:20 UTC+2, Tony Kelman a écrit :
>>
>> I have just tagged and uploaded release candidate 2 for Julia version 
>> 0.5.0. Binaries are available from
>>
>>
>> https://s3.amazonaws.com/julialang/bin/linux/x64/0.5/julia-0.5.0-rc2-linux-x86_64.tar.gz
>>  
>>
>> https://s3.amazonaws.com/julialang/bin/linux/x86/0.5/julia-0.5.0-rc2-linux-i686.tar.gz
>>  
>>
>> https://s3.amazonaws.com/julialang/bin/osx/x64/0.5/julia-0.5.0-rc2-osx10.7+.dmg
>>  
>>
>> https://s3.amazonaws.com/julialang/bin/winnt/x64/0.5/julia-0.5.0-rc2-win64.exe
>>  
>>
>> https://s3.amazonaws.com/julialang/bin/winnt/x86/0.5/julia-0.5.0-rc2-win32.exe
>>  
>> https://s3.amazonaws.com/julialang/bin/checksums/julia-0.5.0-rc2.sha256 
>> https://s3.amazonaws.com/julialang/bin/checksums/julia-0.5.0-rc2.md5 
>> For gpg signatures (with this key http://julialang.org/juliareleases.asc) 
>> of the Linux tar.gz binaries, append .asc to the filename.
>>
>> (arm binaries are taking a while to build, I will upload them later - we 
>> will also put links to this release candidate on the web site soon)
>>
>> The primary thing this does not yet include that we do plan on getting 
>> into the final 0.5.0 release is proxy support for the package manager. A 
>> preliminary version of that has been merged to master but still has some 
>> build system issues that need to be worked out. We will put out a release 
>> candidate 3 next week that will hopefully have this resolved, along with 
>> any other major bug fixes that happen by then. If all goes well and no 
>> major blocking issues come up after that, RC3 could possibly be the last 
>> release candidate and promoted to final, but we will see how it goes next 
>> week. Follow the progress at 
>> https://github.com/JuliaLang/julia/issues/17418 and please report any 
>> issues you find.
>>
>> -Tony
>>
>>

[julia-users] Re: ANN: Julia 0.5.0-rc2 now available

[Henri Girard]

Thank you... The first one is my mistake when I copy it I started by the 
second line for the second one I will look further in the doc.

(By the way I did your regression test : First is first time runing in 
julia second is straight after

0.4.5 

julia> tic();using PyPlot;toc() 
INFO: Precompiling module PyPlot... 
elapsed time: 85.385106597 seconds 
85.385106597 

julia> tic();using PyPlot;toc() 
elapsed time: 0.000178282 seconds 
0.000178282 

0.5 rc2 

julia> tic(); using PyPlot; toc(); 
elapsed time: 8.056058961 seconds 

julia> tic(); using PyPlot; toc(); 
elapsed time: 0.000141473 seconds 



Le vendredi 12 août 2016 14:38:20 UTC+2, Tony Kelman a écrit :
>
> I have just tagged and uploaded release candidate 2 for Julia version 
> 0.5.0. Binaries are available from
>
>
> https://s3.amazonaws.com/julialang/bin/linux/x64/0.5/julia-0.5.0-rc2-linux-x86_64.tar.gz
>  
>
> https://s3.amazonaws.com/julialang/bin/linux/x86/0.5/julia-0.5.0-rc2-linux-i686.tar.gz
>  
>
> https://s3.amazonaws.com/julialang/bin/osx/x64/0.5/julia-0.5.0-rc2-osx10.7+.dmg
>  
>
> https://s3.amazonaws.com/julialang/bin/winnt/x64/0.5/julia-0.5.0-rc2-win64.exe
>  
>
> https://s3.amazonaws.com/julialang/bin/winnt/x86/0.5/julia-0.5.0-rc2-win32.exe
>  
> https://s3.amazonaws.com/julialang/bin/checksums/julia-0.5.0-rc2.sha256 
> https://s3.amazonaws.com/julialang/bin/checksums/julia-0.5.0-rc2.md5 
> For gpg signatures (with this key http://julialang.org/juliareleases.asc) 
> of the Linux tar.gz binaries, append .asc to the filename.
>
> (arm binaries are taking a while to build, I will upload them later - we 
> will also put links to this release candidate on the web site soon)
>
> The primary thing this does not yet include that we do plan on getting 
> into the final 0.5.0 release is proxy support for the package manager. A 
> preliminary version of that has been merged to master but still has some 
> build system issues that need to be worked out. We will put out a release 
> candidate 3 next week that will hopefully have this resolved, along with 
> any other major bug fixes that happen by then. If all goes well and no 
> major blocking issues come up after that, RC3 could possibly be the last 
> release candidate and promoted to final, but we will see how it goes next 
> week. Follow the progress at 
> https://github.com/JuliaLang/julia/issues/17418 and please report any 
> issues you find.
>
> -Tony
>
>

Re: [julia-users] Re: Symbols as Dict keys - efficiency / memory allocation?

[Cedric St-Jean]

Good points.

Given that symbols have a name which is a string, I don't see how they
could be a bits-type unless they just became numbers to index into a global
array somewhere.. i.e. a pointer. Stack-allocating non-bits-type
immutables is on the roadmap to 1.0, so that should solve your problems.

Maybe you can solve the CategorizedPoint problem by using an @enum.

On Wed, Aug 17, 2016 at 6:48 AM, Oliver Schulz  wrote:

> Hello Andreas,
>
> consider iterating over a Dict{Symbol,Int64}:
>
> d = Dict(:foo => 42, :bar => 77)
> for x in d println("$(typeof(x)), $(isbits(x))") end
>
>
> During iteration, a lot of stack allocated "Pair{Symbol,Int64}" objects
> are created. That is very bad for performance.
>
> immutable CategorizedPoint
> x::Float64
> y::Float64
> category::Symbol
> end
>
>
> Also, consider a (hypothetical) data type for categorized points (with a
> finite number of categories - but extensible, so that Symbol is a better
> fit than using an enum):
>
> immutable CategorizedPoint
> x::Float64
> y::Float64
> category::Symbol
> end
>
> I would definitely want this to be a bits-type, and not have every
> instance heap-allocated.
>
>
> Cheers,
>
> Oliver
>
>
> On Wednesday, August 17, 2016 at 11:44:13 AM UTC+2, Andreas Lobinger wrote:
>>
>> Hello colleague,
>>
>> On Wednesday, August 17, 2016 at 11:12:51 AM UTC+2, Oliver Schulz wrote:
>>>
>>> Sure -  I was assuming that as Symbol (as an interned string) is
>>> basically just a pointer. So comparisons are O(1), etc. What I'd like to
>>> understand is, why can't it be a bitstype? Currently, it's not, so Symbol
>>> cannot be used in a lightweight (i.e. stack-allocated) immutable type. I
>>> assume there's a good reason for it, I just want to understand why.
>>>
>>
>> Could you make an example how you would like to use Symbol in lightweight
>> types?
>>
>>

Re: [julia-users] Re: How to set JULIA_NUM_THREADS for remote workers?

[Oliver Schulz]

Hi Bart,

~/.ssh/environment could work, but it's a bit too global for my taste. :-)

Cheers,

Oliver


On Wednesday, August 17, 2016 at 1:11:43 PM UTC+2, Bart Janssens wrote:
>
>
>
> On Wed, Aug 17, 2016 at 12:53 PM Oliver Schulz  > wrote:
>
>> Sure, but *how* do I preset JULIA_NUM_THREADS in the environment of the 
>> workers? It'll still be application dependent, so it can be in a .profile 
>> or so (which probably won't be run anyway for the worker processes).
>>
>>
>>
> Hi Oliver,
>
> If the workers are launched through ssh, you might be able to use 
> ~/.ssh/environment. It's explained in the ssh manpage but I haven't tested 
> it.
>
> Cheers,
>
> Bart
>

Re: [julia-users] Re: How to set JULIA_NUM_THREADS for remote workers?

[Oliver Schulz]

Yup: https://github.com/JuliaLang/julia/issues/18074

On Wednesday, August 17, 2016 at 12:54:51 PM UTC+2, Jeffrey Sarnoff wrote:
>
> yep -- that sums it up, you could open an issue (on your issue).
>
> On Wed, Aug 17, 2016 at 6:52 AM, Oliver Schulz  > wrote:
>
>> Sure, but *how* do I preset JULIA_NUM_THREADS in the environment of the 
>> workers? It'll still be application dependent, so it can be in a .profile 
>> or so (which probably won't be run anyway for the worker processes).
>>
>>
>> On Wednesday, August 17, 2016 at 12:00:16 PM UTC+2, Jeffrey Sarnoff wrote:
>>>
>>> Hi Oliver,
>>>
>>> I omitted two letters "the environment" should have been "their 
>>> environment":
>>> "and in [each of] *their* [remote] enviroment[s] JULIA_NUM_THREADS had 
>>> been preset [in each remote environment before each remote Julia had been 
>>> started]..
>>>
>>> from in the REPL ?addprocs
>>> " Note that workers do not run a .juliarc.jl startup script, nor do they
>>>   synchronize their global state (such as global variables, new method 
>>> definitions,
>>>   and loaded modules) with any of the other running processes."
>>>
>>>  so it seems environment variables are one of those nonsynced things
>>>
>>> -- Jeffrey
>>>
>>>
>>> On Wednesday, August 17, 2016 at 5:17:53 AM UTC-4, Oliver Schulz wrote:

 Hi Jeff,

 > If your remote workers are remotely local invocations of Julia and in 
 the environment JULIA_NUM_THREADS has been preset, then the remote workers 
 will be using that many threads

 I tried it, and at least when the workers are started via SSH (using 
 addprocs([host1, ...])), that doesn't seem to be the case, 
 JULIA_NUM_THREADS 
 doesn't seem to be passed on. It would actually be very helpful to be 
 able to forward (or explicitly set) environment variables for remote 
 workers.

 Cheers,

 Oliver


 On Wednesday, August 17, 2016 at 12:16:11 AM UTC+2, Jeffrey Sarnoff 
 wrote:
>
> Hi Oliver,
> As I understand it:
> JULIA_NUM_THREADS is an environment variable read by the local 
> invocation of Julia.  It is not a run-time passable value. If your remote 
> workers are remotely local invocations of Julia and in the environment 
> JULIA_NUM_THREADS has been preset, then the remote workers will be using 
> that many threads (if the have the cores).
>
>
> On Tuesday, August 16, 2016 at 11:52:20 AM UTC-4, Oliver Schulz wrote:
>>
>> I guess the answer is "no", then?
>>
>> On Monday, August 1, 2016 at 3:26:17 PM UTC+2, Oliver Schulz wrote:
>>>
>>> Is it possible to pass on or explicitly set JULIA_NUM_THREADS for 
>>> remote workers started via
>>>
>>> addprocs([host1, ...])
>>>
>>> ?
>>>
>>>
>

Re: [julia-users] Re: How to set JULIA_NUM_THREADS for remote workers?

[Bart Janssens]

On Wed, Aug 17, 2016 at 12:53 PM Oliver Schulz 
wrote:

> Sure, but *how* do I preset JULIA_NUM_THREADS in the environment of the
> workers? It'll still be application dependent, so it can be in a .profile
> or so (which probably won't be run anyway for the worker processes).
>
>
>
Hi Oliver,

If the workers are launched through ssh, you might be able to use
~/.ssh/environment. It's explained in the ssh manpage but I haven't tested
it.

Cheers,

Bart

Re: [julia-users] Re: How to set JULIA_NUM_THREADS for remote workers?

[Jeffrey Sarnoff]

yep -- that sums it up, you could open an issue (on your issue).

On Wed, Aug 17, 2016 at 6:52 AM, Oliver Schulz  wrote:

> Sure, but *how* do I preset JULIA_NUM_THREADS in the environment of the
> workers? It'll still be application dependent, so it can be in a .profile
> or so (which probably won't be run anyway for the worker processes).
>
>
> On Wednesday, August 17, 2016 at 12:00:16 PM UTC+2, Jeffrey Sarnoff wrote:
>>
>> Hi Oliver,
>>
>> I omitted two letters "the environment" should have been "their
>> environment":
>> "and in [each of] *their* [remote] enviroment[s] JULIA_NUM_THREADS had
>> been preset [in each remote environment before each remote Julia had been
>> started]..
>>
>> from in the REPL ?addprocs
>> " Note that workers do not run a .juliarc.jl startup script, nor do they
>>   synchronize their global state (such as global variables, new method
>> definitions,
>>   and loaded modules) with any of the other running processes."
>>
>>  so it seems environment variables are one of those nonsynced things
>>
>> -- Jeffrey
>>
>>
>> On Wednesday, August 17, 2016 at 5:17:53 AM UTC-4, Oliver Schulz wrote:
>>>
>>> Hi Jeff,
>>>
>>> > If your remote workers are remotely local invocations of Julia and in
>>> the environment JULIA_NUM_THREADS has been preset, then the remote workers
>>> will be using that many threads
>>>
>>> I tried it, and at least when the workers are started via SSH (using
>>> addprocs([host1, ...])), that doesn't seem to be the case, JULIA_NUM_THREADS
>>> doesn't seem to be passed on. It would actually be very helpful to be
>>> able to forward (or explicitly set) environment variables for remote
>>> workers.
>>>
>>> Cheers,
>>>
>>> Oliver
>>>
>>>
>>> On Wednesday, August 17, 2016 at 12:16:11 AM UTC+2, Jeffrey Sarnoff
>>> wrote:

 Hi Oliver,
 As I understand it:
 JULIA_NUM_THREADS is an environment variable read by the local
 invocation of Julia.  It is not a run-time passable value. If your remote
 workers are remotely local invocations of Julia and in the environment
 JULIA_NUM_THREADS has been preset, then the remote workers will be using
 that many threads (if the have the cores).


 On Tuesday, August 16, 2016 at 11:52:20 AM UTC-4, Oliver Schulz wrote:
>
> I guess the answer is "no", then?
>
> On Monday, August 1, 2016 at 3:26:17 PM UTC+2, Oliver Schulz wrote:
>>
>> Is it possible to pass on or explicitly set JULIA_NUM_THREADS for
>> remote workers started via
>>
>> addprocs([host1, ...])
>>
>> ?
>>
>>

[julia-users] Re: How to set JULIA_NUM_THREADS for remote workers?

[Oliver Schulz]

Sure, but *how* do I preset JULIA_NUM_THREADS in the environment of the 
workers? It'll still be application dependent, so it can be in a .profile 
or so (which probably won't be run anyway for the worker processes).

On Wednesday, August 17, 2016 at 12:00:16 PM UTC+2, Jeffrey Sarnoff wrote:
>
> Hi Oliver,
>
> I omitted two letters "the environment" should have been "their 
> environment":
> "and in [each of] *their* [remote] enviroment[s] JULIA_NUM_THREADS had 
> been preset [in each remote environment before each remote Julia had been 
> started]..
>
> from in the REPL ?addprocs
> " Note that workers do not run a .juliarc.jl startup script, nor do they
>   synchronize their global state (such as global variables, new method 
> definitions,
>   and loaded modules) with any of the other running processes."
>
>  so it seems environment variables are one of those nonsynced things
>
> -- Jeffrey
>
>
> On Wednesday, August 17, 2016 at 5:17:53 AM UTC-4, Oliver Schulz wrote:
>>
>> Hi Jeff,
>>
>> > If your remote workers are remotely local invocations of Julia and in 
>> the environment JULIA_NUM_THREADS has been preset, then the remote workers 
>> will be using that many threads
>>
>> I tried it, and at least when the workers are started via SSH (using 
>> addprocs([host1, ...])), that doesn't seem to be the case, JULIA_NUM_THREADS 
>> doesn't seem to be passed on. It would actually be very helpful to be 
>> able to forward (or explicitly set) environment variables for remote 
>> workers.
>>
>> Cheers,
>>
>> Oliver
>>
>>
>> On Wednesday, August 17, 2016 at 12:16:11 AM UTC+2, Jeffrey Sarnoff wrote:
>>>
>>> Hi Oliver,
>>> As I understand it:
>>> JULIA_NUM_THREADS is an environment variable read by the local 
>>> invocation of Julia.  It is not a run-time passable value. If your remote 
>>> workers are remotely local invocations of Julia and in the environment 
>>> JULIA_NUM_THREADS has been preset, then the remote workers will be using 
>>> that many threads (if the have the cores).
>>>
>>>
>>> On Tuesday, August 16, 2016 at 11:52:20 AM UTC-4, Oliver Schulz wrote:

 I guess the answer is "no", then?

 On Monday, August 1, 2016 at 3:26:17 PM UTC+2, Oliver Schulz wrote:
>
> Is it possible to pass on or explicitly set JULIA_NUM_THREADS for 
> remote workers started via
>
> addprocs([host1, ...])
>
> ?
>
>

[julia-users] Re: Symbols as Dict keys - efficiency / memory allocation?

[Oliver Schulz]

Hello Andreas,

consider iterating over a Dict{Symbol,Int64}:

d = Dict(:foo => 42, :bar => 77)
for x in d println("$(typeof(x)), $(isbits(x))") end


During iteration, a lot of stack allocated "Pair{Symbol,Int64}" objects are 
created. That is very bad for performance.

immutable CategorizedPoint
x::Float64
y::Float64
category::Symbol
end


Also, consider a (hypothetical) data type for categorized points (with a 
finite number of categories - but extensible, so that Symbol is a better 
fit than using an enum):

immutable CategorizedPoint
x::Float64
y::Float64
category::Symbol
end

I would definitely want this to be a bits-type, and not have every instance 
heap-allocated.


Cheers,

Oliver


On Wednesday, August 17, 2016 at 11:44:13 AM UTC+2, Andreas Lobinger wrote:
>
> Hello colleague,
>
> On Wednesday, August 17, 2016 at 11:12:51 AM UTC+2, Oliver Schulz wrote:
>>
>> Sure -  I was assuming that as Symbol (as an interned string) is 
>> basically just a pointer. So comparisons are O(1), etc. What I'd like to 
>> understand is, why can't it be a bitstype? Currently, it's not, so Symbol 
>> cannot be used in a lightweight (i.e. stack-allocated) immutable type. I 
>> assume there's a good reason for it, I just want to understand why.
>>
>
> Could you make an example how you would like to use Symbol in lightweight 
> types?
>
>

[julia-users] Re: ANN: Julia 0.5.0-rc2 now available

[Uwe Fechner]

Hello,
first, I had to move the line:
gangoffourplot(P,tf(1))
after the line, where you define P. This plot works fine.

Next, I can reproduce that 
stepplot(CLs)
doesn't work. But it does not work on Julia 0.4.6 either, so this problem 
is not related to
Julia 5.0 RC2.

Uwe, Ubuntu 14.04, 64 bit

On Wednesday, August 17, 2016 at 11:38:29 AM UTC+2, Henri Girard wrote:
>
> Using 0.5 RC2, this morning I couldn't get the second plot, the first is 
> ok, but the second doesn't display, and worse even in julia console.
> I will try the regression test 
>
> using ControlSystems, Plots
> gangoffourplot(P,tf(1))
>
> J = 2.0
> b = 0.04
> K = 1.0
> R = 0.08
> L = 1e-4
>
> # Create the model transfer function
> s = tf("s")
> P = K/(s*((J*s + b)*(L*s + R) + K^2))
> # This generates the system
> # TransferFunction:
> #1.0
> # -
> # 0.0002s^3 + 0.160004s^2 + 1.0032s
> #
> #Continuous-time transfer function model
>
> # Create an array of closed loop systems for different values of Kp
> CLs = TransferFunction[kp*P/(1 + kp*P) for kp = [1, 5, 15]];
>
> # Plot the step response of the controllers
> stepplot(CLs);
> # Add legend. This is specific to the PyPlot back-end, see documentation 
> for more info.
> PyPlot.legend(["Kp = 1", "Kp = 5", "Kp = 15"])
>
> Le vendredi 12 août 2016 14:38:20 UTC+2, Tony Kelman a écrit :
>>
>> I have just tagged and uploaded release candidate 2 for Julia version 
>> 0.5.0. Binaries are available from
>>
>>
>> https://s3.amazonaws.com/julialang/bin/linux/x64/0.5/julia-0.5.0-rc2-linux-x86_64.tar.gz
>>  
>>
>> https://s3.amazonaws.com/julialang/bin/linux/x86/0.5/julia-0.5.0-rc2-linux-i686.tar.gz
>>  
>>
>> https://s3.amazonaws.com/julialang/bin/osx/x64/0.5/julia-0.5.0-rc2-osx10.7+.dmg
>>  
>>
>> https://s3.amazonaws.com/julialang/bin/winnt/x64/0.5/julia-0.5.0-rc2-win64.exe
>>  
>>
>> https://s3.amazonaws.com/julialang/bin/winnt/x86/0.5/julia-0.5.0-rc2-win32.exe
>>  
>> https://s3.amazonaws.com/julialang/bin/checksums/julia-0.5.0-rc2.sha256 
>> https://s3.amazonaws.com/julialang/bin/checksums/julia-0.5.0-rc2.md5 
>> For gpg signatures (with this key http://julialang.org/juliareleases.asc) 
>> of the Linux tar.gz binaries, append .asc to the filename.
>>
>> (arm binaries are taking a while to build, I will upload them later - we 
>> will also put links to this release candidate on the web site soon)
>>
>> The primary thing this does not yet include that we do plan on getting 
>> into the final 0.5.0 release is proxy support for the package manager. A 
>> preliminary version of that has been merged to master but still has some 
>> build system issues that need to be worked out. We will put out a release 
>> candidate 3 next week that will hopefully have this resolved, along with 
>> any other major bug fixes that happen by then. If all goes well and no 
>> major blocking issues come up after that, RC3 could possibly be the last 
>> release candidate and promoted to final, but we will see how it goes next 
>> week. Follow the progress at 
>> https://github.com/JuliaLang/julia/issues/17418 and please report any 
>> issues you find.
>>
>> -Tony
>>
>>

[julia-users] Re: How to set JULIA_NUM_THREADS for remote workers?

[Jeffrey Sarnoff]

Hi Oliver,

I omitted two letters "the environment" should have been "their 
environment":
"and in [each of] *their* [remote] enviroment[s] JULIA_NUM_THREADS had been 
preset [in each remote environment before each remote Julia had been 
started]..

from in the REPL ?addprocs
" Note that workers do not run a .juliarc.jl startup script, nor do they
  synchronize their global state (such as global variables, new method 
definitions,
  and loaded modules) with any of the other running processes."

 so it seems environment variables are one of those nonsynced things

-- Jeffrey


On Wednesday, August 17, 2016 at 5:17:53 AM UTC-4, Oliver Schulz wrote:
>
> Hi Jeff,
>
> > If your remote workers are remotely local invocations of Julia and in 
> the environment JULIA_NUM_THREADS has been preset, then the remote workers 
> will be using that many threads
>
> I tried it, and at least when the workers are started via SSH (using 
> addprocs([host1, ...])), that doesn't seem to be the case, JULIA_NUM_THREADS 
> doesn't seem to be passed on. It would actually be very helpful to be 
> able to forward (or explicitly set) environment variables for remote 
> workers.
>
> Cheers,
>
> Oliver
>
>
> On Wednesday, August 17, 2016 at 12:16:11 AM UTC+2, Jeffrey Sarnoff wrote:
>>
>> Hi Oliver,
>> As I understand it:
>> JULIA_NUM_THREADS is an environment variable read by the local invocation 
>> of Julia.  It is not a run-time passable value. If your remote workers are 
>> remotely local invocations of Julia and in the environment 
>> JULIA_NUM_THREADS has been preset, then the remote workers will be using 
>> that many threads (if the have the cores).
>>
>>
>> On Tuesday, August 16, 2016 at 11:52:20 AM UTC-4, Oliver Schulz wrote:
>>>
>>> I guess the answer is "no", then?
>>>
>>> On Monday, August 1, 2016 at 3:26:17 PM UTC+2, Oliver Schulz wrote:

 Is it possible to pass on or explicitly set JULIA_NUM_THREADS for 
 remote workers started via

 addprocs([host1, ...])

 ?

[julia-users] Re: Symbols as Dict keys - efficiency / memory allocation?

[Andreas Lobinger]

Hello colleague,

On Wednesday, August 17, 2016 at 11:12:51 AM UTC+2, Oliver Schulz wrote:
>
> Sure -  I was assuming that as Symbol (as an interned string) is basically 
> just a pointer. So comparisons are O(1), etc. What I'd like to understand 
> is, why can't it be a bitstype? Currently, it's not, so Symbol cannot be 
> used in a lightweight (i.e. stack-allocated) immutable type. I assume 
> there's a good reason for it, I just want to understand why.
>

Could you make an example how you would like to use Symbol in lightweight 
types?

Re: [julia-users] Determining L1 cache size?

[Oliver Schulz]

Great - thanks, Eric!

Would it be possible to enable precompilation for SIMD.jl (so that another 
precompiled packages can have it as a dependency)?

Uh, and can I ask another question regarding SIMD.jl? Is it possible to 
shift SIMD vectors by one element? Ideally, I'd
like to shift one element out and another element in (for a filter 
algorithm).

On Tuesday, August 16, 2016 at 8:02:30 PM UTC+2, Erik Schnetter wrote:
>
> Ooh! Yes, I need to register it. Thanks for the reminder!
>
> -erik
>
> On Tue, Aug 16, 2016 at 12:01 PM, Oliver Schulz  > wrote:
>
>> Thanks, Eric!
>>
>> I should have known you had something up your sleeve - I was actually 
>> thinking about adding SIMD.jl to the mix, too. :-)
>>
>> Can I ask - do you plan to register SIMD? Or is it more like a prototype?
>>
>>
>> Cheers,
>>
>> Oliver
>>
>>
>> On Tuesday, August 16, 2016 at 3:41:54 PM UTC+2, Erik Schnetter wrote:
>>>
>>> ```Julia
>>> Pkg.add("Hwloc")
>>> using Hwloc
>>> topo = load_topology()
>>> print(topo)
>>> ```
>>>
>>> I just see that Hwloc has problems with Julia 0.5; I'll tag a new 
>>> release.
>>>
>>> -erik
>>>
>>>
>>> On Tue, Aug 16, 2016 at 5:12 AM, Oliver Schulz >> > wrote:
>>>
 Hi,

 is there a (cross-platform) way to get the CPU L1 cache size from 
 Julia? I'd like to determine the optimal block size(s) for an array 
 algorithm.


 Cheers,

 Oliver


>>>
>>>
>>> -- 
>>> Erik Schnetter  
>>> http://www.perimeterinstitute.ca/personal/eschnetter/
>>>
>>
>
>
> -- 
> Erik Schnetter  
> http://www.perimeterinstitute.ca/personal/eschnetter/
>

[julia-users] Re: ANN: Julia 0.5.0-rc2 now available

[Henri Girard]

Using 0.5 RC2, this morning I couldn't get the second plot, the first is 
ok, but the second doesn't display, and worse even in julia console.
I will try the regression test 

using ControlSystems, Plots
gangoffourplot(P,tf(1))

J = 2.0
b = 0.04
K = 1.0
R = 0.08
L = 1e-4

# Create the model transfer function
s = tf("s")
P = K/(s*((J*s + b)*(L*s + R) + K^2))
# This generates the system
# TransferFunction:
#1.0
# -
# 0.0002s^3 + 0.160004s^2 + 1.0032s
#
#Continuous-time transfer function model

# Create an array of closed loop systems for different values of Kp
CLs = TransferFunction[kp*P/(1 + kp*P) for kp = [1, 5, 15]];

# Plot the step response of the controllers
stepplot(CLs);
# Add legend. This is specific to the PyPlot back-end, see documentation 
for more info.
PyPlot.legend(["Kp = 1", "Kp = 5", "Kp = 15"])

Le vendredi 12 août 2016 14:38:20 UTC+2, Tony Kelman a écrit :
>
> I have just tagged and uploaded release candidate 2 for Julia version 
> 0.5.0. Binaries are available from
>
>
> https://s3.amazonaws.com/julialang/bin/linux/x64/0.5/julia-0.5.0-rc2-linux-x86_64.tar.gz
>  
>
> https://s3.amazonaws.com/julialang/bin/linux/x86/0.5/julia-0.5.0-rc2-linux-i686.tar.gz
>  
>
> https://s3.amazonaws.com/julialang/bin/osx/x64/0.5/julia-0.5.0-rc2-osx10.7+.dmg
>  
>
> https://s3.amazonaws.com/julialang/bin/winnt/x64/0.5/julia-0.5.0-rc2-win64.exe
>  
>
> https://s3.amazonaws.com/julialang/bin/winnt/x86/0.5/julia-0.5.0-rc2-win32.exe
>  
> https://s3.amazonaws.com/julialang/bin/checksums/julia-0.5.0-rc2.sha256 
> https://s3.amazonaws.com/julialang/bin/checksums/julia-0.5.0-rc2.md5 
> For gpg signatures (with this key http://julialang.org/juliareleases.asc) 
> of the Linux tar.gz binaries, append .asc to the filename.
>
> (arm binaries are taking a while to build, I will upload them later - we 
> will also put links to this release candidate on the web site soon)
>
> The primary thing this does not yet include that we do plan on getting 
> into the final 0.5.0 release is proxy support for the package manager. A 
> preliminary version of that has been merged to master but still has some 
> build system issues that need to be worked out. We will put out a release 
> candidate 3 next week that will hopefully have this resolved, along with 
> any other major bug fixes that happen by then. If all goes well and no 
> major blocking issues come up after that, RC3 could possibly be the last 
> release candidate and promoted to final, but we will see how it goes next 
> week. Follow the progress at 
> https://github.com/JuliaLang/julia/issues/17418 and please report any 
> issues you find.
>
> -Tony
>
>

[julia-users] Re: How to set JULIA_NUM_THREADS for remote workers?

[Oliver Schulz]

Hi Jeff,

> If your remote workers are remotely local invocations of Julia and in the 
environment JULIA_NUM_THREADS has been preset, then the remote workers will 
be using that many threads

I tried it, and at least when the workers are started via SSH (using 
addprocs([host1, ...])), that doesn't seem to be the case, JULIA_NUM_THREADS 
doesn't seem to be passed on. It would actually be very helpful to be able 
to forward (or explicitly set) environment variables for remote workers.

Cheers,

Oliver


On Wednesday, August 17, 2016 at 12:16:11 AM UTC+2, Jeffrey Sarnoff wrote:
>
> Hi Oliver,
> As I understand it:
> JULIA_NUM_THREADS is an environment variable read by the local invocation 
> of Julia.  It is not a run-time passable value. If your remote workers are 
> remotely local invocations of Julia and in the environment 
> JULIA_NUM_THREADS has been preset, then the remote workers will be using 
> that many threads (if the have the cores).
>
>
> On Tuesday, August 16, 2016 at 11:52:20 AM UTC-4, Oliver Schulz wrote:
>>
>> I guess the answer is "no", then?
>>
>> On Monday, August 1, 2016 at 3:26:17 PM UTC+2, Oliver Schulz wrote:
>>>
>>> Is it possible to pass on or explicitly set JULIA_NUM_THREADS for remote 
>>> workers started via
>>>
>>> addprocs([host1, ...])
>>>
>>> ?
>>>
>>>

[julia-users] Re: Symbols as Dict keys - efficiency / memory allocation?

[Oliver Schulz]

Sure -  I was assuming that as Symbol (as an interned string) is basically 
just a pointer. So comparisons are O(1), etc. What I'd like to understand 
is, why can't it be a bitstype? Currently, it's not, so Symbol cannot be 
used in a lightweight (i.e. stack-allocated) immutable type. I assume 
there's a good reason for it, I just want to understand why.

On Tuesday, August 16, 2016 at 10:03:14 PM UTC+2, Cedric St-Jean wrote:
>
> Scott is right, storing/comparing/hashing symbols is probably all done on 
> the pointers, so it's as efficient as you can hope for.
>
> On Tuesday, August 16, 2016 at 11:44:46 AM UTC-4, Oliver Schulz wrote:
>>
>> Hi,
>>
>> I was thinking of using Symbols as keys in a Dict{Symbol,Int64}, instead 
>> of Strings. However, Symbol is not a bitstype, so there I guess there will 
>> still be a lot of individual memory allocation when working with such a 
>> Dict, right?
>>
>> Actually, why isn't Symbol a bitstype (according to 
>> https://github.com/JuliaLang/julia/issues/5880, symbols are not GCed)?
>>
>>
>> Cheers,
>>
>> Oliver
>>
>>

[julia-users] Re: How to Manipulate each character in of a string using a for loop in Julia ?

['Greg Plowman' via julia-users]

I think Jacob meant:

for char in a
   write(io, char + 1)
end

Re: [julia-users] Wrapping an array from C

[Bart Janssens]

On Tue, Aug 16, 2016 at 10:29 PM Kiran Pamnany

> If I use a composite type, Bar, instead of a bits type, then I get back an
> array of Bar, but each element is #undef. The memory for the elements of
> the array is present (sizeof(Bar) was used to allocate the space), but the
> array isn't set up correctly. How can I do this?
>
>
>
It looks like ptr is actually an array of pointers? If so, you could access
the first element using:
unsafe_load(convert(Ptr{atyp}, p[1]))

Cheers,

Bart

Re: [julia-users] DataFrame inline IO with csv in variable

[Tamas Papp]

Listing various combinations of functions in the help docs is neither
necessary nor feasible IMO, you learn idiomatic use of Julia by reading
code (especially see libraries by core team members), writing code, and
asking on the list.

I agree that it is hard to search when you don't know what you are
searching for, so just ask here. The terseness of my reply was not meant
to be impolite.

On Wed, Aug 17 2016, Colin Beckingham wrote:

> Many thanks, works perfectly. I see the question was asked before; I did do 
> a search but I guess my criteria were poor. Might be a good entry in the 
> help docs.
>
> On Tuesday, 16 August 2016 08:10:10 UTC-4, Tamas Papp wrote:
>>
>> readtable(IOBuffer(string)) 
>>
>> On Tue, Aug 16 2016, Colin Beckingham wrote: 
>>
>> > I'm trying to follow the examples in the DataFrame docs explaining the 
>> > ability to use a " csv"""...""" " format to import data from a string. 
>> > My data already exist in a string variable, but I am having a problem 
>> > expanding the string inside the "csv""" """ " expression. 
>> > I can write the string into a file and read the file into the df with 
>> > readtable(), but clearly this is wasteful unless there are other reasons 
>> to 
>> > have the raw data handy. 
>> > Is it possible to expand the contents of the string variable inside the 
>> > triple quotes? 
>>
>>

[julia-users] Re: How to Manipulate each character in of a string using a for loop in Julia ?

[Rishabh Raghunath]

Hello Jacob !!..
I get this error while trying to run your code:

Error evaluating cyclic.jl
LoadError: MethodError: `+` has no method matching +(::ASCIIString, ::Int64)
Closest candidates are:
  +(::Any, ::Any, !Matched::Any, !Matched::Any...)
  +(!Matched::Int64, ::Int64)
  +(!Matched::Complex{Bool}, ::Real)
  ...
 [inlined code] from /home/rishabh/CodeVita/ProjectJulia/cyclic.jl:6
 in anonymous at ./no file:4294967295
while loading /home/rishabh/CodeVita/ProjectJulia/cyclic.jl, in expression 
starting on line 5


On Wednesday, August 17, 2016 at 12:00:25 PM UTC+5:30, Rishabh Raghunath 
wrote:
>
>
> Hello fellow Julia Users!!
>
> How do you manipulate the individual characters comprising a string in 
> Julia using a for loop ?
> For example:
> ###
>
> a = "abcd"
>
>   for i in length(a)
>a[i]+=1
>  end
>
> print(a)
>
> 
>  I am expecting to get my EXPECTED OUTPUT as" bcde  "
>
>  BUT I get the following error:  
> ##
>
>  ERROR: MethodError: `setindex!` has no method matching 
> setindex!(::ASCIIString, ::Char, ::Int64)
>  [inlined code] from ./none:2
>  in anonymous at ./no file:4294967295
>
> ##
> I also tried using:
>
> for i in eachindex(a) instead of the for loop in the above program .. And 
> I get the same error..
>
> Please tell me what i should do to get my desired output ..
> Please respond ASAP..
> Thanks..
>

Re: [julia-users] How to Manipulate each character in of a string using a for loop in Julia ?

[Jacob Quinn]

Strings are immutable (similar to other languages). There are several
different ways to get what you want, but I tend to utilize IOBuffer a lot:

a = "abcd"
io = IOBuffer()

for char in a
write(io, a + 1)
end

println(takebuf_string(io))

-Jacob

On Wed, Aug 17, 2016 at 12:30 AM, Rishabh Raghunath  wrote:

>
> Hello fellow Julia Users!!
>
> How do you manipulate the individual characters comprising a string in
> Julia using a for loop ?
> For example:
> ###
>
> a = "abcd"
>
>   for i in length(a)
>a[i]+=1
>  end
>
> print(a)
>
> 
>  I am expecting to get my EXPECTED OUTPUT as" bcde  "
>
>  BUT I get the following error:
> ##
>
>  ERROR: MethodError: `setindex!` has no method matching
> setindex!(::ASCIIString, ::Char, ::Int64)
>  [inlined code] from ./none:2
>  in anonymous at ./no file:4294967295
>
> ##
> I also tried using:
>
> for i in eachindex(a) instead of the for loop in the above program .. And
> I get the same error..
>
> Please tell me what i should do to get my desired output ..
> Please respond ASAP..
> Thanks..
>

Re: [julia-users] DataFrame inline IO with csv in variable

[Colin Beckingham]

Many thanks, works perfectly. I see the question was asked before; I did do 
a search but I guess my criteria were poor. Might be a good entry in the 
help docs.

On Tuesday, 16 August 2016 08:10:10 UTC-4, Tamas Papp wrote:
>
> readtable(IOBuffer(string)) 
>
> On Tue, Aug 16 2016, Colin Beckingham wrote: 
>
> > I'm trying to follow the examples in the DataFrame docs explaining the 
> > ability to use a " csv"""...""" " format to import data from a string. 
> > My data already exist in a string variable, but I am having a problem 
> > expanding the string inside the "csv""" """ " expression. 
> > I can write the string into a file and read the file into the df with 
> > readtable(), but clearly this is wasteful unless there are other reasons 
> to 
> > have the raw data handy. 
> > Is it possible to expand the contents of the string variable inside the 
> > triple quotes? 
>
>

[julia-users] How to Manipulate each character in of a string using a for loop in Julia ?

[Rishabh Raghunath]

Hello fellow Julia Users!!

How do you manipulate the individual characters comprising a string in 
Julia using a for loop ?
For example:
###

a = "abcd"

  for i in length(a)
   a[i]+=1
 end

print(a)


 I am expecting to get my EXPECTED OUTPUT as" bcde  "

 BUT I get the following error:  
##

 ERROR: MethodError: `setindex!` has no method matching 
setindex!(::ASCIIString, ::Char, ::Int64)
 [inlined code] from ./none:2
 in anonymous at ./no file:4294967295

##
I also tried using:

for i in eachindex(a) instead of the for loop in the above program .. And I 
get the same error..

Please tell me what i should do to get my desired output ..
Please respond ASAP..
Thanks..