[julia-users] Problem with (lib)fontconfig

[Tony Kelman]

The package assumes the presence of libfontconfig on your path, see 
https://github.com/dcjones/Fontconfig.jl/blob/master/src/Fontconfig.jl

This is clearly not the case on Windows, which you appear to be using based on 
your C:\Users package path. The package author has not done the basic courtesy 
of putting a note in the package readme that it makes this assumption and does 
not support Windows. You should suggest adding such a note to the package 
readme with a pull request.

Though since libfontconfig is a fairly well known library, I suspect it would 
be fairly simple to use WinRPM to obtain a compiled version of the library 
automatically for Windows users.

[julia-users] Problem with (lib)fontconfig

[digxx]

When trying to load Fontconfig I get this error:

INFO: Precompiling module fontconfig...
ERROR: LoadError: error compiling __init__: could not load library 
"libfontconfig"
no error
 in include at boot.jl:261
 in include_from_node1 at loading.jl:304
 [inlined code] from none:2
 in anonymous at no file:0
 in process_options at client.jl:284
 in _start at client.jl:411
while loading C:\Users\Diger\.julia\v0.4\fontconfig\src\fontconfig.jl, in 
expression starting on line 147
ERROR: Failed to precompile fontconfig to 
C:\Users\Diger\.julia\lib\v0.4\fontconfig.ji
 in error at error.jl:21


Tried to remove and reinstall it didnt help
Any idea what the problem here is?

Re: [julia-users] Re: Anyone know how to get help with Jupyter?

[Daniel Carrera]

On 1 November 2015 at 01:40, David P. Sanders  wrote:

> You can try the IPython Gitter chat channel:
>
> https://gitter.im/ipython/ipython
>
>
Thanks.



> If your question is specifically about the IOctave kernel, then you should
> probably post an issue there.
>
> In fact, there already is one:
> https://github.com/Calysto/octave_kernel/issues/18
>


Thanks for the link. It just tested with IJulia. With IJulia I also get the
problem of only displaying the last result, so at least that issue is not
specific to IOctave. But with IJulia I can write "display()" lines, and
with IOctave they don't help. So that issue seems to be specific to IOctave.

Cheers,
Daniel.

Re: [julia-users] Re: Anyone know how to get help with Jupyter?

[Daniel Carrera]

I tried both. Thanks for trying to help.

On 1 November 2015 at 01:32, Steven G. Johnson 
wrote:

> It's display(x), not disp(x)

[julia-users] Re: copy a local variable to all parallel workers

[Andre Bieler]

Thanks for the clarification.

How do I make it available to pfunc.showData then?
(without having to explicitly send it as an argument in the function call)

The goal would be to have a large chunk of data having locally available to 
the workers
which then can independently work on it and only send the computed results.
(I cannot do shared arrays as the actual data will not be bit types)

Thanks

Re: [julia-users] Re: Order of multiple for-loops

[Glen O]

That's the long, complicated way to do it, in my opinion.

An easier approach in the long run would be to simply define a new Array 
type (RMArray (for Row Major Array), perhaps), and overload the getindex 
and setindex functions for them, etc.

Indeed, perhaps someone could make a package that does this? I'm kind of 
surprised there isn't already such a package.

On Saturday, 31 October 2015 03:54:10 UTC+10, Tomas Lycken wrote:
>
> However, Julia is an incredibly expressive language, and you could quite 
> easily write a macro that re-orders the indices for you:
>
> julia> macro rowmajor(expr)
>@assert expr.head == :ref
>
>Expr(:ref, expr.args[1], expr.args[end:-1:2]...)
>end
>
> julia> macroexpand(:(@rowmajor A[i,j]))
> :(A[j,i])
>
> Of course, this only works on one indexing expression at a time, but it 
> can be extended to work recursively through an expression tree and rewrite 
> all array indexing expressions it finds, allowing usage as
>
> @rowmajor begin
> # formulate row-major algorithm here
> end
>
> No need to change the language - just bend it to do what you need :)
>
> ​
>

[julia-users] Custom parsing of csv files in DataFrames

[Gabriel Mitchell]

I am using DataFrames function `readtable` to get my data out of CSV files. 
The column are heterogenous but most sensible things like names, dates and 
currency values (with dollar signs) . All of these are getting parsed into 
strings. But I would like the currencies to parse into a fixed point type 
(and maybe the dates into integers or a date type). Can anyone point me to 
some module(s) extending the functionality of `readtable` to get parsing of 
currency? Of dates? I read the bit of docs 
here: 
http://dataframesjl.readthedocs.org/en/latest/io.html#advanced-options-for-reading-csv-files
 
but didn't see anything that would help with my problem.

[julia-users] Re: Julia and Spark

[ssarkarayushnetdev]

Is there any implementation with streams of RDDs for Julia ? 

On Monday, April 20, 2015 at 11:54:10 AM UTC-7, wil...@gmail.com wrote:
>
> Unfortunately, Spark.jl is an incorrect RDD implementation. Instead of 
> creating transformations as independent abstraction operations with a lazy 
> evaluation, the package has all transformations immediately executed upon 
> their call. This is completely undermines whole purpose of RDD as 
> fault-tolerant parallel data structure.
>
> On Saturday, April 18, 2015 at 4:04:23 AM UTC-4, Tanmay K. Mohapatra wrote:
>>
>> There was some attempt made towards a pure Julia RDD in Spark.jl (
>> https://github.com/d9w/Spark.jl).
>> We also have DistributedArrays (
>> https://github.com/JuliaParallel/DistributedArrays.jl), Blocks (
>> https://github.com/JuliaParallel/Blocks.jl) and (
>> https://github.com/JuliaStats/DataFrames.jl).
>>
>> I wonder if it is possible to leverage any of these for a pure Julia RDD.
>> And MachineLearning.jl 
>> 
>>  or 
>> something similar could probably be the equivalent of MLib.
>>
>>
>> On Friday, April 17, 2015 at 9:24:03 PM UTC+5:30, wil...@gmail.com wrote:
>>>
>>> Of course, a Spark data access infrastructure is unbeatable, due to 
>>> mature JVM-based libraries for accessing various data sources and formats 
>>> (avro, parquet, hdfs). That includes SQL support as well. But, look at 
>>> Python and R bindings, these are just facades for JVM calls. MLLib is 
>>> written in Scala, Streaming API as well, and then all this called from 
>>> Python or R, all data transformations happen on JVM level. It would be more 
>>> efficient write code in Scala then use any non-JVM bindings. Think of 
>>> overhead for RPC and data serialization over huge volumes of data needed to 
>>> be processed and you'll understand why Dpark exists. BTW, machine learning 
>>> libraries in JVM, good luck. It only works because of large computational 
>>> resources used, but even that has its limits.
>>>
>>> On Thursday, April 16, 2015 at 6:29:58 PM UTC-4, Andrei Zh wrote:

 Julia bindings for Spark would provide much more than just RDD, they 
 will give us access to multiple big data components for streaming, machine 
 learning, SQL capabilities and much more. 

 On Friday, April 17, 2015 at 12:54:32 AM UTC+3, wil...@gmail.com wrote:
>
> However, I wonder, how hard it would be to implement RDD in Julia? It 
> looks straight forward from a RDD paper 
>  how 
> to implement it. It is a robust abstraction that can be used in any 
> parallel computation.
>
> On Thursday, April 16, 2015 at 3:32:32 AM UTC-4, Steven Sagaert wrote:
>>
>> yes that's a solid approach. For my personal julia - java 
>> integrations I also run the JVM in a separate process.
>>
>> On Wednesday, April 15, 2015 at 9:30:28 PM UTC+2, wil...@gmail.com 
>> wrote:
>>>
>>> 1) simply wrap the Spark java API via JavaCall. This is the low 
 level approach. BTW I've experimented with javaCall and found it was 
 unstable & also lacking functionality (e.g. there's no way to shutdown 
 the 
 jvm or create a pool of JVM analogous to DB connections) so that might 
 need 
 some work before trying the Spark integration.

>>>
>>> Using JavaCall is not an option, especially when JVM became 
>>> close-sourced, see https://github.com/aviks/JavaCall.jl/issues/7.
>>>
>>> Python bindings are done through Py4J, which is RPC to JVM. If you 
>>> look at the sparkR , 
>>> it is done in a same way. sparkR uses a RPC interface to communicate 
>>> with a 
>>> Netty-based Spark JVM backend that translates R calls into JVM calls, 
>>> keeps 
>>> SparkContext on a JVM side, and ships serialized data to/from R.
>>>
>>> So it is just a matter of writing Julia RPC to JVM and wrapping 
>>> necessary Spark methods in a Julia friendly way. 
>>>
>>

[julia-users] Re: Deprecation warnings using julia on Atom

[Jeremy Cavanagh]

Hi Colin,

I was having the same problems while trying to get julia to work in atom 
and was hoping that this thread would provide a solution. However, I was 
alao trying to get the hydrogen to work as well but kept getting an error 
which I could not figure out the cause so posted to an issue:

https://github.com/willwhitney/hydrogen/issues/127#issuecomment-152661805

After following this great advice not only does hydrogen run without 
errors, but, the deprecation warnings that you and I were getting also 
disappeared. I am assuming that you are working on OS X.

Hope this helps.

On Wednesday, October 28, 2015 at 12:57:43 AM UTC+1, colint...@gmail.com 
wrote:
>
> Hi all,
>
> I'm using Julia v0.4 with the Atom package, on Atom 1.0 with the packages 
> ink, julia-client, and language-julia (and I'm really enjoying this as an 
> IDE solution).
>
> I can toggle the Julia console in Atom, and enter code directly into it 
> without any errors or warnings. However, as soon as I try to evaluate a 
> line of code from the Atom editor, I get a large number of deprecation 
> warnings, either of the form:
>
> WARNING: Base.Uint8 is deprecated, use UInt8 instead.
>   likely near no file:422
>
> or
>
> WARNING: Base.Uint8 is deprecated, use UInt8 instead.
>   likely near no file:422
> in skip at /home/colin/.julia/v0.4/LNR/src/LNR.jl:171
>
> Has anyone else encountered this and is there a fix? I had a look through 
> the LNR source, and there is nothing in it that should be triggering a 
> deprecation warning, nor is there even a line 171 (it only goes up to about 
> line 130).
>
> Note, I can just ignore the deprecation warnings, and continue on working 
> without a problem, so this isn't an urgent issue. Just wondering if I've 
> stuffed up the install process somehow.
>
> Cheers,
>
> Colin
>

[julia-users] Automatic conversion from 1x1 matrix/array/vector to a real/float num.

[Patrick Kofod Mogensen]

It is currently intended, but as far as I understand, singleton dimensions 
should be dropped automatically in v0.5.

[julia-users] Re: Automatic conversion from 1x1 matrix/array/vector to a real/float num.

[DNF]

In this particular example, why not use 
rand()
instead of 
rand(1,1)
?



On Saturday, October 31, 2015 at 5:33:05 PM UTC+1, Xiaowei Song wrote:
>
> v=zeros(2,3)
> x=rand(1,1)
> v[2,1]=x
> # `convert` has no method matching convert(::Type{Float64}, 
> ::Array{Float64, 2})
>
> #work around
> function returnNum(x)
>   return x[1]
> end
> v[2,2]=returnNum(v)
>
> Has anyone encountered this situation and want a change?
> Is there any other better work around?
> Or should Julia add the automatic conversion from 1x1 matrix to a num?
>
> Thanks,
>

[julia-users] Re: Confused about doc strings and macros

[andrew cooke]

hi, thanks.

you're absolutely right - the error is there (i just wrote a long message 
saying it wasn't, and then had to delete it, because it plainly is - i am 
not sure how i missed it).

i'll do what you suggest.

thanks again,
andrew


On Saturday, 31 October 2015 11:19:33 UTC-3, Michael Hatherly wrote:
>
> For user-defined macros you need to make use of Base.@__doc__ to mark the 
> subexpression returned by your macro that should be documented. In this 
> case it’s probably the Expr(:type, ...) that the docstring should be 
> attached to rather than the subsequent method definitions.
>
> So replacing the line 
> https://github.com/andrewcooke/AutoHashEquals.jl/blob/e3e80dfb190a8f8932fcce1cbdc6e4bcf79ea520/src/AutoHashEquals.jl#L81
>  
> with
> Base.@__doc__($(esc(typ))) 
>
> should be enough to allow documenting code generated via @auto_hash_equals. 
>
>
> There should have been an error message that points to the docs for 
> @__doc__, is that not showing up?
>
> It is available in v0.4 AFAIK.
>
> — Mike
>
> On Saturday, 31 October 2015 15:53:39 UTC+2, andrew cooke wrote:
>>
>>
>> I want to use the (relatively?) new docstrings, but have data structures 
>> that look like:
>>
>> """This is MyType."""
>> @auto_hash_equals type MyType
>>attribute::Something
>> end
>>
>> where the macro comes from 
>> https://github.com/andrewcooke/AutoTypeParameters.jl
>>
>> unfortunately, this gives the error: LoadError: invalid doc expression 
>> (it seems that the docstring is being "applied" to the macro call, not the 
>> data type).
>>
>> and it's not clear to me what the best solution is.  should i modify my 
>> macros so that they can take a string, and then move the macro to before 
>> the string?  or is there something i can do that will make docstrings 
>> understand that it should be applied after the macro?
>>
>> thanks,
>> andrew
>>
>>
>>

[julia-users] Confused about doc strings and macros

[andrew cooke]

I want to use the (relatively?) new docstrings, but have data structures 
that look like:

"""This is MyType."""
@auto_hash_equals type MyType
   attribute::Something
end

where the macro comes from 
https://github.com/andrewcooke/AutoTypeParameters.jl

unfortunately, this gives the error: LoadError: invalid doc expression (it 
seems that the docstring is being "applied" to the macro call, not the data 
type).

and it's not clear to me what the best solution is.  should i modify my 
macros so that they can take a string, and then move the macro to before 
the string?  or is there something i can do that will make docstrings 
understand that it should be applied after the macro?

thanks,
andrew

[julia-users] Can anyone explain this strange behavior of @doc, and/or tell me how to deal with it?

[Dan Spielman]

I would like to write code that prints out all the docstrings in a module.
For example, here is my simple module:

"doc this module"
module TestModule

"add a to b"
function fx(a,b)
a+b
end

export fx
end

using TestModule



The following works, but is very fragile:

fnm = 0  # otherwise, get error
nms = names(TestModule)
for nm in nms
fnm = eval(nm)
docnm = @doc(fnm)
println(stringmime("text/plain",docnm))
end


It produces

doc this module


add a to b 


Without the "fnm = 0", I get the error:

LoadError: "fnm" is not defined in module Main


If I try to wrap this loop in a function, I can get multiple behaviors.
The following is the best case, and assumes that I already ran the
previous block of code:


function printDocs(mod)
nms = names(mod)

for nm in nms
fnm = eval(nm)
docnm = @doc(fnm)
println(stringmime("text/plain",docnm))
end
end
printDocs(TestModule) 


produces


add a to b


add a to b


(it lost the module, and turned it into fx)

If I run the function without previously printing the documentation, I get

LoadError: "fnm" is not defined in module Main 


So, I try

fnm = 0
printDocs(TestModule) 


Which gives me the lovely documentation for zero:


0 (zero; BrE: `/ˈzɪərəʊ/` or AmE: `/ˈziːroʊ/`) is both a number and the 
numerical digit used to represent that number in numerals. It fulfills a 
central role in mathematics as the additive identity of the integers, real 
numbers, and many other algebraic structures. As a digit, 0 is used as a 
placeholder in place value systems. Names for the number 0 in English 
include zero, nought or (US) naught (`/ˈnɔːt/`), nil, or — in contexts 
where at least one adjacent digit distinguishes it from the letter "O" — oh 
or o (`/ˈoʊ/`). Informal or slang terms for zero include zilch and zip. 
Ought and aught (/ˈɔːt/), as well as cipher, have also been used 
historically. 



If anyone can explain this, it would be very helpful.
I am using Julia 0.4.0 on a Mac running OS 10.11

Thanks,
  Dan
 

Re: [julia-users] Re: Confused about doc strings and macros

[Mauro]

Have a look at:
https://github.com/JuliaLang/julia/pull/13006

I'm not sure it has been backported yet to 0.4

On Sat, 2015-10-31 at 14:55, andrew cooke  wrote:
> gah, sorry.  linked to the wrong macro package.  i don't think this is
> particularly relevant, but it should have been
> https://github.com/andrewcooke/AutoHashEquals.jl
>
> On Saturday, 31 October 2015 10:53:39 UTC-3, andrew cooke wrote:
>>
>>
>> I want to use the (relatively?) new docstrings, but have data structures
>> that look like:
>>
>> """This is MyType."""
>> @auto_hash_equals type MyType
>>attribute::Something
>> end
>>
>> where the macro comes from
>> https://github.com/andrewcooke/AutoTypeParameters.jl
>>
>> unfortunately, this gives the error: LoadError: invalid doc expression (it
>> seems that the docstring is being "applied" to the macro call, not the data
>> type).
>>
>> and it's not clear to me what the best solution is.  should i modify my
>> macros so that they can take a string, and then move the macro to before
>> the string?  or is there something i can do that will make docstrings
>> understand that it should be applied after the macro?
>>
>> thanks,
>> andrew
>>
>>
>>

[julia-users] Re: Getting my Python code to work in Julia (PyCall and all that)

[Cedric St-Jean]

Have you tried just `@pycall module_name` ? If your module is in your 
python search-path 
 
(or the current directory), it should be found. See the examples 
.


On Saturday, October 31, 2015 at 12:33:05 PM UTC-4, jcal...@gmail.com wrote:
>
> Dear community, 
> As so many of us, I have tried binding in my own Python code into Julia. 
> Standard libraries can be called easily using PyCall. Fine and dandy, but 
> what if I wish to 
> call my own functions written in Python? Do I need to add them to some 
> path or copy them somewhere before I can call them with PyCall?
> (I am using OS X btw.) 
> Are there good tutorials on this?
> Thanks !
>

[julia-users] Anyone know how to get help with Jupyter?

[Daniel Carrera]

I apologize for asking a non-Julia question here, but for the life of me I 
cannot figure out how to contact the people working on Jupyter so I can ask 
them a question:

http://jupyter.org/index.html

I know that some Julia people are involved in Jupyter, so I hope somebody 
here knows. I am feeling a bit frustrated right now. You would think that 
they would have a visible link somewhere that told you how you can contact 
them. You would imagine that they would have a mailing list, or an IRC 
channel. I'm guessing that they probably do. But I just cannot find that 
information on their website, and I REALLY tried to find it.

Does anyone know how I can ask a question to the Jupyter people? I just 
want to ask them how I can have multiple lines of output from the same 
cell. Back when it was called IPython you used to be able to write a single 
cell with several instructions, like:

-
5+5
5+6
5+7
-

All that in the same cell. And then output would be 10, 11, 12 (all in 
different lines). Now I only get "12" -- the last line of output.

So I am trying to find the Jupyter users mailing list, or IRC channel, or 
whatever they use so I can ask my question. I would be grateful for any 
pointers.

Cheers,
Daniel.

[julia-users] Re: Anyone know how to get help with Jupyter?

[Cedric St-Jean]

Does

display(5+5)
display(5+6)
display(5+7)

work for you? I find it useful for displaying multiple plots/images.

On Saturday, October 31, 2015 at 4:57:01 PM UTC-4, Daniel Carrera wrote:
>
> I apologize for asking a non-Julia question here, but for the life of me I 
> cannot figure out how to contact the people working on Jupyter so I can ask 
> them a question:
>
> http://jupyter.org/index.html
>
> I know that some Julia people are involved in Jupyter, so I hope somebody 
> here knows. I am feeling a bit frustrated right now. You would think that 
> they would have a visible link somewhere that told you how you can contact 
> them. You would imagine that they would have a mailing list, or an IRC 
> channel. I'm guessing that they probably do. But I just cannot find that 
> information on their website, and I REALLY tried to find it.
>
> Does anyone know how I can ask a question to the Jupyter people? I just 
> want to ask them how I can have multiple lines of output from the same 
> cell. Back when it was called IPython you used to be able to write a single 
> cell with several instructions, like:
>
> -
> 5+5
> 5+6
> 5+7
> -
>
> All that in the same cell. And then output would be 10, 11, 12 (all in 
> different lines). Now I only get "12" -- the last line of output.
>
> So I am trying to find the Jupyter users mailing list, or IRC channel, or 
> whatever they use so I can ask my question. I would be grateful for any 
> pointers.
>
> Cheers,
> Daniel.
>

Re: [julia-users] Loops really faster than vectorized code?

[Yichao Yu]

On Sat, Oct 31, 2015 at 5:27 PM, Christopher Alexander
 wrote:
> Yichao,
>
> Can you explain exactly what you mean by this?
>
> "...hoist the indexing on i out of the loop over j and use SubArray or
> ArrayView instead of creating a new array everytime. "
>
> I hadn't been aware of SubArray or ArrayView until just now, and I think
> this would help me understand better how to write truly more efficient Julia
> code (although about 95% of the time even my imperfect code is blowing its
> equivalents using python/numpy out of the water).

Right now, when you write `a = b[:, c]`, `a` is a new copy of the part
of array `b` you are asking for. This is slow and is in general the
problem you want to avoid by devectorizing (i.e. one of the main
problem with "vectorized" code is that you may need to create a lot of
intermediate copies).

This is why using subarrays can help. As long as your array is bigger
than the object header, creating a subarray/arrayview should be much
faster since the data is not copied.

As for hoisting out of the loop. AFAICT, your `vx = x[:, i]` is not
changing at all when you loop over `j`. you should move it out of the
loop over `j`. i.e, sth like

```
for i in ...
vx = v[:, i]
for j in ...
# vx = v[:, i] # instead of this
end
end
```

>
> Thanks!!
>
> Chris
>
> On Saturday, October 31, 2015 at 1:00:48 PM UTC-4, Yichao Yu wrote:
>>
>> On Sat, Oct 31, 2015 at 8:23 AM,   wrote:
>> > Hi,
>> > In Julia one continuously runs into recommendations to de-vectorize ones
>> > code and instead should write loops. This is appealing as conceptually,
>> > loops tend to be more straight-forward in many cases. However, when
>> > trying
>> > to follow this, I often observed this to not be the case and
>> > vector-based
>> > code to be far superior in terms of runtime.
>> >
>> > An example is attached:
>> >
>> >
>> >
>> > function
>> >
>> > emp_L_loop(x::Array{Float64,2},f::Array{Float64,2},fct_metric_inp::Function,fct_metric_outp::Function)
>> >  #returns empirical Hoelder const L computed on the basis of sample
>> > in
>> > dat
>> >  ns = size(x,2)#number of sample pts
>> >  if ns <=1
>> >   return 0
>> >  end
>> >  L = 0.
>> >  vx = x[:,1]
>> >  vx2 = x[:,1]
>> >  vf = f[:,1]
>> >  vf2 = f[:,1]
>> >  for i=1:ns
>> >for j=1:i-1
>> >vx = x[:,i]
>> >vx2 = x[:,j]
>> > dx = fct_metric_inp(vx,vx2)
>> > if dx >0
>> > vf = f[:,i]
>> > vf2 = f[:,j]
>> > L = max(L,(fct_metric_outp(vf,vf2)) ./ dx)#Lip const
>> > estimator
>> > end
>> >end
>> >  end
>> >return L
>> >end
>> >
>>
>> This is not "devectorizing" at all. Your "loop" is creating a large
>> number of arrays and that will certainly slow things down.
>>
>> If you only want to make small change to your code above, hoist the
>> indexing on i out of the loop over j and use SubArray or ArrayView
>> instead of creating a new array everytime. (which will hopefully be
>> the default sometime in 0.5).
>>
>> If you actually want to devectorize it, write out the loop over each
>> element.
>>
>> >
>> >
>> > function
>> >
>> > emp_L_vectorized(x::Array{Float64,2},f::Array{Float64,2},fct_metric_inp::Function,fct_metric_outp::Function)
>> >  ns = size(x,2)#number of sample pts
>> >
>> >  if ns <=1
>> >   return 0
>> >  end
>> >  L = 0
>> >  for j=1:ns
>> >if j+1 < ns
>> >  xcmp = x[:,j+1:end]
>> >  fcmp = f[:,j+1:end]
>> >  nxcmp = size(xcmp,2)
>> >  X = repmat(x[:,j],1,nxcmp)
>> >  F = repmat(f[:,j],1,nxcmp)
>> >  m_rowvec=fct_metric_inp(X,xcmp) #
>> >  inds = m_rowvec .> 0
>> >  m_rowvec = (m_rowvec[inds])
>> >  diffs_f = fct_metric_outp(F,fcmp)
>> >  diffs_f = diffs_f[inds]
>> >  #diffs_f = abs(sample_f_old-f[i]) - hpa.alpha
>> >  L =max(L,maximum(diffs_f./m_rowvec));
>> >   end
>> >  end
>> > return L
>> >end
>> >
>> > #metrics-:
>> >   # --- assume they compute metrics between x[:,i] and
>> > y[:,i]
>> > for i =1...
>> >   # result is a row vector of distances
>> >vecmetric_maxnorm(x::Array{Float64,2},y::Array{Float64,2}) =
>> > maximum(abs(x-y),1)#treats matrices as collections of vectors
>> >vecmetric_maxnorm(x::Array{Float64,1},y::Array{Float64,1}) =
>> > maximum(abs(x-y))
>> >
>> > # = main code:
>> >
>> > x = 1.*rand(1,4000)
>> > f = (x) -> sqrt(x)
>> >
>> > fx = f(x)
>> >
>> >
>> > @time emp_L_loop(x,fx,vecmetric_maxnorm,vecmetric_maxnorm)
>> > # result: elapsed time: 14.265720419 seconds (6526748828 bytes
>> > allocated,
>> > 64.87% gc time)
>> >
>> >
>> > @time emp_L_vectorized(x,fx,vecmetric_maxnorm,vecmetric_maxnorm)
>> > #elapsed time: 1.912960284 seconds (860891136 bytes allocated, 67.08% gc
>> > time)
>> >
>> >
>> > I have also observed that the 

[julia-users] Re: Automatic conversion from 1x1 matrix/array/vector to a real/float num.

[Glen O]

There are a few neat ways to make it work nicely. One is to access the 
value. That is, 
v[2,1]=x[]

Another is to assign as a matrix:
v[2:2,1]=x

Automatic conversion between single-value arrays and numbers could be 
problematic, if care isn't taken. For example, what happens if v is an 
Array of type Any? Does it store the value, or does it store the array 
holding the value? Note that it matters when it comes time to modify the 
entry.

On Sunday, 1 November 2015 02:33:05 UTC+10, Xiaowei Song wrote:
>
> v=zeros(2,3)
> x=rand(1,1)
> v[2,1]=x
> # `convert` has no method matching convert(::Type{Float64}, 
> ::Array{Float64, 2})
>
> #work around
> function returnNum(x)
>   return x[1]
> end
> v[2,2]=returnNum(v)
>
> Has anyone encountered this situation and want a change?
> Is there any other better work around?
> Or should Julia add the automatic conversion from 1x1 matrix to a num?
>
> Thanks,
>

[julia-users] Re: Anyone know how to get help with Jupyter?

[Steven G. Johnson]

I usually communicate with the Jupyter people via github.

As Cedric says, you can use side effects (e.g. calls to display) to output 
multiple things.  

You can also make your cell output a tuple:

5+5,
5+6,
5+7

but this does not give nice multimedia output and the formatting leaves 
something to be desired.  I filed an issue for this, but no one has tried 
to implement it:

https://github.com/ipython/ipython/issues/6345

[julia-users] Re: Confused about doc strings and macros

[andrew cooke]

gah, sorry.  linked to the wrong macro package.  i don't think this is 
particularly relevant, but it should have been 
https://github.com/andrewcooke/AutoHashEquals.jl

On Saturday, 31 October 2015 10:53:39 UTC-3, andrew cooke wrote:
>
>
> I want to use the (relatively?) new docstrings, but have data structures 
> that look like:
>
> """This is MyType."""
> @auto_hash_equals type MyType
>attribute::Something
> end
>
> where the macro comes from 
> https://github.com/andrewcooke/AutoTypeParameters.jl
>
> unfortunately, this gives the error: LoadError: invalid doc expression (it 
> seems that the docstring is being "applied" to the macro call, not the data 
> type).
>
> and it's not clear to me what the best solution is.  should i modify my 
> macros so that they can take a string, and then move the macro to before 
> the string?  or is there something i can do that will make docstrings 
> understand that it should be applied after the macro?
>
> thanks,
> andrew
>
>
>

[julia-users] Re: Order of multiple for-loops

[DNF]

To me it makes sense to think of the loops as 'inner' and 'outer'. So here:
for i2=1:n,
   i1=1:m
   A[ i1, i2 ] = 10 * i1 + i2
 end

i1 is the 'inner' loop variable. And here:
A = [ 10 * i1 + i2  for  i2=1:n, i1=1:m ] 

i1 is the 'outer' loop variable. Makes things quite easy to think about. 
The inner loop is always fastest.

As for column vs row majority, I like that the fastest dimension is either 
the first or the last dimension, and then the first is surely easiest. 
Having the *second* dimension/index being fastest, just seems a bit 
arbitrary. The first index is also the 'inner' index in my mind.


On Friday, October 30, 2015 at 10:32:36 PM UTC+1, Tk wrote:
>
> Thanks very much for various information. And I am sorry if my question 
> might be misleading or indicate
> confusion or complaints about memory order (indeed, column-major is 
> perfectly fine for me).
> So let me explain a bit more about my question...
>
> Given that Julia's arrays are column-major order, the following loop is 
> most efficient for m x n array A:
>
>  for i2=1:n,
>i1=1:m
>A[ i1, i2 ] = 10 * i1 + i2
> end
>
> or equivalently,
>
>  for i2=1:n,  i1=1:m  # (1)
>A[ i1, i2 ] = 10 * i1 + i2
> end
>
> To create the same array using comprehension, I once thought that the 
> expression might be
> something like this
>
>  
> A = [ 10 * i1 + i2  for  i2=1:n, i1=1:m ] 
>
> # (2) 
>
> where the index most far from "for" runs faster. Then for-loops in (1) and 
> (2) would take the same form.
> In fact, Julia employs a different definition
>
> A = [ 10 * i1 + i2  for i1=1:m, i2=1:n ] # (3)
>
> where the index closer to "for" runs faster.
> So part of my question was why the definition (2) was not employed (within 
> the column-major order)
> but (3) was employed.
>
> But come to think of it..., the definition (2) is clearly not nice-looking 
> (because the indices are swapped)
> while Julia's definition in (3) is more intuitive (although the order in 
> (1) and (3) is different).
> So I guess this would be the reason for this choice (sorry if this is a 
> silly question...).
> And I now understand Stephan's comment about row-major order (thanks :)
>
> RE how to memorize the definition (3), the following form is close to (3):
>
> B = [ [ 10 * i1 + i2 for i1=1:m ] for i2=1:n ]
>
> although this gives an array of arrays. So, the definition (3) may be 
> viewed as a "rectangularized" or serialized
> version of B. This is also easy for me to memorize it (Btw, the above 
> expression is similar to implied do-loops
> in Fortran, but in fact the content is very different!)
>
> Anyway, thanks very much :)
>

[julia-users] Loops really faster than vectorized code?

[jcalliess]

Hi, 
In Julia one continuously runs into recommendations to de-vectorize ones 
code and instead should write loops. This is appealing as conceptually, 
loops tend to be more straight-forward in many cases. However, when trying 
to follow this, I often observed this to not be the case and vector-based 
code to be far superior in terms of runtime. 

An example is attached:



function 
emp_L_loop(x::Array{Float64,2},f::Array{Float64,2},fct_metric_inp::Function,fct_metric_outp::Function)
 #returns empirical Hoelder const L computed on the basis of sample in 
dat
 ns = size(x,2)#number of sample pts
 if ns <=1
  return 0
 end
 L = 0.
 vx = x[:,1]
 vx2 = x[:,1]
 vf = f[:,1]
 vf2 = f[:,1]
 for i=1:ns
   for j=1:i-1
   vx = x[:,i]
   vx2 = x[:,j]
dx = fct_metric_inp(vx,vx2)
if dx >0
vf = f[:,i]
vf2 = f[:,j]
L = max(L,(fct_metric_outp(vf,vf2)) ./ dx)#Lip const estimator
end
   end
 end
   return L
   end



function 
emp_L_vectorized(x::Array{Float64,2},f::Array{Float64,2},fct_metric_inp::Function,fct_metric_outp::Function)
 ns = size(x,2)#number of sample pts

 if ns <=1
  return 0
 end
 L = 0
 for j=1:ns
   if j+1 < ns
 xcmp = x[:,j+1:end]
 fcmp = f[:,j+1:end]
 nxcmp = size(xcmp,2)
 X = repmat(x[:,j],1,nxcmp)
 F = repmat(f[:,j],1,nxcmp)
 m_rowvec=fct_metric_inp(X,xcmp) #
 inds = m_rowvec .> 0
 m_rowvec = (m_rowvec[inds])
 diffs_f = fct_metric_outp(F,fcmp) 
 diffs_f = diffs_f[inds]
 #diffs_f = abs(sample_f_old-f[i]) - hpa.alpha
 L =max(L,maximum(diffs_f./m_rowvec));
  end
 end
return L
   end

#metrics-:
  # --- assume they compute metrics between x[:,i] and y[:,i] 
for i =1...
  # result is a row vector of distances
   vecmetric_maxnorm(x::Array{Float64,2},y::Array{Float64,2}) = 
 maximum(abs(x-y),1)#treats matrices as collections of vectors
   vecmetric_maxnorm(x::Array{Float64,1},y::Array{Float64,1}) = 
maximum(abs(x-y))

# = main code:

x = 1.*rand(1,4000)
f = (x) -> sqrt(x)

fx = f(x)


@time emp_L_loop(x,fx,vecmetric_maxnorm,vecmetric_maxnorm)
# result: elapsed time: 14.265720419 seconds (6526748828 bytes allocated, 
64.87% gc time)


@time emp_L_vectorized(x,fx,vecmetric_maxnorm,vecmetric_maxnorm)
#elapsed time: 1.912960284 seconds (860891136 bytes allocated, 67.08% gc 
time)


I have also observed that the vectorized metrics (attached) outperform the 
standard implementations on my machine. 
What am I doing wrong?




test_loopvsvec1.jl
Description: Binary data

[julia-users] Automatic conversion from 1x1 matrix/array/vector to a real/float num.

[Xiaowei Song]

v=zeros(2,3)
x=rand(1,1)
v[2,1]=x
# `convert` has no method matching convert(::Type{Float64}, 
::Array{Float64, 2})

#work around
function returnNum(x)
  return x[1]
end
v[2,2]=returnNum(v)

Has anyone encountered this situation and want a change?
Is there any other better work around?
Or should Julia add the automatic conversion from 1x1 matrix to a num?

Thanks,

Re: [julia-users] convert Array{Float,1} to Array{Float,2)?

[jcalliess]

Actually there is an easier way: a =a''
Not sure how the runtime compares though..


On Tuesday, 27 October 2015 14:01:22 UTC+2, michae...@gmail.com wrote:
>
> Thanks. I knew there had to be an easy way.
>
> On Tuesday, October 27, 2015 at 12:59:43 PM UTC+1, Yichao Yu wrote:
>>
>> On Tue, Oct 27, 2015 at 5:47 AM,   wrote: 
>> > I have a function that expects a Matrix argument, but for the case I'm 
>> > working on, the natural way to construct the argument gives a vector. 
>> So, 
>> > suppose I have a vector, e.g., a=rand(10). What is the best way to 
>> convert 
>> > this to a 10X1 array? I can do it using a loop, but I thought there 
>> might be 
>> > a built-in convertor, which I fail to find. 
>> > Thx 
>>
>> reshape(a, (10, 1)) if you don't mind the two to share the same 
>> storage and you cannot resize the first one anymore. 
>>
>

[julia-users] Re: Performance compared to Matlab

[jcalliess]

Since people seem to recommend de-vectorisation a lot -- I should comment 
on my observation that de-vectorisation in Julia often makes things 
slower... will get some code out and start a new topic on this...


On Thursday, 22 October 2015 00:56:39 UTC+3, Lindsey Kuper wrote:
>
> It's fantastic to see some good ParallelAccelerator results "in the 
> wild"!  Thanks for sharing.
>
> Lindsey
>
> On Wednesday, October 21, 2015 at 1:23:53 PM UTC-7, Kristoffer Carlsson 
> wrote:
>>
>> Btw it is really cool to see julia running at 400% CPU when running a 
>> list comprehension.
>>
>> I did some more benchmarks with larger N to reduce the noise a bit and 
>> the difference is actually not that great between Matlab and Julia. 
>> However, tying with Matlabs parallellized vectorized maps is great imho.
>>
>> julia> @time h, f = Jakes_Flat( 926, 1e-6, 500, 0, 1, 0 )
>>   0.585940 seconds (153 allocations: 495.918 MB, 12.47% gc time)
>> (
>>
>>
>> >> tic; Jakes_Flat( 926, 1E-6, 500, 0, 1, 0 ); toc
>> Elapsed time is 0.609867 seconds.
>>
>>
>>
>>
>> On Wednesday, October 21, 2015 at 10:17:18 PM UTC+2, Kristoffer Carlsson 
>> wrote:
>>>
>>> For fun (and science) I tried out the new package 
>>> https://github.com/IntelLabs/ParallelAccelerator.jl for this problem.
>>>
>>> Here is the code:
>>>
>>> function Jakes_Flat( fd, Ts, Ns, t0 = 0, E0 = 1, phi_N = 0 )
>>> # Inputs:
>>> #
>>> # Outputs:
>>>   N0  = 8;  # As suggested by Jakes
>>>   N   = 4*N0+2; # An accurate approximation
>>>   wd  = 2*pi*fd;# Maximum Doppler frequency
>>>   
>>>   ts   = collect(t0 + (0:Ns-1)*Ts)
>>>   tf  = ts[end] + Ts;
>>>   Ns = collect(1:N0)
>>>
>>>   coswt = [ cosvec(ts, wd)'; cosmat(ts, Ns, wd, N) ]
>>>   h = E0/sqrt(2*N0+1)*exp(im*[ phi_N pi/(N0+1)*(1:N0)']) * coswt
>>>   return h, tf;
>>> end
>>>
>>> @acc function cosvec(ts, wd)
>>> Float64[sqrt(2)*cos(wd*t) for t in ts]
>>> end
>>>
>>> @acc function cosmat(ts, Ns, wd, N)
>>> Float64[2*cos(wd*cos(2*pi/N*n)*t) for n in Ns, t in ts]
>>> end
>>>
>>>
>>> Benchmarking this I get:
>>>
>>> julia> @time Jakes_Flat( 926, 1e-6, 5, 0, 1, 0 )
>>>   0.004779 seconds (115 allocations: 4.965 MB)
>>>
>>> and without calling the accelerated functions (by putting @noacc in 
>>> front of the function calls, I get):
>>>
>>> julia> @time Jakes_Flat_noacc( 926, 1e-6, 5, 0, 1, 0 )
>>>   0.019072 seconds (75 allocations: 8.396 MB)
>>>
>>> The matlab code on my computer runs at:
>>>
>>> >> tic; Jakes_Flat( 926, 1E-6, 5, 0, 1, 0 ); toc
>>> Elapsed time is 0.009936 seconds.
>>>
>>> So.. great victory for ParallelAccelerator.jl?
>>>
>>> On Sunday, October 18, 2015 at 1:17:50 PM UTC+2, Vishnu Raj wrote:

 Although Julia homepage shows using Julia over Matlab gains more in 
 performance, my experience is quite opposite.
 I was trying to simulate channel evolution using Jakes Model for 
 wireless communication system.

 Matlab code is:
 function [ h, tf ] = Jakes_Flat( fd, Ts, Ns, t0, E0, phi_N )
 %JAKES_FLAT 
 %   Inputs:
 %   fd, Ts, Ns  : Doppler frequency, sampling time, number of 
 samples
 %   t0, E0  : initial time, channel power
 %   phi_N   : initial phase of the maximum Doppler frequeny
 %   sinusoid
 %
 %   Outputs:
 %   h, tf   : complex fading vector, current time

 if nargin < 6,  phi_N = 0;  end
 if nargin < 5,  E0 = 1; end
 if nargin < 4,  t0 = 0; end
 
 N0 = 8; % As suggested by Jakes
 N  = 4*N0 + 2;  % an accurate approximation
 wd = 2*pi*fd;   % Maximum Doppler frequency[rad]
 t  = t0 + [0:Ns-1]*Ts;  % Time vector
 tf = t(end) + Ts;   % Final time
 coswt = [ sqrt(2)*cos(wd*t); 2*cos(wd*cos(2*pi/N*[1:N0]')*t) ];
 h  = E0/sqrt(2*N0+1)*exp(j*[phi_N pi/(N0+1)*[1:N0]])*coswt;
 end
 Enter code here...

 My call results in :
 >> tic; Jakes_Flat( 926, 1E-6, 5, 0, 1, 0 ); toc
 Elapsed time is 0.008357 seconds.


 My corresponding Julia code is
 function Jakes_Flat( fd, Ts, Ns, t0 = 0, E0 = 1, phi_N = 0 )
 # Inputs:
 #
 # Outputs:
   N0  = 8;  # As suggested by Jakes
   N   = 4*N0+2; # An accurate approximation
   wd  = 2*pi*fd;# Maximum Doppler frequency
   t   = t0 + [0:Ns-1]*Ts;
   tf  = t[end] + Ts;
   coswt = [ sqrt(2)*cos(wd*t'); 2*cos(wd*cos(2*pi/N*[1:N0])*t') ]
   h = E0/sqrt(2*N0+1)*exp(im*[ phi_N pi/(N0+1)*[1:N0]']) * coswt
   return h, tf;
 end
 # Saved this as "jakes_model.jl"


 My first call results in 
 julia> include( "jakes_model.jl" )
 Jakes_Flat (generic function with 4 methods)

 julia> @time Jakes_Flat( 926, 1e-6, 5, 0, 1, 0 )
 elapsed time: 0.65922234 seconds (61018916 bytes allocated)

 julia> @time Jakes_Flat( 926, 1e-6, 5, 

[julia-users] Getting my Python code to work in Julia (PyCall and all that)

[jcalliess]

Dear community, 
As so many of us, I have tried binding in my own Python code into Julia. 
Standard libraries can be called easily using PyCall. Fine and dandy, but 
what if I wish to 
call my own functions written in Python? Do I need to add them to some path 
or copy them somewhere before I can call them with PyCall?
(I am using OS X btw.) 
Are there good tutorials on this?
Thanks !

[julia-users] Re: Confused about doc strings and macros

[Michael Hatherly]

For user-defined macros you need to make use of Base.@__doc__ to mark the 
subexpression returned by your macro that should be documented. In this 
case it’s probably the Expr(:type, ...) that the docstring should be 
attached to rather than the subsequent method definitions.

So replacing the line 
https://github.com/andrewcooke/AutoHashEquals.jl/blob/e3e80dfb190a8f8932fcce1cbdc6e4bcf79ea520/src/AutoHashEquals.jl#L81
 
with
Base.@__doc__($(esc(typ))) 

should be enough to allow documenting code generated via @auto_hash_equals. 

There should have been an error message that points to the docs for @__doc__, 
is that not showing up?

It is available in v0.4 AFAIK.

— Mike

On Saturday, 31 October 2015 15:53:39 UTC+2, andrew cooke wrote:
>
>
> I want to use the (relatively?) new docstrings, but have data structures 
> that look like:
>
> """This is MyType."""
> @auto_hash_equals type MyType
>attribute::Something
> end
>
> where the macro comes from 
> https://github.com/andrewcooke/AutoTypeParameters.jl
>
> unfortunately, this gives the error: LoadError: invalid doc expression (it 
> seems that the docstring is being "applied" to the macro call, not the data 
> type).
>
> and it's not clear to me what the best solution is.  should i modify my 
> macros so that they can take a string, and then move the macro to before 
> the string?  or is there something i can do that will make docstrings 
> understand that it should be applied after the macro?
>
> thanks,
> andrew
>
>
>

Re: [julia-users] Can anyone explain this strange behavior of @doc, and/or tell me how to deal with it?

[Yichao Yu]

IIRC `@doc` works in global scope for retrieving documents. What you
are seeing is just the difference in globals and locals.

see `doc/genstdlib.jl` for an example of how to iterate over documents.

On Sat, Oct 31, 2015 at 10:05 AM, Dan Spielman  wrote:
> I would like to write code that prints out all the docstrings in a module.
> For example, here is my simple module:
>
> "doc this module"
> module TestModule
>
> "add a to b"
> function fx(a,b)
> a+b
> end
>
> export fx
> end
>
> using TestModule
>
>
>
> The following works, but is very fragile:
>
> fnm = 0  # otherwise, get error
> nms = names(TestModule)
> for nm in nms
> fnm = eval(nm)
> docnm = @doc(fnm)
> println(stringmime("text/plain",docnm))
> end
>
>
> It produces
>
> doc this module
>
>
> add a to b
>
>
> Without the "fnm = 0", I get the error:
>
> LoadError: "fnm" is not defined in module Main
>

`fnm` is local to the for loop unless defined in the global scope already.

>
> If I try to wrap this loop in a function, I can get multiple behaviors.
> The following is the best case, and assumes that I already ran the
> previous block of code:
>
>
> function printDocs(mod)
> nms = names(mod)
>
> for nm in nms
> fnm = eval(nm)
> docnm = @doc(fnm)
> println(stringmime("text/plain",docnm))
> end
> end
> printDocs(TestModule)
>
>
> produces
>
>
> add a to b
>
>
> add a to b
>
>
> (it lost the module, and turned it into fx)

You are very likely just seeing the left over global variable fnm from
your previous run in the global scope.

>
> If I run the function without previously printing the documentation, I get
>
> LoadError: "fnm" is not defined in module Main
>

And this confirm my guess.

>
> So, I try
>
> fnm = 0
> printDocs(TestModule)
>

and you are seeing the global variable `fnm`

>
> Which gives me the lovely documentation for zero:
>

Isn't it lovely ;-p

>
> 0 (zero; BrE: `/ˈzɪərəʊ/` or AmE: `/ˈziːroʊ/`) is both a number and the
> numerical digit used to represent that number in numerals. It fulfills a
> central role in mathematics as the additive identity of the integers, real
> numbers, and many other algebraic structures. As a digit, 0 is used as a
> placeholder in place value systems. Names for the number 0 in English
> include zero, nought or (US) naught (`/ˈnɔːt/`), nil, or — in contexts where
> at least one adjacent digit distinguishes it from the letter "O" — oh or o
> (`/ˈoʊ/`). Informal or slang terms for zero include zilch and zip. Ought and
> aught (/ˈɔːt/), as well as cipher, have also been used historically.
>
>
>
> If anyone can explain this, it would be very helpful.
> I am using Julia 0.4.0 on a Mac running OS 10.11
>
> Thanks,
>   Dan
>

Re: [julia-users] Does anyone have a fork/branch of Sundials.jl that works on 0.4/0.5?

[Simon Frost]

Dear Alex,

Thanks for letting me know about the PR. As for the minor API differences, 
e.g. cvode in Sundials.jl returns a matrix, whereas (e.g.) ode23s returns a 
tuple with time steps and the dependent variable y separately. Not a big 
deal really, but it would be nice to avoid having to deal with two 
different types of output.

Best,
Simon

On Wednesday, October 28, 2015 at 12:30:09 PM UTC, Alex wrote:
>
> Hi Simon,
>
> Christian Haargaard just created a PR (
> https://github.com/JuliaLang/Sundials.jl/pull/56), which means that there 
> is now (or should be) a working branch.
>
> May I ask what you mean by "minor API differences"? 
>
> Best,
>
> Alex.
>
> On Saturday, 17 October 2015 10:58:10 UTC+2, Simon Frost wrote:
>
>> Dear Mauro,
>>
>> I think the fixes, at least for cvode, have already been done in one of 
>> the branches, but there isn't a PR yet. Apart from some minor API 
>> differences, which hopefully will be ironed out in the future, cvode in 
>> Sundials.jl is a lot faster than ode23s in ODE.jl.
>>
>> Best
>> Simon
>>
>

Re: [julia-users] Re: Julia and Spark

[Jey Kottalam]

Could you please define "streams of RDDs"?

On Sat, Oct 31, 2015 at 12:59 AM,  wrote:

> Is there any implementation with streams of RDDs for Julia ?
>
>
> On Monday, April 20, 2015 at 11:54:10 AM UTC-7, wil...@gmail.com wrote:
>>
>> Unfortunately, Spark.jl is an incorrect RDD implementation. Instead of
>> creating transformations as independent abstraction operations with a lazy
>> evaluation, the package has all transformations immediately executed upon
>> their call. This is completely undermines whole purpose of RDD as
>> fault-tolerant parallel data structure.
>>
>> On Saturday, April 18, 2015 at 4:04:23 AM UTC-4, Tanmay K. Mohapatra
>> wrote:
>>>
>>> There was some attempt made towards a pure Julia RDD in Spark.jl (
>>> https://github.com/d9w/Spark.jl).
>>> We also have DistributedArrays (
>>> https://github.com/JuliaParallel/DistributedArrays.jl), Blocks (
>>> https://github.com/JuliaParallel/Blocks.jl) and (
>>> https://github.com/JuliaStats/DataFrames.jl).
>>>
>>> I wonder if it is possible to leverage any of these for a pure Julia RDD.
>>> And MachineLearning.jl
>>> 
>>>  or
>>> something similar could probably be the equivalent of MLib.
>>>
>>>
>>> On Friday, April 17, 2015 at 9:24:03 PM UTC+5:30, wil...@gmail.com
>>> wrote:

 Of course, a Spark data access infrastructure is unbeatable, due to
 mature JVM-based libraries for accessing various data sources and formats
 (avro, parquet, hdfs). That includes SQL support as well. But, look at
 Python and R bindings, these are just facades for JVM calls. MLLib is
 written in Scala, Streaming API as well, and then all this called from
 Python or R, all data transformations happen on JVM level. It would be more
 efficient write code in Scala then use any non-JVM bindings. Think of
 overhead for RPC and data serialization over huge volumes of data needed to
 be processed and you'll understand why Dpark exists. BTW, machine learning
 libraries in JVM, good luck. It only works because of large computational
 resources used, but even that has its limits.

 On Thursday, April 16, 2015 at 6:29:58 PM UTC-4, Andrei Zh wrote:
>
> Julia bindings for Spark would provide much more than just RDD, they
> will give us access to multiple big data components for streaming, machine
> learning, SQL capabilities and much more.
>
> On Friday, April 17, 2015 at 12:54:32 AM UTC+3, wil...@gmail.com
> wrote:
>>
>> However, I wonder, how hard it would be to implement RDD in Julia? It
>> looks straight forward from a RDD paper
>>  how
>> to implement it. It is a robust abstraction that can be used in any
>> parallel computation.
>>
>> On Thursday, April 16, 2015 at 3:32:32 AM UTC-4, Steven Sagaert wrote:
>>>
>>> yes that's a solid approach. For my personal julia - java
>>> integrations I also run the JVM in a separate process.
>>>
>>> On Wednesday, April 15, 2015 at 9:30:28 PM UTC+2, wil...@gmail.com
>>> wrote:

 1) simply wrap the Spark java API via JavaCall. This is the low
> level approach. BTW I've experimented with javaCall and found it was
> unstable & also lacking functionality (e.g. there's no way to 
> shutdown the
> jvm or create a pool of JVM analogous to DB connections) so that 
> might need
> some work before trying the Spark integration.
>

 Using JavaCall is not an option, especially when JVM became
 close-sourced, see https://github.com/aviks/JavaCall.jl/issues/7.

 Python bindings are done through Py4J, which is RPC to JVM. If you
 look at the sparkR ,
 it is done in a same way. sparkR uses a RPC interface to communicate 
 with a
 Netty-based Spark JVM backend that translates R calls into JVM calls, 
 keeps
 SparkContext on a JVM side, and ships serialized data to/from R.

 So it is just a matter of writing Julia RPC to JVM and wrapping
 necessary Spark methods in a Julia friendly way.

>>>

Re: [julia-users] Loops really faster than vectorized code?

[Yichao Yu]

On Sat, Oct 31, 2015 at 8:23 AM,   wrote:
> Hi,
> In Julia one continuously runs into recommendations to de-vectorize ones
> code and instead should write loops. This is appealing as conceptually,
> loops tend to be more straight-forward in many cases. However, when trying
> to follow this, I often observed this to not be the case and vector-based
> code to be far superior in terms of runtime.
>
> An example is attached:
>
>
>
> function
> emp_L_loop(x::Array{Float64,2},f::Array{Float64,2},fct_metric_inp::Function,fct_metric_outp::Function)
>  #returns empirical Hoelder const L computed on the basis of sample in
> dat
>  ns = size(x,2)#number of sample pts
>  if ns <=1
>   return 0
>  end
>  L = 0.
>  vx = x[:,1]
>  vx2 = x[:,1]
>  vf = f[:,1]
>  vf2 = f[:,1]
>  for i=1:ns
>for j=1:i-1
>vx = x[:,i]
>vx2 = x[:,j]
> dx = fct_metric_inp(vx,vx2)
> if dx >0
> vf = f[:,i]
> vf2 = f[:,j]
> L = max(L,(fct_metric_outp(vf,vf2)) ./ dx)#Lip const estimator
> end
>end
>  end
>return L
>end
>

This is not "devectorizing" at all. Your "loop" is creating a large
number of arrays and that will certainly slow things down.

If you only want to make small change to your code above, hoist the
indexing on i out of the loop over j and use SubArray or ArrayView
instead of creating a new array everytime. (which will hopefully be
the default sometime in 0.5).

If you actually want to devectorize it, write out the loop over each element.

>
>
> function
> emp_L_vectorized(x::Array{Float64,2},f::Array{Float64,2},fct_metric_inp::Function,fct_metric_outp::Function)
>  ns = size(x,2)#number of sample pts
>
>  if ns <=1
>   return 0
>  end
>  L = 0
>  for j=1:ns
>if j+1 < ns
>  xcmp = x[:,j+1:end]
>  fcmp = f[:,j+1:end]
>  nxcmp = size(xcmp,2)
>  X = repmat(x[:,j],1,nxcmp)
>  F = repmat(f[:,j],1,nxcmp)
>  m_rowvec=fct_metric_inp(X,xcmp) #
>  inds = m_rowvec .> 0
>  m_rowvec = (m_rowvec[inds])
>  diffs_f = fct_metric_outp(F,fcmp)
>  diffs_f = diffs_f[inds]
>  #diffs_f = abs(sample_f_old-f[i]) - hpa.alpha
>  L =max(L,maximum(diffs_f./m_rowvec));
>   end
>  end
> return L
>end
>
> #metrics-:
>   # --- assume they compute metrics between x[:,i] and y[:,i]
> for i =1...
>   # result is a row vector of distances
>vecmetric_maxnorm(x::Array{Float64,2},y::Array{Float64,2}) =
> maximum(abs(x-y),1)#treats matrices as collections of vectors
>vecmetric_maxnorm(x::Array{Float64,1},y::Array{Float64,1}) =
> maximum(abs(x-y))
>
> # = main code:
>
> x = 1.*rand(1,4000)
> f = (x) -> sqrt(x)
>
> fx = f(x)
>
>
> @time emp_L_loop(x,fx,vecmetric_maxnorm,vecmetric_maxnorm)
> # result: elapsed time: 14.265720419 seconds (6526748828 bytes allocated,
> 64.87% gc time)
>
>
> @time emp_L_vectorized(x,fx,vecmetric_maxnorm,vecmetric_maxnorm)
> #elapsed time: 1.912960284 seconds (860891136 bytes allocated, 67.08% gc
> time)
>
>
> I have also observed that the vectorized metrics (attached) outperform the
> standard implementations on my machine.
> What am I doing wrong?
>
>

[julia-users] Re: Anyone know how to get help with Jupyter?

[David P. Sanders]

El sábado, 31 de octubre de 2015, 14:57:01 (UTC-6), Daniel Carrera escribió:
>
> I apologize for asking a non-Julia question here, but for the life of me I 
> cannot figure out how to contact the people working on Jupyter so I can ask 
> them a question:
>
> http://jupyter.org/index.html
>

You can try the IPython Gitter chat channel:

https://gitter.im/ipython/ipython

If your question is specifically about the IOctave kernel, then you should 
probably post an issue there.

In fact, there already is one:
https://github.com/Calysto/octave_kernel/issues/18
 

>
>
> I know that some Julia people are involved in Jupyter, so I hope somebody 
> here knows. I am feeling a bit frustrated right now. You would think that 
> they would have a visible link somewhere that told you how you can contact 
> them. You would imagine that they would have a mailing list, or an IRC 
> channel. I'm guessing that they probably do. But I just cannot find that 
> information on their website, and I REALLY tried to find it.
>
> Does anyone know how I can ask a question to the Jupyter people? I just 
> want to ask them how I can have multiple lines of output from the same 
> cell. Back when it was called IPython you used to be able to write a single 
> cell with several instructions, like:
>
> -
> 5+5
> 5+6
> 5+7
> -
>
> All that in the same cell. And then output would be 10, 11, 12 (all in 
> different lines). Now I only get "12" -- the last line of output.
>
> So I am trying to find the Jupyter users mailing list, or IRC channel, or 
> whatever they use so I can ask my question. I would be grateful for any 
> pointers.
>
> Cheers,
> Daniel.
>

Re: [julia-users] Re: Anyone know how to get help with Jupyter?

[Daniel Carrera]

I have added an issue/request on GitHub. I hope this is the right way to
reach the Jupyter devs. Let's see what they say.

https://github.com/ipython/ipython/issues/8940

Cheers,
Daniel.

On 1 November 2015 at 00:40, Daniel Carrera  wrote:

>
>
> On 31 October 2015 at 22:45, Steven G. Johnson 
> wrote:
>
>> I usually communicate with the Jupyter people via github.
>>
>
>
> You mean, by filing issues? Or is there some kind of forum?
>
>
>
>> As Cedric says, you can use side effects (e.g. calls to display) to
>> output multiple things.
>>
>> You can also make your cell output a tuple:
>>
>> 5+5,
>> 5+6,
>> 5+7
>>
>
> Neither option seems to work for the IOctave kernel. If I cannot solve
> this I will just divide the cell into parts. But I don't understand why
> this stopped working. It used to work with the IPython notebook.
> Previously, the code cells were just like entering commands in Octave. So
> you had to insert semicolons to suppress output.
>
> I looked through the open issues. I don't think anyone has reported this
> problem yet.
>
> Daniel.
>

Re: [julia-users] Re: Anyone know how to get help with Jupyter?

[Daniel Carrera]

I tried the Octave equivalent, "disp(5+5)" but it does not work. If it did
work, it would still not be ideal because this notebook is introductory
lecture and I prefer to minimize extraneous code.


On 31 October 2015 at 22:14, Cedric St-Jean  wrote:

> Does
>
> display(5+5)
> display(5+6)
> display(5+7)
>
> work for you? I find it useful for displaying multiple plots/images.
>
> On Saturday, October 31, 2015 at 4:57:01 PM UTC-4, Daniel Carrera wrote:
>>
>> I apologize for asking a non-Julia question here, but for the life of me
>> I cannot figure out how to contact the people working on Jupyter so I can
>> ask them a question:
>>
>> http://jupyter.org/index.html
>>
>> I know that some Julia people are involved in Jupyter, so I hope somebody
>> here knows. I am feeling a bit frustrated right now. You would think that
>> they would have a visible link somewhere that told you how you can contact
>> them. You would imagine that they would have a mailing list, or an IRC
>> channel. I'm guessing that they probably do. But I just cannot find that
>> information on their website, and I REALLY tried to find it.
>>
>> Does anyone know how I can ask a question to the Jupyter people? I just
>> want to ask them how I can have multiple lines of output from the same
>> cell. Back when it was called IPython you used to be able to write a single
>> cell with several instructions, like:
>>
>> -
>> 5+5
>> 5+6
>> 5+7
>> -
>>
>> All that in the same cell. And then output would be 10, 11, 12 (all in
>> different lines). Now I only get "12" -- the last line of output.
>>
>> So I am trying to find the Jupyter users mailing list, or IRC channel, or
>> whatever they use so I can ask my question. I would be grateful for any
>> pointers.
>>
>> Cheers,
>> Daniel.
>>
>

Re: [julia-users] Re: Anyone know how to get help with Jupyter?

[Daniel Carrera]

On 31 October 2015 at 22:45, Steven G. Johnson 
wrote:

> I usually communicate with the Jupyter people via github.
>


You mean, by filing issues? Or is there some kind of forum?



> As Cedric says, you can use side effects (e.g. calls to display) to output
> multiple things.
>
> You can also make your cell output a tuple:
>
> 5+5,
> 5+6,
> 5+7
>

Neither option seems to work for the IOctave kernel. If I cannot solve this
I will just divide the cell into parts. But I don't understand why this
stopped working. It used to work with the IPython notebook. Previously, the
code cells were just like entering commands in Octave. So you had to insert
semicolons to suppress output.

I looked through the open issues. I don't think anyone has reported this
problem yet.

Daniel.

Re: [julia-users] Re: IDE for Julia

[Charles Novaes de Santana]

It looks very nice!! The basic interface looks pretty much with Rstudio.
Thanks for sharing! (and for keeping this thread alive heehhe)

Best,

Charles

On 30 October 2015 at 16:03, Tomas Mikoviny 
wrote:

> maybe someone with more javascript insight can get inspired by this python
> IDE one day :)
>
> https://github.com/yhat/rodeo
>
>
> On Thursday, August 27, 2015 at 5:12:22 AM UTC+2, Deb Midya wrote:
>>
>> Hi,
>>
>> Thanks in advance.
>>
>> I am new to Julia and using Julia-0.3.7 on Windows 8.
>>
>> I am looking for an IDE for Julia (like RStudio in R).
>>
>> Once again, thank you very much for the time you have given..
>>
>> Regards,
>>
>> Deb
>>
>


-- 
Um axé! :)

--
Charles Novaes de Santana, PhD
http://www.imedea.uib-csic.es/~charles

Re: [julia-users] Re: Anyone know how to get help with Jupyter?

[Steven G. Johnson]

It's display(x), not disp(x)

[julia-users] Re: copy a local variable to all parallel workers

[Jonathan Malmaud]

The method 'pfunc.showData' isn't able to see the variable `data`, which 
exists as a global variable in the worker's 'Main' module, not it's 
'pfuncs' module. 

On Saturday, October 31, 2015 at 12:14:34 AM UTC-5, Andre Bieler wrote:
>
> I have a similar question about getting data to all workers. Consider the 
> following code:
>
> '''
> addprocs(2)
> using pfuncs
>
> #=
> @everywhere function showData()
>   @show(data)
> end
> =#
>
> function pstep()
>   @sync begin
> for p in workers()
> @async begin
> remotecall_fetch(p, showData)
> end
> end
>   end
> end
>
> @everywhere data = myid()
> pstep()
> '''
>
> If I uncomment the part where the showData() function is defined, it works 
> as expected. If I put the definition of showData() into the file pfuncs.jl 
> and import it as in the example above, it does not work.
>
> From the manual I figure
>
> "using DummyModule causes the module to be loaded on all processes; 
> however, the module is brought into scope only on the one executing the 
> statement."
>
> addresses the problem, but does not help me solve it.
> I also tried the let block posted by Jameson.
>
> How do I get the same behaviour as with the @everywhere function 
> definition in the commented block but being able to use modules?
>
> For completeness here is the pfuncs.jl module:
>
> '''
> module pfuncs
>   export showData
>
> function showData()
>   @show(data)
> end
>
> end
> '''
>
>
>
>