Re: [julia-users] Extreme garbage collection time
Devectorization should never slow anything down. If it does, then you have
some other problem. Here, M is a global variable, and that's probably what's
killing you. Performance tip #1:
http://docs.julialang.org/en/latest/manual/performance-tips/
--Tim
On Friday, April 03, 2015 09:43:51 AM Adam Labadorf wrote:
Hi,
I am struggling with an issue related to garbage collection taking up the
vast majority (99%) of compute time on a simple nested for loop. Code
excerpt below:
# julia version 0.3.7
# counts is an MxN matrix of Float64
# N=15000
# M=108
# h_cols and c_cols are indices \in {1:M}
using HypothesisTests, ArrayViews
ratios = Array(Float64,M)
function compute!(S,tot_i::Int64,i::Int64,j::Int64)
ratios = view(counts,:,i)./view(counts,:,j)
#for k=1:M
# ratios[k] = counts[k,i]/counts[k,j]
#end
#ratios = counts[:,i]./counts[:,j]
t = UnequalVarianceTTest(ratios[h_cols],ratios[c_cols])
S[tot_i] = (pvalue(t),i,j)
end
for i=1:N-1
@time for j=(i+1):N
tot_i += 1
compute!(S,tot_i,i,j)
end
end
The loop begins fast, output from time:
elapsed time: 1.023850054 seconds (62027220 bytes allocated)
elapsed time: 0.170916977 seconds (45785624 bytes allocated)
elapsed time: 0.171598156 seconds (45782760 bytes allocated)
elapsed time: 0.173866309 seconds (45779896 bytes allocated)
elapsed time: 0.170267172 seconds (45777032 bytes allocated)
elapsed time: 0.171754713 seconds (45774168 bytes allocated)
elapsed time: 0.170110142 seconds (45771304 bytes allocated)
elapsed time: 0.175199053 seconds (45768440 bytes allocated)
elapsed time: 0.179893161 seconds (45765576 bytes allocated)
elapsed time: 0.212172824 seconds (45762712 bytes allocated)
elapsed time: 0.252750549 seconds (45759848 bytes allocated)
elapsed time: 0.254874855 seconds (45756984 bytes allocated)
elapsed time: 0.231003319 seconds (45754120 bytes allocated)
elapsed time: 0.235060195 seconds (45751256 bytes allocated)
elapsed time: 0.235379355 seconds (45748392 bytes allocated)
elapsed time: 0.927622743 seconds (45746168 bytes allocated, 77.65% gc time)
elapsed time: 0.9132931 seconds (45742664 bytes allocated, 78.35% gc time)
But as soon as it starts doing gc the % time spent in increases almost
indefinitely, output from time much later:
elapsed time: 0.174122929 seconds (36239160 bytes allocated)
elapsed time: 18.535572658 seconds (36236168 bytes allocated, 99.22% gc
time)
elapsed time: 19.189478819 seconds (36233176 bytes allocated, 99.26% gc
time)
elapsed time: 21.812889439 seconds (36230184 bytes allocated, 99.35% gc
time)
elapsed time: 22.182467723 seconds (36227192 bytes allocated, 99.30% gc
time)
elapsed time: 0.16984 seconds (36224200 bytes allocated)
The inner loop, despite iterating over fewer and fewer indices has
massively increased the gc, and therefore overall, execution time. I have
tried many things, including creating the compute function, devectorizing
the ratios calculation (which really slowed things down), using view and
sub in various places, profiling with --trace-allocation=all but I can't
figure out why this happens or how to fix it. Doing gc for 22 seconds
obviously kills the performance, and since there are about 22M iterations
this is prohibitive. Can anyone suggest something I can try to improve the
performance here?
Thanks,
Adam
[julia-users] Spark and Julia
I am changing the subject of this thread from GSOC to Spark. I was just looking around and found this: https://github.com/d9w/Spark.jl The real question is with all the various systems out there, what is the level of abstraction that julia should work with. Julia's DataFrames is one level of abstraction, which could also transparently map to csv files (rather than doing readtable), or a database table, or an HBase table. Why would Spark users want Julia, and why would Julia users want Spark? I guess if we can nail this down - the rest of the integration is probably easy to figure out. -viral On Saturday, March 28, 2015 at 1:14:51 AM UTC+5:30, Jeff Waller wrote: Well with very little time lest (15 minutes?) I'd like to reiterate the idea of Julia-Spark. Simply put, the basic idea is too allow big data to flow into Julia via Spark.
[julia-users] Re: GPU-ODE
Do you mean writing ODE solvers in Julia that run on GPUs? If so, the answer is no. You can easily ccall CUDA/opencl solvers though. -viral On Friday, April 3, 2015 at 11:18:35 AM UTC+5:30, pauld11718 wrote: Is it possible in Julia - GPU based ODE solvers? There are ~250 Independent systems, each system has ~350 equations. So, was looking for GPU based ODE solvers. Any experiences? If yes, how? If no, any suggestion to do the same ?
[julia-users] Re: GPU-ODE
ok no problem Is julia capable yet of creating a standalone executable? If no, any time soon to expect such? On Saturday, April 4, 2015 at 12:19:56 PM UTC+5:30, Viral Shah wrote: Do you mean writing ODE solvers in Julia that run on GPUs? If so, the answer is no. You can easily ccall CUDA/opencl solvers though. -viral On Friday, April 3, 2015 at 11:18:35 AM UTC+5:30, pauld11718 wrote: Is it possible in Julia - GPU based ODE solvers? There are ~250 Independent systems, each system has ~350 equations. So, was looking for GPU based ODE solvers. Any experiences? If yes, how? If no, any suggestion to do the same ?
Re: [julia-users] Re: GPU-ODE
You can already compile your user code into the julia system image and provide that to someone. Standalone executables are not too far away. Most likely it will happen in 0.5. -viral On 04-Apr-2015, at 1:52 pm, pauld11718 pauld11...@gmail.com wrote: ok no problem Is julia capable yet of creating a standalone executable? If no, any time soon to expect such? On Saturday, April 4, 2015 at 12:19:56 PM UTC+5:30, Viral Shah wrote: Do you mean writing ODE solvers in Julia that run on GPUs? If so, the answer is no. You can easily ccall CUDA/opencl solvers though. -viral On Friday, April 3, 2015 at 11:18:35 AM UTC+5:30, pauld11718 wrote: Is it possible in Julia - GPU based ODE solvers? There are ~250 Independent systems, each system has ~350 equations. So, was looking for GPU based ODE solvers. Any experiences? If yes, how? If no, any suggestion to do the same ?
[julia-users] Unexpected behaviour when running parallelized code on single worker
There's something weird happening in my recently parallelized code. When running it without adding some worker processes first, the results are completely off and after some investigation I found that this was due to assignment operations going wrong - results of computations were assigned to different Arrays than the intended ones. A small working example illustrating the point: x1 = linspace(1, 3, 3) x2 = linspace(1, 3, 3) x3 = linspace(1, 3, 3) function getresults(x1::Array, x2::Array, x3::Array) result1 = SharedArray(Float64, (3,3,3)) result2 = similar(result1) result3 = similar(result1) @sync @parallel for a=1:3 for b=1:3 for c=1:3 result1[a,b,c] = x1[a]*x2[b]*x3[c] result2[a,b,c] = sqrt(x1[a]*x2[b]*x3[c]) result3[a,b,c] = (x1[a]*x2[b]*x3[c])^2 end end end return sdata(result1), sdata(result2), sdata(result3) end (r1,r2,r3) = getresults(x1, x2, x3) nprocs()==CPU_CORES || addprocs(CPU_CORES-1) (r1_par,r2_par,r3_par) = getresults(x1, x2, x3) When I run this on my system (v0.3.6), the parallelized version works as intended, while running the code without adding workers first gives the expected results for r1 and r3, but r2 holds the same results as r3. The behaviour in my original problem was similar, the code returns three Arrays, but running it without additional workers those Arrays all return the same contents. Is there something in the @sync or @parallel macros that causes this? How should a code be written to ensure that it works both with one and multiple cores?
[julia-users] macros generating macros in modules
I should start off, not entirely sure this is an okay thing to do with Julia. Suppose I want to create a macro that generates another macro, I can write: macro meta_macro(x) quote macro foo(y) $x+y end end end and then call it and the generated macro: @meta_macro(5) println(@foo(3)) # 8 Okay, wasn't entirely sure that was going to work, but it appears to. However, if I try to move the exact same macro code into a module, it no longer works: module meta_macro_mod export @meta_macro, @foo macro meta_macro(x) quote macro foo(y) $x+y end end end end Even without trying to invoke `@foo`, when I execute `@meta_macro`, I get: ERROR: syntax: invalid macro definition It seems like how the macro gets evaluated changes when it's in the module. If I change it to a module to just doing an 'include', everything works again.
[julia-users] mapping to colums of a matrix
Hi,
Suppose I have a function that maps an atom of type T, eg Float64, into
Vector{T}, and takes another argument n that determines its length.
What is the idiomatic/fast way of collecting the values in the columns
of a matrix?
Currently I am using this:
@doc Map elements of `x` into columns of a matrix using `f`.
Result is assumed to have the same element type as `x`. -
function maptocols{T}(f,x::Vector{T},n)
k = length(x)
b = Array(T, n, k)
for j = 1:k
b[:,j] = f(x[j],n)
end
b
end
Eg
julia maptocols((x,n) - x.*[1:n;], [1,2,3,4], 4)
4x4 Array{Int64,2}:
1 2 3 4
2 4 6 8
3 6 9 12
4 8 12 16
which is OK but of course I would prefer a one-liner if there is one
provided by the language (could not find it though).
Best,
Tamas
[julia-users] Re: Spark and Julia
On Saturday, April 4, 2015 at 2:22:38 AM UTC-4, Viral Shah wrote:
I am changing the subject of this thread from GSOC to Spark. I was just
looking around and found this:
https://github.com/d9w/Spark.jl
https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fd9w%2FSpark.jlsa=Dsntz=1usg=AFQjCNGQwuYJvzEZVDmzbAm0SsqeV2l46Q
Hey, wow, that's interesting, is this an attempt to reimplement Spark or
create a binding?
The real question is with all the various systems out there, what is the
level of abstraction that julia should work with. Julia's DataFrames is one
level of abstraction, which could also transparently map to csv files
(rather than doing readtable), or a database table, or an HBase table. Why
would Spark users want Julia, and why would Julia users want Spark? I guess
if we can nail this down - the rest of the integration is probably easy to
figure out.
As a potential user, I will try to answer in a few parts
There are currently 3 official language bindings (Java, Scala, Python) and
some unofficial ones as well
and R in the works; one thing that users would want is whatever the others
get but in the language they
desire with an abstraction similar to the other language bindings so that
examples in other languages
could be readily translated to theirs.
Whatever the abstraction turns out the be, there are at least 3 big things
that Spark offers; simplification,
speed, and lazy evaluation. The abstraction should not make that
cumbersome.
For me, the advantage of Julia is the syntax, the speed, and the connection
to all of the Julia packages
and because of that the community of Julia package authors. The advantage
of Spark is the machinery
of Spark, access to mlib and likewise the community of Spark users.
How about an example? This is simply from Spark examples -- good old
K-means. This is assuming
the Python binding because probably Julia and Python are most alike, how
would we expect this to
look using Julia?
from pyspark.mllib.clustering import KMeans
from numpy import array
from math import sqrt
# Load and parse the data
data = sc.textFile(data/mllib/kmeans_data.txt)
parsedData = data.map(lambda line: array([float(x) for x in line.split(' ')]))
# Build the model (cluster the data)
clusters = KMeans.train(parsedData, 2, maxIterations=10,
runs=10, initializationMode=random)
# Evaluate clustering by computing Within Set Sum of Squared Errors
def error(point):
center = clusters.centers[clusters.predict(point)]
return sqrt(sum([x**2 for x in (point - center)]))
WSSSE = parsedData.map(lambda point: error(point)).reduce(lambda x, y: x + y)
print(Within Set Sum of Squared Error = + str(WSSSE))
Re: [julia-users] mapping to colums of a matrix
Try the cat, hcat, vcat family of functions.
--Tim
On Saturday, April 04, 2015 05:04:21 PM Tamas Papp wrote:
Hi,
Suppose I have a function that maps an atom of type T, eg Float64, into
Vector{T}, and takes another argument n that determines its length.
What is the idiomatic/fast way of collecting the values in the columns
of a matrix?
Currently I am using this:
@doc Map elements of `x` into columns of a matrix using `f`.
Result is assumed to have the same element type as `x`. -
function maptocols{T}(f,x::Vector{T},n)
k = length(x)
b = Array(T, n, k)
for j = 1:k
b[:,j] = f(x[j],n)
end
b
end
Eg
julia maptocols((x,n) - x.*[1:n;], [1,2,3,4], 4)
4x4 Array{Int64,2}:
1 2 3 4
2 4 6 8
3 6 9 12
4 8 12 16
which is OK but of course I would prefer a one-liner if there is one
provided by the language (could not find it though).
Best,
Tamas
[julia-users] [ANN] JuliaIO and FileIO
Hi there,
FileIO has the aim to make it very easy to read any arbitrary file.
I hastily copied together a proof of concept by taking code from Images.jl.
JuliaIO is the umbrella group, which takes IO packages with no home. If
someone wrote an IO package, but doesn't have time to implement the FileIO
interface, giving it to JuliaIO might be a good idea in order to keep the
package usable.
Concept of FileIO is described in the readme:
Meta package for FileIO. Purpose is to open a file and return the
respective Julia object, without doing any research on how to open the file.
f = filetest.jpg # - File{:jpg}read(f) # - Imageread(filetest.obj) # -
Meshread(filetest.csv) # - DataFrame
So far only Images are supported and MeshIO is on the horizon.
It is structured the following way: There are three levels of abstraction,
first FileIO, defining the file_str macro etc, then a meta package for a
certain class of file, e.g. Images or Meshes. This meta package defines the
Julia datatype (e.g. Mesh, Image) and organizes the importer libraries.
This is also a good place to define IO library independant tests for
different file formats. Then on the last level, there are the low-level
importer libraries, which do the actual IO. They're included via Mike Innes
Requires https://github.com/one-more-minute/Requires.jl package, so that
it doesn't introduce extra load time if not needed. This way, using FileIO
without reading/writing anything should have short load times.
As an implementation example please look at FileIO - ImageIO -
ImageMagick. This should already work as a proof of concept. Try:
using FileIO # should be very fast, thanks to Mike Innes Requires
packageread(filetest.jpg) # takes a little longer as it needs to load the IO
libraryread(filetest.jpg) # should be fastread(File(documents, images,
myimage.jpg) # automatic joinpath via File constructor
Please open issues if things are not clear or if you find flaws in the
concept/implementation.
If you're interested in working on this infrastructure I'll be pleased to
add you to the group JuliaIO.
Best,
Simon
Re: [julia-users] Extreme garbage collection time
Silly me, ignoring all the commented out lines assuming they were
comments...yes, this is almost certainly it.
On Saturday, April 4, 2015 at 3:24:50 AM UTC-5, Tim Holy wrote:
Devectorization should never slow anything down. If it does, then you have
some other problem. Here, M is a global variable, and that's probably
what's
killing you. Performance tip #1:
http://docs.julialang.org/en/latest/manual/performance-tips/
--Tim
On Friday, April 03, 2015 09:43:51 AM Adam Labadorf wrote:
Hi,
I am struggling with an issue related to garbage collection taking up
the
vast majority (99%) of compute time on a simple nested for loop. Code
excerpt below:
# julia version 0.3.7
# counts is an MxN matrix of Float64
# N=15000
# M=108
# h_cols and c_cols are indices \in {1:M}
using HypothesisTests, ArrayViews
ratios = Array(Float64,M)
function compute!(S,tot_i::Int64,i::Int64,j::Int64)
ratios = view(counts,:,i)./view(counts,:,j)
#for k=1:M
# ratios[k] = counts[k,i]/counts[k,j]
#end
#ratios = counts[:,i]./counts[:,j]
t = UnequalVarianceTTest(ratios[h_cols],ratios[c_cols])
S[tot_i] = (pvalue(t),i,j)
end
for i=1:N-1
@time for j=(i+1):N
tot_i += 1
compute!(S,tot_i,i,j)
end
end
The loop begins fast, output from time:
elapsed time: 1.023850054 seconds (62027220 bytes allocated)
elapsed time: 0.170916977 seconds (45785624 bytes allocated)
elapsed time: 0.171598156 seconds (45782760 bytes allocated)
elapsed time: 0.173866309 seconds (45779896 bytes allocated)
elapsed time: 0.170267172 seconds (45777032 bytes allocated)
elapsed time: 0.171754713 seconds (45774168 bytes allocated)
elapsed time: 0.170110142 seconds (45771304 bytes allocated)
elapsed time: 0.175199053 seconds (45768440 bytes allocated)
elapsed time: 0.179893161 seconds (45765576 bytes allocated)
elapsed time: 0.212172824 seconds (45762712 bytes allocated)
elapsed time: 0.252750549 seconds (45759848 bytes allocated)
elapsed time: 0.254874855 seconds (45756984 bytes allocated)
elapsed time: 0.231003319 seconds (45754120 bytes allocated)
elapsed time: 0.235060195 seconds (45751256 bytes allocated)
elapsed time: 0.235379355 seconds (45748392 bytes allocated)
elapsed time: 0.927622743 seconds (45746168 bytes allocated, 77.65% gc
time)
elapsed time: 0.9132931 seconds (45742664 bytes allocated, 78.35% gc
time)
But as soon as it starts doing gc the % time spent in increases almost
indefinitely, output from time much later:
elapsed time: 0.174122929 seconds (36239160 bytes allocated)
elapsed time: 18.535572658 seconds (36236168 bytes allocated, 99.22% gc
time)
elapsed time: 19.189478819 seconds (36233176 bytes allocated, 99.26% gc
time)
elapsed time: 21.812889439 seconds (36230184 bytes allocated, 99.35% gc
time)
elapsed time: 22.182467723 seconds (36227192 bytes allocated, 99.30% gc
time)
elapsed time: 0.16984 seconds (36224200 bytes allocated)
The inner loop, despite iterating over fewer and fewer indices has
massively increased the gc, and therefore overall, execution time. I
have
tried many things, including creating the compute function,
devectorizing
the ratios calculation (which really slowed things down), using view and
sub in various places, profiling with --trace-allocation=all but I can't
figure out why this happens or how to fix it. Doing gc for 22 seconds
obviously kills the performance, and since there are about 22M
iterations
this is prohibitive. Can anyone suggest something I can try to improve
the
performance here?
Thanks,
Adam
[julia-users] Error messages related to macros
Hi -- I've only used Julia for about three months now, but I think it's great; Thanks to everyone involved in its creation! I've also accumulated a set of questions during this time that I'm finally getting around to asking... So first question: Are there plans to improve the messages related to errors that occur while using macros? Obviously error messages in general can be a difficult engineering problem, but Julia's error messages related to macros are particularly problematic. At one point I tried to catalog many different error scenarios, which I collected here: Macro Errors http://nbviewer.ipython.org/github/lilinjn/lilKanren/blob/master/crumpled-paper/julialang/JLangMacroErrorMessages.ipynb. The notebook is perhaps a little too disorganized to follow completely, but essentially it explores many different cases where, for example, the reported line numbers don't seem correlated to anything useful (they are just plain wrong). Here is my explanation of why this is important to me. There are a few packages that attempt to provide powerful macros that are meant to be a) fairly general purpose and b) typically used to wrap fairly large expressions (as opposed to, say, @show). Examples are @match from Match.jl and maybe @as from Lazy.jl . So for instance, when I first came across Match.jl, it seemed perfect for quickly writing simple interpreters or code generators that used some subset (or all) of Julia's syntax. But basically this ends up being difficult in practice due to the error messages -- tiny typos generate opaque messages that make progress difficult. I've made a (somewhat feeble) attempt to understand what the issues might be. As far as I can tell I don't believe it's related to optimizations that happen during compilation. Rather the Julia macro system (very nicely!) tries to mitigate hygiene issues by auto-gensym'ing (not sure what else to call it) unescaped locals in the expression returned by the macro. This is not an easy analysis and it happens in the bowels of femtolisp, and somewhere in this process, the line number tracking/error context information is getting garbled (I could be totally wrong about this...). The combination of theory of programming languages + numerical computation + systems-level core(e.g.libuv,libunwind) trifecta that is Julia makes my heart sing, and making sure Julia remains much more than a better matlab is very important to me, so, obviously I'd like to help fix things, but I'm not entirely sure I know how to fix this particular issue. I'm curious if others (the authors of Match.jl or Lazy.jl?) feel the same as I do about this, and would appreciate any comments on the topic. Cheers, nehal
Re: [julia-users] Re: GPU-ODE
To complile it in sysimg you need to use the script in: https://github.com/JuliaLang/julia/blob/master/contrib/build_executable.jl I have only used it for building a small application.
[julia-users] Re: GPU-ODE
You could try cuda-sim using PyCall: http://www.theosysbio.bio.ic.ac.uk/resources/cuda-sim/ I will try to see what we can do about getting this to work with Julia in a more streamlined fashion. Michael. On Friday, April 3, 2015 at 6:48:35 AM UTC+1, pauld11718 wrote: Is it possible in Julia - GPU based ODE solvers? There are ~250 Independent systems, each system has ~350 equations. So, was looking for GPU based ODE solvers. Any experiences? If yes, how? If no, any suggestion to do the same ?
[julia-users] Re: [ANN] JuliaIO and FileIO
That' s a very nice idea. Having a common way to load files with different
backends is very neat and very useful. Even the idea of having a file_str
macro is a very Julian way to do thing, I believe.
Maybe FastaIO could benefit from this model (and also other parsers for
biological data). We should contact BioJulia and FastaIO guys to see what
can be done.
Em sábado, 4 de abril de 2015 12:41:14 UTC-3, Simon Danisch escreveu:
Hi there,
FileIO has the aim to make it very easy to read any arbitrary file.
I hastily copied together a proof of concept by taking code from Images.jl.
JuliaIO is the umbrella group, which takes IO packages with no home. If
someone wrote an IO package, but doesn't have time to implement the FileIO
interface, giving it to JuliaIO might be a good idea in order to keep the
package usable.
Concept of FileIO is described in the readme:
Meta package for FileIO. Purpose is to open a file and return the
respective Julia object, without doing any research on how to open the file.
f = filetest.jpg # - File{:jpg}read(f) # - Imageread(filetest.obj) # -
Meshread(filetest.csv) # - DataFrame
So far only Images are supported and MeshIO is on the horizon.
It is structured the following way: There are three levels of abstraction,
first FileIO, defining the file_str macro etc, then a meta package for a
certain class of file, e.g. Images or Meshes. This meta package defines the
Julia datatype (e.g. Mesh, Image) and organizes the importer libraries.
This is also a good place to define IO library independant tests for
different file formats. Then on the last level, there are the low-level
importer libraries, which do the actual IO. They're included via Mike Innes
Requires https://github.com/one-more-minute/Requires.jl package, so
that it doesn't introduce extra load time if not needed. This way, using
FileIO without reading/writing anything should have short load times.
As an implementation example please look at FileIO - ImageIO -
ImageMagick. This should already work as a proof of concept. Try:
using FileIO # should be very fast, thanks to Mike Innes Requires
packageread(filetest.jpg) # takes a little longer as it needs to load the
IO libraryread(filetest.jpg) # should be fastread(File(documents,
images, myimage.jpg) # automatic joinpath via File constructor
Please open issues if things are not clear or if you find flaws in the
concept/implementation.
If you're interested in working on this infrastructure I'll be pleased to
add you to the group JuliaIO.
Best,
Simon
[julia-users] Re: macros generating macros in modules
The issue I'm dealing with is that I have a macro that I want to pass a list of functions that the macro can employ for parsing grammars. The macro itself exists already, and it currently has its own static list of functions. I'm trying to figure out a way to allow the user of the library to customize it. For example: @grammar foo begin start = list(integer, r, ) end where `list` is a user supplied function. The issue that I keep running into is that there doesn't seem to be any easy way of just passing in a function to a macro. Turning an expression into a function is problematic, because `eval`s are: (a) only work in the modules namespace, and (b) are frowned on. I've thought about manually looking up the symbol name in the various namespaces, but while I've found methods to list the functions, I haven't found a way to do the reverse. Ideally, it would be nice to generate an expression with `call` that took a symbol and a namespace, that way no lookups would have to be done. Having hit a dead end in this direction, it occurred to me that it might be possible to do something like this: @set_parsers(list, etc) @grammar foo begin ... end where '@set_parsers` would generate the `@grammar` macro with the static list it already has plus whatever else the user adds in. Which works, except for the whole module issue. Thanks, A On Saturday, April 4, 2015 at 12:41:27 PM UTC-4, Patrick O'Leary wrote: On Saturday, April 4, 2015 at 9:04:10 AM UTC-5, Abe Schneider wrote: I should start off, not entirely sure this is an okay thing to do with Julia. Suppose I want to create a macro that generates another macro... I'm not sure whether this should work or not, but either way I'm not sure how it's any more expressive than macros alone? Can you describe what it is you want to do at a high level in a little more detail? There might be another way.
Re: [julia-users] Re: R's splineDesign in Julia
Hi Tamas, I'm not familiar with R's splineDesign, so I'm not sure this helps, but you can get internal spline coefficients in Dierckx with `get_coeffs(spl)` where `spl` is a Spline1D object. Or look into the internals here: https://github.com/kbarbary/Dierckx.jl/blob/master/src/Dierckx.jl#L83 . - Kyle On Sat, Apr 4, 2015 at 12:54 PM, Tamas Papp tkp...@gmail.com wrote: I have looked at Dierckx.jl, but could not figure out how to evaluate the B-spline basis (all the functions in the documentation appear to do fitting directly). Can you please give me a hint? Thanks, Tamas On Sat, Apr 04 2015, pauld11718 pauld11...@gmail.com wrote: https://github.com/kbarbary/Dierckx.jl https://docs.scipy.org/doc/scipy-0.15.1/reference/interpolate.html (use with PyCall) https://github.com/lgautier/Rif.jl (R and Julia Interfacing) On Sunday, April 5, 2015 at 12:57:28 AM UTC+5:30, Tamas Papp wrote: Hi, I am looking for a function (library) that allows evaluation of a B-spline basis (specified by the knots and the order), at given points, _and_ also allows the evaluation of derivatives for the same family. R's splineDesign in the splines package has this, but looking at the splines libraries in Julia I could not find one that does B-splines and derivatives. Using internals from some library would be fine, too. (Knowing that there is no such thing at the moment would also be useful, I might implement it, but would prefer to avoid that because B-spline special cases are tricky). Best, Tamas
Re: [julia-users] Re: macros generating macros in modules
I don't understand how what I'm trying to do falls outside the domain of transforming code. My main goal is to take a user-defined grammar and create relevant code. More to the point, I think the general ability to generate expressions based on functions/namespaces is also solidly in the domain of transforms. I can imagine plenty of use-cases for language-bindings, communication, etc. Yes, there are many simple macros one can create, which are useful, but there exists many useful things one can do with macros that can get more complex. A On Saturday, April 4, 2015 at 10:11:25 PM UTC-4, Jameson wrote: I think the underlying issue may be that you are trying to have the macro do too much, when you should instead be just doing source-code transforms and preprocessing. One such example of a great use of a macro is simply: macro special_foo(syntax_tree) return quote special_foo( $(QuoteNode(syntax_tree) ) end end On Sat, Apr 4, 2015 at 8:05 PM Abe Schneider abe.sc...@gmail.com javascript: wrote: The issue I'm dealing with is that I have a macro that I want to pass a list of functions that the macro can employ for parsing grammars. The macro itself exists already, and it currently has its own static list of functions. I'm trying to figure out a way to allow the user of the library to customize it. For example: @grammar foo begin start = list(integer, r, ) end where `list` is a user supplied function. The issue that I keep running into is that there doesn't seem to be any easy way of just passing in a function to a macro. Turning an expression into a function is problematic, because `eval`s are: (a) only work in the modules namespace, and (b) are frowned on. I've thought about manually looking up the symbol name in the various namespaces, but while I've found methods to list the functions, I haven't found a way to do the reverse. Ideally, it would be nice to generate an expression with `call` that took a symbol and a namespace, that way no lookups would have to be done. Having hit a dead end in this direction, it occurred to me that it might be possible to do something like this: @set_parsers(list, etc) @grammar foo begin ... end where '@set_parsers` would generate the `@grammar` macro with the static list it already has plus whatever else the user adds in. Which works, except for the whole module issue. Thanks, A On Saturday, April 4, 2015 at 12:41:27 PM UTC-4, Patrick O'Leary wrote: On Saturday, April 4, 2015 at 9:04:10 AM UTC-5, Abe Schneider wrote: I should start off, not entirely sure this is an okay thing to do with Julia. Suppose I want to create a macro that generates another macro... I'm not sure whether this should work or not, but either way I'm not sure how it's any more expressive than macros alone? Can you describe what it is you want to do at a high level in a little more detail? There might be another way.
[julia-users] Re: Error messages related to macros
This doesn't address all of your questions, but it's definitely a recognized issue that the error messages from macros leave much to be desired. See the discussions at: https://github.com/JuliaLang/julia/issues/1334 https://github.com/JuliaLang/julia/pull/6910 https://github.com/JuliaLang/julia/issues/8701 On Saturday, April 4, 2015 at 11:33:08 AM UTC-4, Nehal Patel wrote: Hi -- I've only used Julia for about three months now, but I think it's great; Thanks to everyone involved in its creation! I've also accumulated a set of questions during this time that I'm finally getting around to asking... So first question: Are there plans to improve the messages related to errors that occur while using macros? Obviously error messages in general can be a difficult engineering problem, but Julia's error messages related to macros are particularly problematic. At one point I tried to catalog many different error scenarios, which I collected here: Macro Errors http://nbviewer.ipython.org/github/lilinjn/lilKanren/blob/master/crumpled-paper/julialang/JLangMacroErrorMessages.ipynb. The notebook is perhaps a little too disorganized to follow completely, but essentially it explores many different cases where, for example, the reported line numbers don't seem correlated to anything useful (they are just plain wrong). Here is my explanation of why this is important to me. There are a few packages that attempt to provide powerful macros that are meant to be a) fairly general purpose and b) typically used to wrap fairly large expressions (as opposed to, say, @show). Examples are @match from Match.jl and maybe @as from Lazy.jl . So for instance, when I first came across Match.jl, it seemed perfect for quickly writing simple interpreters or code generators that used some subset (or all) of Julia's syntax. But basically this ends up being difficult in practice due to the error messages -- tiny typos generate opaque messages that make progress difficult. I've made a (somewhat feeble) attempt to understand what the issues might be. As far as I can tell I don't believe it's related to optimizations that happen during compilation. Rather the Julia macro system (very nicely!) tries to mitigate hygiene issues by auto-gensym'ing (not sure what else to call it) unescaped locals in the expression returned by the macro. This is not an easy analysis and it happens in the bowels of femtolisp, and somewhere in this process, the line number tracking/error context information is getting garbled (I could be totally wrong about this...). The combination of theory of programming languages + numerical computation + systems-level core(e.g.libuv,libunwind) trifecta that is Julia makes my heart sing, and making sure Julia remains much more than a better matlab is very important to me, so, obviously I'd like to help fix things, but I'm not entirely sure I know how to fix this particular issue. I'm curious if others (the authors of Match.jl or Lazy.jl?) feel the same as I do about this, and would appreciate any comments on the topic. Cheers, nehal
[julia-users] Re: derivatives of Chebyshev polynomials
I have a function that will compute the derivative matrix operator that transforms coefficients of a Chebyshev interpolant on interval [a, b] of degree n (so n+1 total Chebyshev basis functions T_0, T_1, … T_n) into the coefficients of a degree n-1 chebyshev interpolant on [a, b] that is the exact derivative of the first interpolant. Maybe it could help you: # derivative matrix function der_matrix(deg::Int, a::Real=-1, b::Real=1) N = deg D = zeros(Float64, N, N+1) for i=1:N, j=1:N+1 if i == 1 if iseven(i + j) continue end D[i, j] = 2*(j-1)/(b-a) else if j i || iseven(i+j) continue end D[i, j] = 4*(j-1)/(b-a) end end D end On Thursday, April 2, 2015 at 10:38:48 AM UTC-4, Tamas Papp wrote: Hi, Can someone point me to some Julia code that calculates a matrix for the derivatives of the Chebyshev polynomials T_j, at given values, ie d^k T_i(x_j) / dx^kfor i=1,..n, j for some and vector x. The Chebyshev polynomials themselves are very easy to calculate using the recurrence relation, but derivatives are not. Alternatively, maybe this can be extracted from ApproxFun but I have not found a way. Best, Tamas
[julia-users] Re: derivatives of Chebyshev polynomials
Oh I forgot to say how to use it. If c are your coefficients for a degree n interpolant on [a, b] then the coefficients of the derivative of that interpolant are der_matrix(n, a, b) * c On Saturday, April 4, 2015 at 11:39:49 PM UTC-4, Spencer Lyon wrote: I have a function that will compute the derivative matrix operator that transforms coefficients of a Chebyshev interpolant on interval [a, b] of degree n (so n+1 total Chebyshev basis functions T_0, T_1, … T_n) into the coefficients of a degree n-1 chebyshev interpolant on [a, b] that is the exact derivative of the first interpolant. Maybe it could help you: # derivative matrix function der_matrix(deg::Int, a::Real=-1, b::Real=1) N = deg D = zeros(Float64, N, N+1) for i=1:N, j=1:N+1 if i == 1 if iseven(i + j) continue end D[i, j] = 2*(j-1)/(b-a) else if j i || iseven(i+j) continue end D[i, j] = 4*(j-1)/(b-a) end end D end On Thursday, April 2, 2015 at 10:38:48 AM UTC-4, Tamas Papp wrote: Hi, Can someone point me to some Julia code that calculates a matrix for the derivatives of the Chebyshev polynomials T_j, at given values, ie d^k T_i(x_j) / dx^kfor i=1,..n, j for some and vector x. The Chebyshev polynomials themselves are very easy to calculate using the recurrence relation, but derivatives are not. Alternatively, maybe this can be extracted from ApproxFun but I have not found a way. Best, Tamas
Re: [julia-users] Re: macros generating macros in modules
I think the underlying issue may be that you are trying to have the macro do too much, when you should instead be just doing source-code transforms and preprocessing. One such example of a great use of a macro is simply: macro special_foo(syntax_tree) return quote special_foo( $(QuoteNode(syntax_tree) ) end end On Sat, Apr 4, 2015 at 8:05 PM Abe Schneider abe.schnei...@gmail.com wrote: The issue I'm dealing with is that I have a macro that I want to pass a list of functions that the macro can employ for parsing grammars. The macro itself exists already, and it currently has its own static list of functions. I'm trying to figure out a way to allow the user of the library to customize it. For example: @grammar foo begin start = list(integer, r, ) end where `list` is a user supplied function. The issue that I keep running into is that there doesn't seem to be any easy way of just passing in a function to a macro. Turning an expression into a function is problematic, because `eval`s are: (a) only work in the modules namespace, and (b) are frowned on. I've thought about manually looking up the symbol name in the various namespaces, but while I've found methods to list the functions, I haven't found a way to do the reverse. Ideally, it would be nice to generate an expression with `call` that took a symbol and a namespace, that way no lookups would have to be done. Having hit a dead end in this direction, it occurred to me that it might be possible to do something like this: @set_parsers(list, etc) @grammar foo begin ... end where '@set_parsers` would generate the `@grammar` macro with the static list it already has plus whatever else the user adds in. Which works, except for the whole module issue. Thanks, A On Saturday, April 4, 2015 at 12:41:27 PM UTC-4, Patrick O'Leary wrote: On Saturday, April 4, 2015 at 9:04:10 AM UTC-5, Abe Schneider wrote: I should start off, not entirely sure this is an okay thing to do with Julia. Suppose I want to create a macro that generates another macro... I'm not sure whether this should work or not, but either way I'm not sure how it's any more expressive than macros alone? Can you describe what it is you want to do at a high level in a little more detail? There might be another way.
Re: [julia-users] Re: ANN: Match.jl v0.1.0 released
I posted a small problematic test case Kevin. On Saturday, April 4, 2015 at 2:06:38 PM UTC-7, William Macready wrote: I'm using Julia Version 0.3.7-pre+19. Later today, I'll build a small test and open an issue. thanks Bill
[julia-users] Re: holoviews in julia
I just checked out the holoviews. This is awsome. may be @pyimport will help? On Saturday, 4 April 2015 19:47:02 UTC-4, Philipp Rudiger wrote: Great to see this is getting some interest. I'm one of the authors of HoloViews and put together that initial (and poor) attempt at a Julia integration. If someone is interested in developing this further, I'd be glad to provide assistance in putting together a proper interface, although I'm afraid I won't be much help on the Julia end. Philipp On Saturday, 4 April 2015 00:23:46 UTC+1, Andrei Berceanu wrote: I just found this as a starting point http://philippjfr.com/blog/interfacing-holoviews-with-julia/ On Saturday, April 4, 2015 at 1:21:00 AM UTC+2, Andrei Berceanu wrote: Are you guys familiar with http://ioam.github.io/holoviews ? Is anyone working on a Julia interface or port? //A
[julia-users] Re: holoviews in julia
I just checked out the holoviews. This is awsome. may be @pyimport will help? On Saturday, 4 April 2015 19:47:02 UTC-4, Philipp Rudiger wrote: Great to see this is getting some interest. I'm one of the authors of HoloViews and put together that initial (and poor) attempt at a Julia integration. If someone is interested in developing this further, I'd be glad to provide assistance in putting together a proper interface, although I'm afraid I won't be much help on the Julia end. Philipp On Saturday, 4 April 2015 00:23:46 UTC+1, Andrei Berceanu wrote: I just found this as a starting point http://philippjfr.com/blog/interfacing-holoviews-with-julia/ On Saturday, April 4, 2015 at 1:21:00 AM UTC+2, Andrei Berceanu wrote: Are you guys familiar with http://ioam.github.io/holoviews ? Is anyone working on a Julia interface or port? //A
[julia-users] Re: Spark and Julia
Spark integration is a tricky thing. Python and R bindings go in a great
length to map language specific functions into Spark JVM library calls. I
guess same could be done with JavaCall.jl package in a manner similar to
SparkR. Look at slide 20 from here:
http://spark-summit.org/wp-content/uploads/2014/07/SparkR-SparkSummit.pdf.
Spark is a clever distributed data access paradigm which grew from Hadoop
slowness and limitations. I believe that Julia could provide competitive
model for a distributed data storage given Julia's parallel computing
approach. Right now, I am writing Julia bindings for Mesos. The idea is to
provide, though ClusterManager, access to any Mesos-supervised distributed
system and run Julia code that environment. In conjuction with
DistributedArrays and DataFrames, it will create powerful toolbox for
building distributed systems.
After all, machine learning on JVM, really?!.
On Saturday, April 4, 2015 at 11:21:35 AM UTC-4, Jeff Waller wrote:
On Saturday, April 4, 2015 at 2:22:38 AM UTC-4, Viral Shah wrote:
I am changing the subject of this thread from GSOC to Spark. I was just
looking around and found this:
https://github.com/d9w/Spark.jl
https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fd9w%2FSpark.jlsa=Dsntz=1usg=AFQjCNGQwuYJvzEZVDmzbAm0SsqeV2l46Q
Hey, wow, that's interesting, is this an attempt to reimplement Spark or
create a binding?
The real question is with all the various systems out there, what is the
level of abstraction that julia should work with. Julia's DataFrames is one
level of abstraction, which could also transparently map to csv files
(rather than doing readtable), or a database table, or an HBase table. Why
would Spark users want Julia, and why would Julia users want Spark? I guess
if we can nail this down - the rest of the integration is probably easy to
figure out.
As a potential user, I will try to answer in a few parts
There are currently 3 official language bindings (Java, Scala, Python) and
some unofficial ones as well
and R in the works; one thing that users would want is whatever the others
get but in the language they
desire with an abstraction similar to the other language bindings so that
examples in other languages
could be readily translated to theirs.
Whatever the abstraction turns out the be, there are at least 3 big things
that Spark offers; simplification,
speed, and lazy evaluation. The abstraction should not make that
cumbersome.
For me, the advantage of Julia is the syntax, the speed, and the
connection to all of the Julia packages
and because of that the community of Julia package authors. The advantage
of Spark is the machinery
of Spark, access to mlib and likewise the community of Spark users.
How about an example? This is simply from Spark examples -- good old
K-means. This is assuming
the Python binding because probably Julia and Python are most alike, how
would we expect this to
look using Julia?
from pyspark.mllib.clustering import KMeans
from numpy import array
from math import sqrt
# Load and parse the data
data = sc.textFile(data/mllib/kmeans_data.txt)
parsedData = data.map(lambda line: array([float(x) for x in line.split(' ')]))
# Build the model (cluster the data)
clusters = KMeans.train(parsedData, 2, maxIterations=10,
runs=10, initializationMode=random)
# Evaluate clustering by computing Within Set Sum of Squared Errors
def error(point):
center = clusters.centers[clusters.predict(point)]
return sqrt(sum([x**2 for x in (point - center)]))
WSSSE = parsedData.map(lambda point: error(point)).reduce(lambda x, y: x + y)
print(Within Set Sum of Squared Error = + str(WSSSE))
Re: [julia-users] Re: macros generating macros in modules
Let me ask the question slightly differently. Without constraining yourself to legal Julia syntax, what would you like to go in, and what do you want to come out? I get the feeling there's a design here that doesn't have this complexity--my intuition is that a generic function can be used to drive this just fine (called within a macro to facilitate the final splice.) But that's still hard to concretely suggest. On Saturday, April 4, 2015 at 9:39:57 PM UTC-5, Abe Schneider wrote: I don't understand how what I'm trying to do falls outside the domain of transforming code. My main goal is to take a user-defined grammar and create relevant code. More to the point, I think the general ability to generate expressions based on functions/namespaces is also solidly in the domain of transforms. I can imagine plenty of use-cases for language-bindings, communication, etc. Yes, there are many simple macros one can create, which are useful, but there exists many useful things one can do with macros that can get more complex. A On Saturday, April 4, 2015 at 10:11:25 PM UTC-4, Jameson wrote: I think the underlying issue may be that you are trying to have the macro do too much, when you should instead be just doing source-code transforms and preprocessing. One such example of a great use of a macro is simply: macro special_foo(syntax_tree) return quote special_foo( $(QuoteNode(syntax_tree) ) end end On Sat, Apr 4, 2015 at 8:05 PM Abe Schneider abe.sc...@gmail.com wrote: The issue I'm dealing with is that I have a macro that I want to pass a list of functions that the macro can employ for parsing grammars. The macro itself exists already, and it currently has its own static list of functions. I'm trying to figure out a way to allow the user of the library to customize it. For example: @grammar foo begin start = list(integer, r, ) end where `list` is a user supplied function. The issue that I keep running into is that there doesn't seem to be any easy way of just passing in a function to a macro. Turning an expression into a function is problematic, because `eval`s are: (a) only work in the modules namespace, and (b) are frowned on. I've thought about manually looking up the symbol name in the various namespaces, but while I've found methods to list the functions, I haven't found a way to do the reverse. Ideally, it would be nice to generate an expression with `call` that took a symbol and a namespace, that way no lookups would have to be done. Having hit a dead end in this direction, it occurred to me that it might be possible to do something like this: @set_parsers(list, etc) @grammar foo begin ... end where '@set_parsers` would generate the `@grammar` macro with the static list it already has plus whatever else the user adds in. Which works, except for the whole module issue. Thanks, A On Saturday, April 4, 2015 at 12:41:27 PM UTC-4, Patrick O'Leary wrote: On Saturday, April 4, 2015 at 9:04:10 AM UTC-5, Abe Schneider wrote: I should start off, not entirely sure this is an okay thing to do with Julia. Suppose I want to create a macro that generates another macro... I'm not sure whether this should work or not, but either way I'm not sure how it's any more expressive than macros alone? Can you describe what it is you want to do at a high level in a little more detail? There might be another way.
[julia-users] Re: macros generating macros in modules
On Saturday, April 4, 2015 at 9:04:10 AM UTC-5, Abe Schneider wrote: I should start off, not entirely sure this is an okay thing to do with Julia. Suppose I want to create a macro that generates another macro... I'm not sure whether this should work or not, but either way I'm not sure how it's any more expressive than macros alone? Can you describe what it is you want to do at a high level in a little more detail? There might be another way.
[julia-users] Re: Unexpected behaviour when running parallelized code on single worker
Have you tried this with a normal Array instead of a SharedArray for the outputs? (There may or may not be other problems and/or bugs in Julia, but I do know that SharedArray is considered experimental.) On Saturday, April 4, 2015 at 9:20:48 AM UTC-5, Nils Gudat wrote: There's something weird happening in my recently parallelized code. When running it without adding some worker processes first, the results are completely off and after some investigation I found that this was due to assignment operations going wrong - results of computations were assigned to different Arrays than the intended ones. A small working example illustrating the point: x1 = linspace(1, 3, 3) x2 = linspace(1, 3, 3) x3 = linspace(1, 3, 3) function getresults(x1::Array, x2::Array, x3::Array) result1 = SharedArray(Float64, (3,3,3)) result2 = similar(result1) result3 = similar(result1) @sync @parallel for a=1:3 for b=1:3 for c=1:3 result1[a,b,c] = x1[a]*x2[b]*x3[c] result2[a,b,c] = sqrt(x1[a]*x2[b]*x3[c]) result3[a,b,c] = (x1[a]*x2[b]*x3[c])^2 end end end return sdata(result1), sdata(result2), sdata(result3) end (r1,r2,r3) = getresults(x1, x2, x3) nprocs()==CPU_CORES || addprocs(CPU_CORES-1) (r1_par,r2_par,r3_par) = getresults(x1, x2, x3) When I run this on my system (v0.3.6), the parallelized version works as intended, while running the code without adding workers first gives the expected results for r1 and r3, but r2 holds the same results as r3. The behaviour in my original problem was similar, the code returns three Arrays, but running it without additional workers those Arrays all return the same contents. Is there something in the @sync or @parallel macros that causes this? How should a code be written to ensure that it works both with one and multiple cores?
Re: [julia-users] macros generating macros in modules
using functions is often much simpler to write and use: function meta_foo(x) function foo(y) x+y end end foo2 = meta_foo(5) println(foo2(3)) # 8 On Sat, Apr 4, 2015 at 10:04 AM Abe Schneider abe.schnei...@gmail.com wrote: I should start off, not entirely sure this is an okay thing to do with Julia. Suppose I want to create a macro that generates another macro, I can write: macro meta_macro(x) quote macro foo(y) $x+y end end end and then call it and the generated macro: @meta_macro(5) println(@foo(3)) # 8 Okay, wasn't entirely sure that was going to work, but it appears to. However, if I try to move the exact same macro code into a module, it no longer works: module meta_macro_mod export @meta_macro, @foo macro meta_macro(x) quote macro foo(y) $x+y end end end end Even without trying to invoke `@foo`, when I execute `@meta_macro`, I get: ERROR: syntax: invalid macro definition It seems like how the macro gets evaluated changes when it's in the module. If I change it to a module to just doing an 'include', everything works again.
[julia-users] R's splineDesign in Julia
Hi, I am looking for a function (library) that allows evaluation of a B-spline basis (specified by the knots and the order), at given points, _and_ also allows the evaluation of derivatives for the same family. R's splineDesign in the splines package has this, but looking at the splines libraries in Julia I could not find one that does B-splines and derivatives. Using internals from some library would be fine, too. (Knowing that there is no such thing at the moment would also be useful, I might implement it, but would prefer to avoid that because B-spline special cases are tricky). Best, Tamas
Re: [julia-users] [ANN] JuliaIO and FileIO
I might not understand you correctly.
FileIO basically just includes all meta packages like ImageIO and MeshO, as
it needs to know of all the possible IO libraries in order to read/write
arbitrary files. The actual implementation of read(file::File{sometype})
happens in the low level IO packages like WavefrontObj.jl or ImageMagick.jl.
To mitigate this tasks, I introduced a layer in between, which breaks this
tasks up into more manageable chunks (the different file categories).
This way, FileIO doesn't have to change much and can stay very stable, as
all of the fun happens in the other layers.
But this is not the only reason for the second layer of abstraction like
ImageIO.
There are a lot of packages, that are only interested in the datatype, but
not in the functionality of that. So in GLAbstraction I don't need
imfilter, etc, etc. I just need some image IO and the Image datatype, in
order to load image files to the GPU.
In essence, every library that extends the image type, without actually
needing the rest of the functionality, only needs ImageIO.
Images.jl itself can than use ImageIO, to define functionality like
imfilter, etc.
I hope this makes sense!
Best,
Simon
2015-04-04 23:47 GMT+02:00 Steve Kelly kd2...@gmail.com:
Simon,
I think this is a great idea! However I am kind of confused with the
separation of interface here. Is the idea that FileIO.jl will have
definitions for each package?
I think the power of the Require.jl package is that packages can define
their own interface. For example Meshes.jl can have `@requires FileIO begin
... end`. I think FileIO is more likely to have a stable API than Meshes,
so it seems better IMO to keep the FileIO method extensions in Meshes. In
the case of Images.jl, putting the FileIO behind a @requires
would eliminate the need for the ImageIO package and make it easier to
update if types change in Images.
Best,
Steve
On Sat, Apr 4, 2015 at 5:23 PM, Matt Bauman mbau...@gmail.com wrote:
Very cool. I've been intersted in doing something like this myself for
quite some time now. Have you seen
https://github.com/JuliaLang/julia/issues/7299 ?
I'll take a look when I get a chance.
Re: [julia-users] Re: ANN: Match.jl v0.1.0 released
I'm using Julia Version 0.3.7-pre+19. Later today, I'll build a small test and open an issue. thanks Bill On Saturday, April 4, 2015 at 1:55:36 PM UTC-7, Kevin Squire wrote: Sorry to hear that--I can't seem to reproduce that problem. Can you open an issue at https://github.com/kmsquire/Match.jl/issues with a small test case that I can use to reproduce? What version of Julia are you using? Cheers, Kevin
Re: [julia-users] [ANN] JuliaIO and FileIO
Simon, I think this is a great idea! However I am kind of confused with the separation of interface here. Is the idea that FileIO.jl will have definitions for each package? I think the power of the Require.jl package is that packages can define their own interface. For example Meshes.jl can have `@requires FileIO begin ... end`. I think FileIO is more likely to have a stable API than Meshes, so it seems better IMO to keep the FileIO method extensions in Meshes. In the case of Images.jl, putting the FileIO behind a @requires would eliminate the need for the ImageIO package and make it easier to update if types change in Images. Best, Steve On Sat, Apr 4, 2015 at 5:23 PM, Matt Bauman mbau...@gmail.com wrote: Very cool. I've been intersted in doing something like this myself for quite some time now. Have you seen https://github.com/JuliaLang/julia/issues/7299 ? I'll take a look when I get a chance.
[julia-users] Can one force a newline in the epilog of Argparse help?
I'm using the remarkable Argparse package, and in the help epilog, I would like to include several examples of calling the program, with each example separated from its predecessor by a newline. Is this possible? For example, if the user types `myprog -h`, then at the end of the help output I would like something along the lines: Example 1: myprog arg1 arg2 --flag1 --opt1 5 does lots of good stuff. Example 2: myprog arg1 --opt2 0 does something completely different. Right now I get instead the following, where the lines are all automatically wrapped: Example 1: myprog arg1 arg2 --flag1 --opt1 5 does lots of good stuff. Example 2: myprog arg1 --opt2 0 does something completely different. Is there some way to force a newline within the epilog text string? Thanks in advance, --Peter
[julia-users] Re: R's splineDesign in Julia
https://github.com/kbarbary/Dierckx.jl https://docs.scipy.org/doc/scipy-0.15.1/reference/interpolate.html (use with PyCall) https://github.com/lgautier/Rif.jl (R and Julia Interfacing) On Sunday, April 5, 2015 at 12:57:28 AM UTC+5:30, Tamas Papp wrote: Hi, I am looking for a function (library) that allows evaluation of a B-spline basis (specified by the knots and the order), at given points, _and_ also allows the evaluation of derivatives for the same family. R's splineDesign in the splines package has this, but looking at the splines libraries in Julia I could not find one that does B-splines and derivatives. Using internals from some library would be fine, too. (Knowing that there is no such thing at the moment would also be useful, I might implement it, but would prefer to avoid that because B-spline special cases are tricky). Best, Tamas
Re: [julia-users] Re: R's splineDesign in Julia
I have looked at Dierckx.jl, but could not figure out how to evaluate the B-spline basis (all the functions in the documentation appear to do fitting directly). Can you please give me a hint? Thanks, Tamas On Sat, Apr 04 2015, pauld11718 pauld11...@gmail.com wrote: https://github.com/kbarbary/Dierckx.jl https://docs.scipy.org/doc/scipy-0.15.1/reference/interpolate.html (use with PyCall) https://github.com/lgautier/Rif.jl (R and Julia Interfacing) On Sunday, April 5, 2015 at 12:57:28 AM UTC+5:30, Tamas Papp wrote: Hi, I am looking for a function (library) that allows evaluation of a B-spline basis (specified by the knots and the order), at given points, _and_ also allows the evaluation of derivatives for the same family. R's splineDesign in the splines package has this, but looking at the splines libraries in Julia I could not find one that does B-splines and derivatives. Using internals from some library would be fine, too. (Knowing that there is no such thing at the moment would also be useful, I might implement it, but would prefer to avoid that because B-spline special cases are tricky). Best, Tamas
[julia-users] Re: ANN: Match.jl v0.1.0 released
Thanks for looking into this Kevin
after updating to 0.1.3 I get the error:
ERROR: type: isa: expected Type{T:Top}, got Function
On Friday, April 3, 2015 at 9:22:36 PM UTC-7, Kevin Squire wrote:
Okay, it was a little more subtle than that: if your code was already
using Compat.jl, then everything would have worked. If not, then
fieldnames isn't defined at all (as you found out).
I believe I've fixed this issue, and I've tagged a new version of
Match.jl. Please try it and let me know if you still have problems.
Cheers!
Kevin
Re: [julia-users] Re: ANN: Match.jl v0.1.0 released
Sorry to hear that--I can't seem to reproduce that problem. Can you open
an issue at https://github.com/kmsquire/Match.jl/issues with a small test
case that I can use to reproduce? What version of Julia are you using?
Cheers,
Kevin
On Sat, Apr 4, 2015 at 1:34 PM, William Macready william.macre...@gmail.com
wrote:
Thanks for looking into this Kevin
after updating to 0.1.3 I get the error:
ERROR: type: isa: expected Type{T:Top}, got Function
On Friday, April 3, 2015 at 9:22:36 PM UTC-7, Kevin Squire wrote:
Okay, it was a little more subtle than that: if your code was already
using Compat.jl, then everything would have worked. If not, then
fieldnames isn't defined at all (as you found out).
I believe I've fixed this issue, and I've tagged a new version of
Match.jl. Please try it and let me know if you still have problems.
Cheers!
Kevin
[julia-users] [ANN] JuliaIO and FileIO
Very cool. I've been intersted in doing something like this myself for quite some time now. Have you seen https://github.com/JuliaLang/julia/issues/7299 ? I'll take a look when I get a chance.
[julia-users] Re: [ANN] JuliaIO and FileIO
Oh brilliant, this is looks very similar :D
I should follow Julia's issues more closely...
Looking forward to your feedback! There are still quite some problems to
figure out!
Am Samstag, 4. April 2015 17:41:14 UTC+2 schrieb Simon Danisch:
Hi there,
FileIO has the aim to make it very easy to read any arbitrary file.
I hastily copied together a proof of concept by taking code from Images.jl.
JuliaIO is the umbrella group, which takes IO packages with no home. If
someone wrote an IO package, but doesn't have time to implement the FileIO
interface, giving it to JuliaIO might be a good idea in order to keep the
package usable.
Concept of FileIO is described in the readme:
Meta package for FileIO. Purpose is to open a file and return the
respective Julia object, without doing any research on how to open the file.
f = filetest.jpg # - File{:jpg}read(f) # - Imageread(filetest.obj) # -
Meshread(filetest.csv) # - DataFrame
So far only Images are supported and MeshIO is on the horizon.
It is structured the following way: There are three levels of abstraction,
first FileIO, defining the file_str macro etc, then a meta package for a
certain class of file, e.g. Images or Meshes. This meta package defines the
Julia datatype (e.g. Mesh, Image) and organizes the importer libraries.
This is also a good place to define IO library independant tests for
different file formats. Then on the last level, there are the low-level
importer libraries, which do the actual IO. They're included via Mike Innes
Requires https://github.com/one-more-minute/Requires.jl package, so
that it doesn't introduce extra load time if not needed. This way, using
FileIO without reading/writing anything should have short load times.
As an implementation example please look at FileIO - ImageIO -
ImageMagick. This should already work as a proof of concept. Try:
using FileIO # should be very fast, thanks to Mike Innes Requires
packageread(filetest.jpg) # takes a little longer as it needs to load the
IO libraryread(filetest.jpg) # should be fastread(File(documents,
images, myimage.jpg) # automatic joinpath via File constructor
Please open issues if things are not clear or if you find flaws in the
concept/implementation.
If you're interested in working on this infrastructure I'll be pleased to
add you to the group JuliaIO.
Best,
Simon
[julia-users] Re: holoviews in julia
Great to see this is getting some interest. I'm one of the authors of HoloViews and put together that initial (and poor) attempt at a Julia integration. If someone is interested in developing this further, I'd be glad to provide assistance in putting together a proper interface, although I'm afraid I won't be much help on the Julia end. Philipp On Saturday, 4 April 2015 00:23:46 UTC+1, Andrei Berceanu wrote: I just found this as a starting point http://philippjfr.com/blog/interfacing-holoviews-with-julia/ On Saturday, April 4, 2015 at 1:21:00 AM UTC+2, Andrei Berceanu wrote: Are you guys familiar with http://ioam.github.io/holoviews ? Is anyone working on a Julia interface or port? //A
