[julia-users] Writing a subset DataFrame to file is 220 times slower than saving the whole DataFrame

[Fred]

Hi,

In the same program,  I save in a file a DataFrame "df" and a subset of 
this DataFrame in another file. The problem I have is that saving the 
subset is much slower than saving the entire DataFrame : 220 times slower. 
It is too slow and I don't what is my mistake.

Thank you for your advices !

in Julia 0.4.5 : 

Saving the entire DataFrame
Saving... results/Stat.csv
1.115944 seconds (13.78 M allocations: 319.534 MB, 2.59% gc time)


Saving the subset of the DataFrame 
Saving... significant/Stat.csv
246.099835 seconds (41.79 M allocations: 376.189 GB, 4.77% gc time)
elapsed time: 251.581459853 seconds


in Julia 0.5 : 

Saving the entire DataFrame
Saving... results/Stat.csv
1.060365 seconds (7.08 M allocations: 116.025 MB, 0.73% gc time)

Saving the subset of the DataFrame 
Saving... significant/Stat.csv
226.813587 seconds (37.40 M allocations: 376.268 GB, 2.42% gc time)
elapsed time: 232.95933586 seconds


# my function to save the results to a file

function write_results(x, name, dir, sep, h)
  outfile = "$dir/$name"
  println("Saving...\t", outfile)
  writetable( outfile, x, separator = sep, header = h)
end


# save my DataFrame df : very fast
@time write_results(df, name, "results", sep, h)


# subset DataFrame s
s = sub(df, (df[:rank_PV] .<= r_max))

# save my subset DataFrame s : incredibly slow !

@time write_results(s, name, "significant", sep, h)

[julia-users] Re: Writing a subset DataFrame to file is 220 times slower than saving the whole DataFrame

[Fred]

Wow ! Impressive difference, it is 100 time faster without sub() :


s = df[df[:rank_PV].<=r_max,:]
@time write_results(s, name, "significant", sep, h)
Saving... significant/Stat.csv
 0.704880 seconds (7.14 M allocations: 164.687 MB, 2.40% gc time)

s = sub(df, (df[:rank_PV] .<= r_max))
@time write_results(s, name, "significant", sep, h)
Saving...significant/Stat.csv
 86.797541 seconds (21.98 M allocations: 104.215 GB, 4.13% gc time)



Thanks Alex !

[julia-users] How to stop program execution at some point

[Fred]

Hi,

I search in Julia the equivalent of Perl 
die;
to stop the execution of a program.

I tried "break" but it produces an error :
ERROR: LoadError: syntax: break or continue outside loop
I did not found "stop" as well.

Thanks !

[julia-users] Re: How to stop program execution at some point

[Fred]

Yes Michele, "die" is able to stop execution at any point, even outside a 
loop, this is what I searched and did not found in Julia.

[julia-users] Re: How to stop program execution at some point

[Fred]

Thank you Tamas, quit() is exactly what I searched ;)

Le lundi 7 novembre 2016 11:40:32 UTC+1, Tamas Papp a écrit :
>
> See quit(), and in general exit(code).
>

[julia-users] How to built an array with some keys of a dictionary ?

[Fred]

Hi,

I have many problems to build arrays with the keys of a dictionary except 
for the simple situation :

a = [uppercase(key) for key in keys(dict)]# works fine

If I try to select the keys with more complex criteria like :

dict = Dict("a" => 1, "b" => 2, "c" => 3, "d" => 4, "e" => 5)

a = Array{String}

for k in keys(dict)
  if dict[k] < 2
continue
  end
push!(a, k) # building array from keys produces errors
end


ERROR: MethodError: `push!` has no method matching push!(::Type{Array{
AbstractString,N}}, ::ASCIIString)




Thank you for your help !

[julia-users] Re: How to built an array with some keys of a dictionary ?

[Fred]

Hi Steven,


I also tried  a = Array{String}() unfortunately it produces errors as well.



julia> a = Array{String}()
WARNING: Base.String is deprecated, use AbstractString instead.
  likely near no file:0
0-dimensional Array{AbstractString,0}:
#undef


julia> for k in keys(dict)
 if dict[k] < 2
   continue
 end
   push!(a, k)
   end
ERROR: MethodError: `push!` has no method matching push!(::Array{
AbstractString,0}, ::ASCIIString)
Closest candidates are:
  push!(::Any, ::Any, ::Any)
  push!(::Any, ::Any, ::Any, ::Any...)
  push!(::Array{Any,1}, ::ANY)
  ...
 [inlined code] from none:5
 in anonymous at no file:0

Re: [julia-users] Re: How to built an array with some keys of a dictionary ?

[Fred]

Thank you very much Mauro, that was the mistake ! :)


> Array{String}(0) 
>
> works.  The other creates a zero-dimensional vector (just like a scalar) 
> to which you cannot push. 
>
> > julia> a = Array{String}() 
> > WARNING: Base.String is deprecated, use AbstractString instead. 
> >  likely near no file:0 
> > 0-dimensional Array{AbstractString,0}: 
> > #undef 
> > 
> > 
> > julia> for k in keys(dict) 
> > if dict[k] < 2 
> >  continue 
> > end 
> >  push!(a, k) 
> >end 
> > ERROR: MethodError: `push!` has no method matching 
> push!(::Array{AbstractString 
> > ,0}, ::ASCIIString) 
> > Closest candidates are: 
> >  push!(::Any, ::Any, ::Any) 
> >  push!(::Any, ::Any, ::Any, ::Any...) 
> >  push!(::Array{Any,1}, ::ANY) 
> >  ... 
> > [inlined code] from none:5 
> > in anonymous at no file:0 
>

Re: [julia-users] Re: How to built an array with some keys of a dictionary ?

[Fred]

Thank you Dan, this solution works to ! I don't know which one is better :)

Le mardi 8 novembre 2016 08:40:37 UTC+1, Dan a écrit :
>
> I meant,
>
> a = Vector{String}()
>
>

[julia-users] How to use variables in subsets of DataFrames ?

[Fred]

Hi !
I would like to know how is it possible to use variables in subsets of 
DataFrames ? I would like to use a syntax like
 df[:,:titles[1]] and df[:,:titles[1:3]] 

Thanks for your help !


julia> using DataFrames


julia> df = readtable("test.csv", separator = '\t')
8x5 DataFrame
| Row | title1 | title2 | title3 | title4 | title5 |
|-||||||
| 1   | 10 | 20 | 30 | 40 | 50 |
| 2   | 11 | 21 | 31 | 41 | 51 |
| 3   | 12 | 22 | 32 | 42 | 52 |
| 4   | 13 | 23 | 33 | 43 | 53 |
| 5   | 14 | 24 | 34 | 44 | 54 |
| 6   | 15 | 25 | 35 | 45 | 55 |
| 7   | 16 | 26 | 36 | 46 | 56 |
| 8   | 17 | 27 | 37 | 47 | 57 |


julia> titles = readdlm("titles.csv")
3x1 Array{Any,2}:
 "title3"
 "title1"
 "title5"


julia> df[:,:title2]
8-element DataArray{Int64,1}:   
  
 20 
  
 21 
  
 22 
  
 23 
  
 24 
  
 25 
  
 26 
  
 27 
  

  
julia> titles[1]
"title3"   
   

  
julia> df[:,:titles[1]]
ERROR: `getindex` has no method matching getindex(::Symbol, ::Int64)   

julia> df[:,:titles[1:3]]
ERROR: `getindex` has no method matching getindex(::Symbol, ::UnitRange{
Int64}) 



title1	title2	title3	title4	title5
10	20	30	40	50
11	21	31	41	51
12	22	32	42	52
13	23	33	43	53
14	24	34	44	54
15	25	35	45	55
16	26	36	46	56
17	27	37	47	57
title3
title1
title5

Re: [julia-users] How to use variables in subsets of DataFrames ?

[Fred]

Great answer Rob ! This is exactly what I was looking for !

Many Thanks !

Fred

Le lundi 28 septembre 2015 17:35:15 UTC+2, Rob J Goedman a écrit :
>
> Fred,
>
> Would  below example work for you?
>
> *julia> **dft = readtable("/Users/rob/Desktop/test.csv", separator = 
> '\t')*
> *8x5 DataFrames.DataFrame*
> *| Row | title1 | title2 | title3 | title4 | title5 |*
> *|-||||||*
> *| 1   | 10 | 20 | 30 | 40 | 50 |*
> *| 2   | 11 | 21 | 31 | 41 | 51 |*
> *| 3   | 12 | 22 | 32 | 42 | 52 |*
> *| 4   | 13 | 23 | 33 | 43 | 53 |*
> *| 5   | 14 | 24 | 34 | 44 | 54 |*
> *| 6   | 15 | 25 | 35 | 45 | 55 |*
> *| 7   | 16 | 26 | 36 | 46 | 56 |*
> *| 8   | 17 | 27 | 37 | 47 | 57 |*
>
> *julia> **titles = names(dft)*
> *5-element Array{Symbol,1}:*
> * :title1*
> * :title2*
> * :title3*
> * :title4*
> * :title5*
>
> *julia> **dft[[2:6], titles[2:5]]*
> *5x4 DataFrames.DataFrame*
> *| Row | title2 | title3 | title4 | title5 |*
> *|-|||||*
> *| 1   | 21 | 31 | 41 | 51 |*
> *| 2   | 22 | 32 | 42 | 52 |*
> *| 3   | 23 | 33 | 43 | 53 |*
> *| 4   | 24 | 34 | 44 | 54 |*
> *| 5   | 25 | 35 | 45 | 55 |*
>
> *julia> **dft[[2:6], titles[3]]*
> *5-element DataArrays.DataArray{Int64,1}:*
> * 31*
> * 32*
> * 33*
> * 34*
> * 35*
>
>
> If you use list comprehension you will need the extra Symbol[] construct:
>
> *julia> **dft[:, Symbol[titles[i] for i in 2:3]]*
> *8x2 DataFrames.DataFrame*
> *| Row | title2 | title3 |*
> *|-|||*
> *| 1   | 20 | 30 |*
> *| 2   | 21 | 31 |*
> *| 3   | 22 | 32 |*
> *| 4   | 23 | 33 |*
> *| 5   | 24 | 34 |*
> *| 6   | 25 | 35 |*
> *| 7   | 26 | 36 |*
> *| 8   | 27 | 37 |*
>
>
> Regards,
> Rob
>
> On Sep 28, 2015, at 2:18 AM, Fred > 
> wrote:
>
> Hi !
> I would like to know how is it possible to use variables in subsets of 
> DataFrames ? I would like to use a syntax like
>  df[:,:titles[1]] and df[:,:titles[1:3]] 
>
> Thanks for your help !
>
>
> julia> using DataFrames
>
>
> julia> df = readtable("test.csv", separator = '\t')
> 8x5 DataFrame
> | Row | title1 | title2 | title3 | title4 | title5 |
> |-||||||
> | 1   | 10 | 20 | 30 | 40 | 50 |
> | 2   | 11 | 21 | 31 | 41 | 51 |
> | 3   | 12 | 22 | 32 | 42 | 52 |
> | 4   | 13 | 23 | 33 | 43 | 53 |
> | 5   | 14 | 24 | 34 | 44 | 54 |
> | 6   | 15 | 25 | 35 | 45 | 55 |
> | 7   | 16 | 26 | 36 | 46 | 56 |
> | 8   | 17 | 27 | 37 | 47 | 57 |
>
>
> julia> titles = readdlm("titles.csv")
> 3x1 Array{Any,2}:
>  "title3"
>  "title1"
>  "title5"
> 
>
> julia> df[:,:title2]
> 8-element DataArray{Int64,1}: 
> 
>  20   
> 
>  21   
> 
>  22   
> 
>  23   
> 
>  24   
> 
>  25   
> 
>  26   
> 
>  27   
> 
>   
> 
> julia> titles[1]
> "title3" 
>  
>   
> 
> julia> df[:,:titles[1]]
> ERROR: `getindex` has no method matching getindex(::Symbol, ::Int64)   
>
> julia> df[:,:titles[1:3]]
> ERROR: `getindex` has no method matching getindex(::Symbol, ::UnitRange{
> Int64}) 
>
>
> 
> 
>
>
>

[julia-users] trying to do a simple parallel program with Julia

[Fred]

Hi !

I am trying to do a simple test of parallel computing with Julia. It is a 
program which read a dataframe and write the number of columns in a text 
file.  I failed  :/  The program gives the following error. If you have an 
idea how to solve it... Thanks in advance ;)

$ julia main_parallel-001.jl 
exception on 3: ERROR: opening file tab2csv: Aucun fichier ou dossier de ce 
type
 in open at ./iostream.jl:117
 in open at ./iostream.jl:125
 in readtable at /home/fred/.julia/v0.3/DataFrames/src/dataframe/io.jl:889
 in process_table at /archives/logiciels/julia/parallel-05.jl:7
 in anonymous at multi.jl:855
 in run_work_thunk at multi.jl:621
 in anonymous at task.jl:855
exception on 2: ERROR: opening file tab2csv: Aucun fichier ou dossier de ce 
type
 in open at ./iostream.jl:117
 in open at ./iostream.jl:125
 in readtable at /home/fred/.julia/v0.3/DataFrames/src/dataframe/io.jl:889
 in process_table at /archives/logiciels/julia/parallel-05.jl:7
 in anonymous at multi.jl:855
 in run_work_thunk at multi.jl:621
 in anonymous at task.jl:855


##
# main program for parallel computing

addprocs(4)

require("parallel-05.jl")

function main()
  # process_table("tab.csv")# give good result
  # process_table("tab2.csv")   # give good result
  pmap(process_table,{"tab.csv","tab2csv"}) # does not work for tab2csv
end

main()




##
# parallel-05.jl


using DataFrames


function process_table(title)
  sep='\t'
  myData = readtable(title ,separator=sep,header=false)
  m = size(myData,2)  # number of 
columns
  file_name = string("processed_",title)
  outfile = open(file_name, "w")
  write(outfile, "$m$sep")# write the 
number of columns in outfile
  close(outfile)
end

[julia-users] Re: trying to do a simple parallel program with Julia

[Fred]

Hi !

I found the "bug"  : a dot was missing in 
pmap(process_table,{"tab.csv","tab2csv"})

instead of

pmap(process_table,{"tab.csv","tab2.csv"})

The program works perfectly, I have to change my glasses 8-) 

[julia-users] Julia parallel when number of tasks > number of CPUs

[Fred]

Hi !

When the number of tasks exceed the number of CPUs, is it safe to send all 
the tasks in pmap() function ?

For example, if I have 20 tasks and 4 CPUs, can I use the command :

pmap(function,[var1, var2,var20]) 

Or is there a way to tel pmap() to process the task 4 by 4 ?

Thanks ! ;)

[julia-users] Re: How to know the size of a large array stored in a csv file (greater than RAM

[Fred]

If you use GNU/Linux or Unix you can obtain the number of lines with : wc 
-l file.csv

Le dimanche 14 juin 2015 09:51:29 UTC+2, paul analyst a écrit :
>
> How to know the size of a large array stored in a csv file (greater than 
> RAM, I can not use readcsv. After that I want to apply the open (file, 
> size, size)
> Paul
>

[julia-users] Re: parallel computing in julia

[Fred]

Hi,

You have started julia and then you try to start it again with julia -p 2.

just type julia -p 2 in a terminal before starting julia :)


$ julia -p 2
   _
   _   _ _(_)_ |  A fresh approach to technical computing
  (_) | (_) (_)|  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.4.5 (2016-03-18 00:58 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org release
|__/   |  x86_64-linux-gnu

[julia-users] Re: parallel computing in julia

[Fred]

To start 4 cpus :

$ julia -p 3
   _
   _   _ _(_)_ |  A fresh approach to technical computing
  (_) | (_) (_)|  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.4.5 (2016-03-18 00:58 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org release
|__/   |  x86_64-linux-gnu


julia> nprocs()
4

[julia-users] replacing "require" by "include" does not import DataFrames

[Fred]

Hi !

I have a old parallel program that works perfectly with "require" and I 
can't manage to make it work with "include". The reason I try to do that is 
that "require" is deprecated.

old program that worked :


main :
require("file.jl")

pmap(function, list_of_dataframes)


file.jl :


using DataFrames

function(df)
  readtable(df)
  some computation
  writetable(df2)
end


new program that never worked :


main :
include("file.jl")

pmap(function, list_of_dataframes)


file.jl :


using DataFrames

@everywhere function(df)
  readtable(df)
  some computation
  writetable(df2)
end



So in summary I replaced "required" by "include" and I had @everywhere in 
front of the function of the file included. But the problem is that 
DataFrames is not imported and I can't read any dataframe 
CapturedException(UndefVarError(:readtable). I tried 
@everywhere using DataFrames
 but it did not solve the problem.


RemoteException(2,CapturedException(UndefVarError(:readtable),Any[(:function,symbol("file.jl"),25,symbol(""),-1,1),(:anonymous,symbol("multi.jl"),923,symbol(""),-1,1),(:run_work_thunk,symbol("multi.jl"),661,symbol(""),-1,1),(:anonymous,symbol("multi.jl"),923,symbol("task.jl"),63,1)]))


thank you for your help !

[julia-users] Re: replacing "require" by "include" does not import DataFrames

[Fred]

I found a solution that works partially with warnings :

push!(LOAD_PATH, pwd())
using file


WARNING: requiring "file" in module "Main" did not define a corresponding 
module.

[julia-users] DataFrame : aggregate with only on column possible ?

[Fred]

Hi,

I have a dataframe df2 and the last column is the sum = b+c :

julia> df2
8x4 DataFrames.DataFrame
│ Row │ a │ b │ c │ sum   │
┝━┿━━━┿━━━┿━━━┿━━━┥
│ 1   │ 1 │ 2 │ -0.163564 │ 1.83644   │
│ 2   │ 2 │ 1 │ 0.731038  │ 1.73104   │
│ 3   │ 3 │ 2 │ 0.0951149 │ 2.09511   │
│ 4   │ 4 │ 1 │ 0.195321  │ 1.19532   │
│ 5   │ 1 │ 2 │ 1.97058   │ 3.97058   │
│ 6   │ 2 │ 1 │ 0.150826  │ 1.15083   │
│ 7   │ 3 │ 2 │ 0.422046  │ 2.42205   │
│ 8   │ 4 │ 1 │ -1.36549  │ -0.365486 │

we can se that the column a have duplicates (1,2,3). I try to find a simple 
way to remove the duplicates that have the lowest sum without changing the 
values of b and c.

I tried :

julia> aggregate(df2, :a,  maximum)


4x4 DataFrames.DataFrame
│ Row │ a │ b_maximum │ c_maximum │ sum_maximum │
┝━┿━━━┿━━━┿━━━┿━┥
│ 1   │ 1 │ 2 │ 1.97058   │ 3.97058 │
│ 2   │ 2 │ 1 │ 0.731038  │ 1.73104 │
│ 3   │ 3 │ 2 │ 0.422046  │ 2.42205 │
│ 4   │ 4 │ 1 │ 0.195321  │ 1.19532 │



but this is wrong because I don't want b_maximum and c_maximum but only 
sum_maximum :


Row │ a │ b │ c │ sum_maximum


I don't think that there is a simple way to do that but I ask the question 
in case ;)

Thank  you very much in advance !

[julia-users] Re: DataFrame : aggregate with only on column possible ?

[Fred]

Thank you Cedric !

To clarify I give you an example :

│ Row │ a │ b │ c│ sum │
┝━┿━━━┿━━━┿━━┿━┥
│ 1   │ X │ 2 │ 10   │12   │
│ 2   │ Y │ 1 │ 3│ 4   │
│ 3   │ Z │ 2 │ 5│ 7   │
│ 4   │ X │ 1 │ 20   │ 21  │
│ 5   │ X │ 2 │ 5│ 7   │
│ 6   │ Z │ 1 │ 8│ 9   │

I want to obtain :

│ Row │ a │ b │ c  │ sum_max│
┝━┿━━━┿━━━┿┿┥
│ 1   │ X │ 1 │20  │21  │
│ 2   │ Y │ 1 │ 3  │ 4  │
│ 3   │ Z │ 1 │ 8  │ 9  │





 
you can see that the lines are unchanged but filtered to obtain the sum 
maximum. In particular the column b contains only "1". 
with aggregate(df2, :a,  maximum) it is not the case because I would also 
obtain the maximum of b (2,1,2) and c. When I have duplicates in column a 
(X,X,X), for example :


│ Row │ a │ b │ c   │sum_max│
┝━┿━━━┿━━━┿━┿━━━┥
│ 1   │ X │ 2 │ 10  │  12   │
│ 4   │ X │ 1 │ 20  │  21   │
│ 5   │ X │ 2 │ 5   │  7│




I want to remove the rows  1 and 5 because their sum is lower than row 4
. So the result is :

│ Row │ a │ b │ c  │ sum_max│
┝━┿━━━┿━━━┿┿┥
│ 4   │ X │ 1 │ 20 │ 21 │



I don't wan't b_max and c_max, only sum_max. I hope my explanation is now 
more clear :)

[julia-users] Re: DataFrame : aggregate with only on column possible ?

[Fred]

Thank you Cédric !

Yes it works with an extra column a_1 :


julia> df
8x4 DataFrames.DataFrame
│ Row │ a │ b │ c │ sum│
┝━┿━━━┿━━━┿━━━┿┥
│ 1   │ 1 │ 2 │ 2.28386   │ -0.0551961 │
│ 2   │ 2 │ 1 │ 0.174953  │ 0.472804   │
│ 3   │ 3 │ 2 │ -1.31137  │ 0.994803   │
│ 4   │ 4 │ 1 │ 0.305978  │ 0.580371   │
│ 5   │ 1 │ 2 │ -0.725308 │ 1.19441│
│ 6   │ 2 │ 1 │ 0.095797  │ 1.19544│
│ 7   │ 3 │ 2 │ -0.481912 │ 0.284116   │
│ 8   │ 4 │ 1 │ 0.811624  │ 1.23245│

julia> by(df, :a) do subdf
  subdf[indmax(subdf[:sum]), :]
   end
4x5 DataFrames.DataFrame
│ Row │ a │ a_1 │ b │ c │ sum  │
┝━┿━━━┿━┿━━━┿━━━┿━━┥
│ 1   │ 1 │ 1   │ 2 │ -0.725308 │ 1.19441  │
│ 2   │ 2 │ 2   │ 1 │ 0.095797  │ 1.19544  │
│ 3   │ 3 │ 3   │ 2 │ -1.31137  │ 0.994803 │
│ 4   │ 4 │ 4   │ 1 │ 0.811624  │ 1.23245  │

[julia-users] Re: ArrayFire.jl - GPU Programming in Julia

[Fred]

Hi !

Thank you for this great package ! I tried to install it on Julia 0.4.5 but 
I obtained :

julia 
   _
   _   _ _(_)_ |  A fresh approach to technical computing
  (_) | (_) (_)|  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.4.5 (2016-03-18 00:58 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org release
|__/   |  x86_64-linux-gnu


julia> using ArrayFire
ERROR: LoadError: LoadError: could not load library "libaf"
libaf: Ne peut ouvrir le fichier d'objet partagé: Aucun fichier ou dossier 
de ce type
 in dlopen at ./libdl.jl:36
 in dlopen at libdl.jl:36
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 in require at ./loading.jl:259
while loading /home/fred/.julia/v0.4/ArrayFire/src/config.jl, in expression 
starting on line 6
while loading /home/fred/.julia/v0.4/ArrayFire/src/ArrayFire.jl, in 
expression starting on line 5

[julia-users] Re: How can I find the common elements of two matrix(or arrays)?

[Fred]

intersect(a,b)
union(a,b)

Le dimanche 3 juillet 2016 19:54:02 UTC+2, siyu song a écrit :
>
> In MATLAB, it's easy to do this by using : find(Mat_a == Mat_b). Is there 
> an easy similar way to do this? 
>

[julia-users] Re: How can I find the common elements of two matrix(or arrays)?

[Fred]

julia> a=[1,3,5,7]
4-element Array{Int64,1}:
 1
 3
 5
 7


julia> b=[2,3,5,6,7]
5-element Array{Int64,1}:
 2
 3
 5
 6
 7


julia> intersect(a,b)
3-element Array{Int64,1}:
 3
 5
 7


julia> union(a,b)
6-element Array{Int64,1}:
 1
 3
 5
 7
 2
 6



Le lundi 4 juillet 2016 04:18:10 UTC+2, siyu song a écrit :
>
> But intersect doesn't tell us the index of the elements in the 
> matrix(array), I think. 
>

[julia-users] changing record separator to read text files

[Fred]

Hi !

It is often very useful to read a text files by blocks of lines, using 
another line separator than '\n'.

Especially in bio-informatics, for example DNA or Protein FASTA sequences 
are separated by '\n>' (see uniprot.txt attached).

In Perl,  it is possible to change the line separator using :
 local $/ = '\n>'
for example.

In Julia I did not found how to do that :

line by line :

julia> f = open("uniprot.txt")
IOStream()

julia> readline(f, '\n>' )
ERROR: syntax: invalid character literal




all lines in an array (I prefer line by line because some files do not fit 
into RAM) :

julia> readdlm("uniprot.txt", '\n>' )
ERROR: syntax: invalid character literal

readdlm("uniprot.txt", '>' ) # works but does not give the expected result


So I suppose that this feature is currently not implemented in Julia ? 

Thanks in advance for your comments !



>sp|P31946|1433B_HUMAN 14-3-3 protein beta/alpha OS=Homo sapiens GN=YWHAB PE=1 
>SV=3
MTMDKSELVQKAKLAEQAERYDDMAAAMKAVTEQGHELSNEERNLLSVAYKNVVGARRSS
WRVISSIEQKTERNEKKQQMGKEYREKIEAELQDICNDVLELLDKYLIPNATQPESKVFY
LKMKGDYFRYLSEVASGDNKQTTVSNSQQAYQEAFEISKKEMQPTHPIRLGLALNFSVFY
YEILNSPEKACSLAKTAFDEAIAELDTLNEESYKDSTLIMQLLRDNLTLWTSENQGDEGD
AGEGEN
>sp|P04439|1A03_HUMAN HLA class I histocompatibility antigen, A-3 alpha chain 
>OS=Homo sapiens GN=HLA-A PE=1 SV=2
MAVMAPRTLSGALALTQTWAGSHSMRYFFTSVSRPGRGEPRFIAVGYVDDTQFVRF
DSDAASQRMEPRAPWIEQEGPEYWDQETRNVKAQSQTDRVDLGTLRGYYNQSEAGSHTIQ
IMYGCDVGSDGRFLRGYRQDAYDGKDYIALNEDLRSWTAADMAAQITKRKWEAAHEAEQL
RAYLDGTCVEWLRRYLENGKETLQRTDPPKTHMTHHPISDHEATLRCWALGFYPAEITLT
WQRDGEDQTQDTELVETRPAGDGTFQKWAAVVVPSGEEQRYTCHVQHEGLPKPLTLRWEL
SSQPTIPIVGIIAGLVLLGAVITGAVVAAVMWRRKSSDRKGGSYTQAASSDSAQGSDVSL
TACKV
>sp|P01889|1B07_HUMAN HLA class I histocompatibility antigen, B-7 alpha chain 
>OS=Homo sapiens GN=HLA-B PE=1 SV=3
MLVMAPRTVSAALALTETWAGSHSMRYFYTSVSRPGRGEPRFISVGYVDDTQFVRF
DSDAASPREEPRAPWIEQEGPEYWDRNTQIYKAQAQTDRESLRNLRGYYNQSEAGSHTLQ
SMYGCDVGPDGRLLRGHDQYAYDGKDYIALNEDLRSWTAADTAAQITQRKWEAAREAEQR
RAYLEGECVEWLRRYLENGKDKLERADPPKTHVTHHPISDHEATLRCWALGFYPAEITLT
WQRDGEDQTQDTELVETRPAGDRTFQKWAAVVVPSGEEQRYTCHVQHEGLPKPLTLRWEP
SSQSTVPIVGIVAGLAVLAVVVIGAVVAAVMCRRKSSGGKGGSYSQAACSDSAQGSDVSL
TA
>sp|P30464|1B15_HUMAN HLA class I histocompatibility antigen, B-15 alpha chain 
>OS=Homo sapiens GN=HLA-B PE=1 SV=2
MRVTAPRTVSGALALTETWAGSHSMRYFYTAMSRPGRGEPRFIAVGYVDDTQFVRF
DSDAASPRMAPRAPWIEQEGPEYWDRETQISKTNTQTYRESLRNLRGYYNQSEAGSHTLQ
RMYGCDVGPDGRLLRGHDQSAYDGKDYIALNEDLSSWTAADTAAQITQRKWEAAREAEQW
RAYLEGLCVEWLRRYLENGKETLQRADPPKTHVTHHPISDHEATLRCWALGFYPAEITLT
WQRDGEDQTQDTELVETRPAGDRTFQKWAAVVVPSGEEQRYTCHVQHEGLPKPLTLRWEP
SSQSTIPIVGIVAGLAVLAVVVIGAVVATVMCRRKSSGGKGGSYSQAASSDSAQGSDVSL
TA

[julia-users] Re: changing record separator to read text files

[Fred]

Thank you very much Dan !

Indeed readuntil works much better. But it seems to accept only one end 
line character. Is it possible to use a more complex pattern ?

julia> readuntil(f, '>')
"sp|P31946|1433B_HUMAN 14-3-3 protein beta/alpha OS=Homo sapiens GN=YWHAB 
PE=1 
SV=3\nMTMDKSELVQKAKLAEQAERYDDMAAAMKAVTEQGHELSNEERNLLSVAYKNVVGARRSS\nWRVISSIEQKTERNEKKQQMGKEYREKIEAELQDICNDVLELLDKYLIPNATQPESKVFY\nLKMKGDYFRYLSEVASGDNKQTTVSNSQQAYQEAFEISKKEMQPTHPIRLGLALNFSVFY\nYEILNSPEKACSLAKTAFDEAIAELDTLNEESYKDSTLIMQLLRDNLTLWTSENQGDEGD\nAGEGEN\n>"

julia> readuntil(f, '\n>')
ERROR: syntax: invalid character literal




Le samedi 9 juillet 2016 18:02:39 UTC+2, Dan a écrit :
>
> `readuntil` is your friend.
>
> `readline` uses `readuntil`. In fact, it is defined as:
>
> readuntil(s,'\n')
>
>
>>

[julia-users] Re: changing record separator to read text files

[Fred]

It works using "" instead of '' !

Thanks !

julia> readuntil(f, "CKV\n")
"\nTA\n>sp|P30464|1B15_HUMAN HLA class I histocompatibility antigen, B-15 
alpha chain OS=Homo sapiens GN=HLA-B PE=1 
SV=2\nMRVTAPRTVSGALALTETWAGSHSMRYFYTAMSRPGRGEPRFIAVGYVDDTQFVRF\nDSDAASPRMAPRAPWIEQEGPEYWDRETQISKTNTQTYRESLRNLRGYYNQSEAGSHTLQ\nRMYGCDVGPDGRLLRGHDQSAYDGKDYIALNEDLSSWTAADTAAQITQRKWEAAREAEQW\nRAYLEGLCVEWLRRYLENGKETLQRADPPKTHVTHHPISDHEATLRCWALGFYPAEITLT\nWQRDGEDQTQDTELVETRPAGDRTFQKWAAVVVPSGEEQRYTCHVQHEGLPKPLTLRWEP\nSSQSTIPIVGIVAGLAVLAVVVIGAVVATVMCRRKSSGGKGGSYSQAASSDSAQGSDVSL\nTA\n"

[julia-users] How to determine the type of a complex data structure

[Fred]

Hi,

I have a this kind of data structure :

type Ions_frag{}
  mz::Float64 
  intensity::Float64
  charge::Int64 
end


I read in performance tips that it is better to define the type of the data 
structure but I cannot find the good one for my data structure. I tried :

type Ions_frag{DataType}
  mz::Float64 
  intensity::Float64
  charge::Int64 
end


but it gives this error :

This may have arisen from a call to the constructor Ions_frag{DataType
}(...),
since type constructors fall back to convert methods.
Closest candidates are:
  call{T}(::Type{T}, ::Any)
  convert{T}(::Type{T}, !Matched::T)


Thank you for your comments !

Re: [julia-users] How to determine the type of a complex data structure

[Fred]

I am referring to all 3, ie the type of Ions_frag :)
Initially I used : 

type Ions_frag
  mz::Float64 
  intensity::Float64
  charge::Int64 
end

and it works. But, after reading the documentation I "understood" that it 
is not optimal in performances. 


Le mardi 12 juillet 2016 16:59:57 UTC+2, Yichao Yu a écrit :
>
>
>
> Which one are you referring to? 
>
> The one you have is a perfectly fine type. I'm not sure what do you 
> want to achieve. 
>
>

Re: [julia-users] How to determine the type of a complex data structure

[Fred]

I use a dictionary to store values :

for example :

d = Dict()

d["key1"] = Ions_frag(501.5, 1500, 3)

Then to obtain the values :

d["key1"].mz # = 501.5

My problem was to store many values in one object. So I found this 
solution, I don't know if it is the best and I hope this approach is 
correct ? :)

Le mardi 12 juillet 2016 17:32:32 UTC+2, Jeffrey Sarnoff a écrit :
>
> You may be misreading the guidance.  There is no advantage to putting a 
> parameter into the type when that parameter is not used anywhere.
>
> type Ions_frag
>mz::Float64 
>intensity::Float64
>charge::Int64 
> end
>  if your use involves changing the fields' values after creating an 
> ions_frag typed variable
> immutable Ions_frag
>mz::Float64 
>intensity::Float64
>charge::Int64 
> end
>
> if your use constructs ions_frag variables appropriately initialized with 
> mz, intensity, charge; and then uses them as they are. 
>
> On Tuesday, July 12, 2016 at 11:10:40 AM UTC-4, Fred wrote:
>>
>> I am referring to all 3, ie the type of Ions_frag :)
>> Initially I used : 
>>
>> type Ions_frag
>>   mz::Float64 
>>   intensity::Float64
>>   charge::Int64 
>> end
>>
>> and it works. But, after reading the documentation I "understood" that it 
>> is not optimal in performances. 
>>
>>
>> Le mardi 12 juillet 2016 16:59:57 UTC+2, Yichao Yu a écrit :
>>>
>>>
>>>
>>> Which one are you referring to? 
>>>
>>> The one you have is a perfectly fine type. I'm not sure what do you 
>>> want to achieve. 
>>>
>>>

Re: [julia-users] How to determine the type of a complex data structure

[Fred]

I tried that "change" because of this part of the documentation (in 
http://docs.julialang.org/en/release-0.4/manual/performance-tips/) :

For example:

julia> type MyType{T<:AbstractFloat}
 a::T
   end
This is a better choice than

julia> type MyStillAmbiguousType
 a::AbstractFloat
   end




Le mardi 12 juillet 2016 17:32:09 UTC+2, Yichao Yu a écrit :
>
> On Tue, Jul 12, 2016 at 11:10 AM, Fred > 
> wrote: 
> > I am referring to all 3, ie the type of Ions_frag :) 
>
> Well, can you quote the performance tip? There's nothing wrong with 
> the type you are using in terms of performance so I'd like to know 
> which part of the doc is misleading. 
>
>

Re: [julia-users] How to determine the type of a complex data structure

[Fred]

Thank you very much Yichao, Tim and Jeffrey for your answers !

I try to summarize. If I understand well, this type definition is correct 
and gives the maximum performances :

type Ions_frag
   mz::Float64
   intensity::Float64
   charge::Int64
end



But if I use :

d = Dict()

d["key1"] = Ions_frag(501.5, 1500, 3)



the type of d is unstable and gives poor performances. So, if we assume 
that "key1" is a String, do you think that I can improve performances like 
that ? :

d = Dict{AbstractString,Any}

d["key1"] = Ions_frag(501.5, 1500, 3)

Re: [julia-users] How to determine the type of a complex data structure

[Fred]

Thank you very much Yichao,

I will try :

Dict{Any,Ions_frag}



and 

immutable Ions_frag
   mz::Float64
   intensity::Float64
   charge::Int64
end

Re: [julia-users] How to determine the type of a complex data structure

[Fred]

Hi !

Here are the results :

type Ions_frag{}
  mz::Float64 
  intensity::Float64
  charge::Int64 
end


computation time : 26.88 s


immutable Ions_frag

   mz::Float64
   intensity::Float64
   charge::Int64
end


computation time : 27.00 s



Ions = Dict{ASCIIString, Ions_frag}()


immutable Ions_frag


   mz::Float64
   intensity::Float64
   charge::Int64
end
computation time : 17.96 s



Ions = Dict{ASCIIString, Ions_frag}()

type Ions_frag{}
  mz::Float64 
  intensity::Float64
  charge::Int64 
end

computation time : 17.58 s



So, in conclusion "immutable" is better than "type" in theory but not in 
practice. The Dict() type has a major effect on performances !

Thanks !

[julia-users] DataFrames : Apply a function by rows

[Fred]

Hi,

In DataFrames, it is easy to apply a function by columns using the colwise() 
function. But I find very difficult and inefficient to apply a function by 
rows.

For example :


 
 julia> df = DataFrame(a=1:5, b=7:11, c=10:14) 
5x3 DataFrames.DataFrame 
| Row | a | b  | c  | 
|-|---||| 
| 1   | 1 | 7  | 10 | 
| 2   | 2 | 8  | 11 | 
| 3   | 3 | 9  | 12 | 
| 4   | 4 | 10 | 13 | 
| 5   | 5 | 11 | 14 | 

 
 
 julia> colwise(mean,df) 
3-element Array{Any,1}: 
 [3.0]  
 [9.0]  
 [12.0]
 
 
 julia> colwise(mean,df[1,1:2]) 
2-element Array{Any,1}: 
 [1.0] 
 [7.0]



To calculate the mean of a row (or a subset), the only way I found is this :

julia> mean(convert(Array,df[1,1:3])) 
6.0
 


I think this is inefficient and probably very slow. I there a better way to 
apply a function by rows ?

Thanks !

Re: [julia-users] DataFrames : Apply a function by rows

[Fred]

 Thanks for the answer. I tried "eachrow" but I have 2 problems :

1- I still have to do an array conversion, I think it is slow

julia> for r in eachrow(df) 
  println(mean(convert(Array,r))) 
   end 
6.0 
7.0 
8.0 
9.0 
10.0



2- I do not manage to use a subset of the row, for example the 2 first 
values :
julia> for r in eachrow(df) 
  println(mean(convert(Array,r))) 
   end 
6.0 
7.0 
8.0 
9.0 
10.0 
 
julia> for r in eachrow(df) 
  println(mean(convert(Array,r[1:2]))) 
   end 
WARNING: [a] concatenation is deprecated; use collect(a) instead 
 in depwarn at deprecated.jl:73 
 in oldstyle_vcat_warning at ./abstractarray.jl:29 
 [inlined code] from none:2 
 in anonymous at no file:0 
while loading no file, in expression starting on line 0 
4.0
 





Le samedi 21 novembre 2015 14:04:11 UTC+1, tshort a écrit :
>
> You can try `eachrow`. It probably won't be fast, though. Here's an 
> example:
>
>
> https://github.com/JuliaStats/DataFrames.jl/blob/master/test/iteration.jl#L34
> On Nov 21, 2015 7:19 AM, "Fred" > 
> wrote:
>
>> Hi,
>>
>> In DataFrames, it is easy to apply a function by columns using the 
>> colwise() function. But I find very difficult and inefficient to apply a 
>> function by rows.
>>
>> For example :
>>
>>
>>  
>>  julia> df = DataFrame(a=1:5, b=7:11, c=10:14) 
>> 5x3 DataFrames.DataFrame 
>> | Row | a | b  | c  | 
>> |-|---||| 
>> | 1   | 1 | 7  | 10 | 
>> | 2   | 2 | 8  | 11 | 
>> | 3   | 3 | 9  | 12 | 
>> | 4   | 4 | 10 | 13 | 
>> | 5   | 5 | 11 | 14 | 
>>
>>  
>>  
>>  julia> colwise(mean,df) 
>> 3-element Array{Any,1}: 
>>  [3.0]  
>>  [9.0]  
>>  [12.0]
>>  
>>  
>>  julia> colwise(mean,df[1,1:2]) 
>> 2-element Array{Any,1}: 
>>  [1.0] 
>>  [7.0]
>>
>>
>>
>> To calculate the mean of a row (or a subset), the only way I found is 
>> this :
>>
>> julia> mean(convert(Array,df[1,1:3])) 
>> 6.0
>>  
>>
>>
>> I think this is inefficient and probably very slow. I there a better way 
>> to apply a function by rows ?
>>
>> Thanks !
>>
>

Re: [julia-users] DataFrames : Apply a function by rows

[Fred]

It is a good idea but how is it possible to iterate two dataframes at the 
same time ? Something like :

julia> df = DataFrame(a=1:5, b=7:11, c=10:14, d=20:24)
5x4 DataFrames.DataFrame
| Row | a | b  | c  | d  |
|-|---||||
| 1   | 1 | 7  | 10 | 20 |
| 2   | 2 | 8  | 11 | 21 |
| 3   | 3 | 9  | 12 | 22 |
| 4   | 4 | 10 | 13 | 23 |
| 5   | 5 | 11 | 14 | 24 |

julia> df1 = df[1:2,]
5x2 DataFrames.DataFrame
| Row | a | b  |
|-|---||
| 1   | 1 | 7  |
| 2   | 2 | 8  |
| 3   | 3 | 9  |
| 4   | 4 | 10 |
| 5   | 5 | 11 |

julia> df1 = df[3:4,]
5x2 DataFrames.DataFrame
| Row | c  | d  |
|-|||
| 1   | 10 | 20 |
| 2   | 11 | 21 |
| 3   | 12 | 22 |
| 4   | 13 | 23 |
| 5   | 14 | 24 |

julia> for r1,r2 in eachrow(df1, df2)
  println(mean(r1,r2))
   end
ERROR: syntax: invalid iteration specification




Le samedi 21 novembre 2015 15:08:34 UTC+1, tshort a écrit :
>
> For the subset, do the indexing after the conversion to an array, or 
> subset the DataFrame first (probably faster).
>

Re: [julia-users] DataFrames : Apply a function by rows

[Fred]

In my last example, the function mean() is not well chosen. In fact, what  
I would like to calculate is a statistical test line by lline, like TTest, 
or Wilcoxon. This is why I need to iterate thought 2 DataFrames at the same 
time if I subset the DataFrame first to increase speed :)


Something like :

julia> for r1,r2 in eachrow(df1, df2)
  println(TTest(r1,r2))
   end
ERROR: syntax: invalid iteration specification




Le samedi 21 novembre 2015 19:17:27 UTC+1, Fred a écrit :
>
> It is a good idea but how is it possible to iterate two dataframes at the 
> same time ? Something like :
>
> julia> df = DataFrame(a=1:5, b=7:11, c=10:14, d=20:24)
> 5x4 DataFrames.DataFrame
> | Row | a | b  | c  | d  |
> |-|---||||
> | 1   | 1 | 7  | 10 | 20 |
> | 2   | 2 | 8  | 11 | 21 |
> | 3   | 3 | 9  | 12 | 22 |
> | 4   | 4 | 10 | 13 | 23 |
> | 5   | 5 | 11 | 14 | 24 |
>
> julia> df1 = df[1:2,]
> 5x2 DataFrames.DataFrame
> | Row | a | b  |
> |-|---||
> | 1   | 1 | 7  |
> | 2   | 2 | 8  |
> | 3   | 3 | 9  |
> | 4   | 4 | 10 |
> | 5   | 5 | 11 |
>
> julia> df1 = df[3:4,]
> 5x2 DataFrames.DataFrame
> | Row | c  | d  |
> |-|||
> | 1   | 10 | 20 |
> | 2   | 11 | 21 |
> | 3   | 12 | 22 |
> | 4   | 13 | 23 |
> | 5   | 14 | 24 |
>
> julia> for r1,r2 in eachrow(df1, df2)
>   println(mean(r1,r2))
>end
> ERROR: syntax: invalid iteration specification
>
>
>
>
> Le samedi 21 novembre 2015 15:08:34 UTC+1, tshort a écrit :
>>
>> For the subset, do the indexing after the conversion to an array, or 
>> subset the DataFrame first (probably faster).
>>
>

Re: [julia-users] DataFrames : Apply a function by rows

[Fred]

Yes, it is a good solution, but it means that DataFrames cannot be used to 
do some calculations by rows, it is a severe limitation. An equivalent of 
colwise() 
whould be very usefull.

Le dimanche 22 novembre 2015 14:11:21 UTC+1, tshort a écrit :
>
> I'd convert the whole DataFrame to a matrix and use a loop over rows.
>

Re: [julia-users] DataFrames : Apply a function by rows

[Fred]

Ok, I hope that exchange could contribute to bring news ideas to improve 
DataFrames although there are other way to do it, like convert a DataFrame 
or a row into array. Thank you for your help !

Le dimanche 22 novembre 2015 15:48:37 UTC+1, tshort a écrit :
>
> Contributions/pull requests from folks that need that are welcome. I don't 
> have that need. For row operations, I can generally get by with loops or 
> `@byrow!` in DataFramesMeta.
>

[julia-users] Array of DataFrames

[Fred]

Hi !

I try to find a solution to import a list of DataFrames.  I tried two 
strategies :
1- build an array of dataframes
2- read arrays and convert them to dataframes after
Is there a way to index dataframes ? (df[1], df[2]...df[n])

Thanks !


using DataFrames

sep = '\t'

dir = readdir("data")
 

data = (Array{DataFrames.DataFrame})[] 
0-element Array{Array{DataFrames.DataFrame,N},1}



for file in dir 
f = readtable("data/$file", separator=sep, header=true) #  
push!(data, f) 
   end 
ERROR: MethodError: `convert` has no method matching convert(::Type{
DataFrames.DataFrame}, ::UTF8String) 
This may have arisen from a call to the constructor DataFrames.DataFrame
(...),   
since type constructors fall back to convert methods.   
 
Closest candidates are: 
  call{T}(::Type{T}, ::Any) 
  convert(::Type{DataFrames.DataFrame}, ::Array{T,2}) 
  convert(::Type{DataFrames.DataFrame}, ::Dict{K,V}) 
  ... 
 in setindex! at array.jl:313 
 in copy! at abstractarray.jl:344 
 in copy! at abstractarray.jl:326 
 in convert at /home/fred/.julia/v0.4/DataFrames/src/abstractdataframe/
abstractdataframe.jl:522 
 in push! at array.jl:432 
 [inlined code] from none:3 
 in anonymous at no file:0

Re: [julia-users] Array of DataFrames

[Fred]

*Thanks Milan ! ;)*

julia> using DataFrames 
 
julia> sep = '\t' 
'\t' 
 
julia> data = DataFrame[] 
0-element Array{DataFrames.DataFrame,1}




 

> The problem is in this line, which creates an array of arrays of data 
> frames. It should simply be 
> DataFrame[] 
> or even 
> [] 
> given that a data frame is a complex object that cannot be stored in 
> contiguously in an array anyway, and that type inference is likely not 
> critical here. 
>
>
> Regards 
>

[julia-users] One click execution of Julia program

[Fred]

Hi !

I would like to know if it is possible to execute a julia program just by 
clicking on the program name on GNU/Linux (of course after a chmod +x 
program.jl).

I manage to do that in bash using
#!/usr/bin/bash



or in Perl
#!/usr/bin/perl



but in Julia I try to insert the lines :
#!/usr/bin/julia
or
#!/usr/bin//env julia



and it does not work. Of course I can make a batch scipt like :

#!/usr/bin/bash
julia test.jl


but it would be better to do that directly.

Thanks in advance and happy new year !

#!/usr/bin/julia

println("hello")

readline()

Re: [julia-users] One click execution of Julia program

[Fred]

Hi Abel

on my system I have :


$ which julia
/usr/bin/julia




Le mercredi 6 janvier 2016 19:06:13 UTC+1, Abel Siqueira a écrit :
>
> Julia is on `/usr/local/bin` in my system. Verify if the path is correct 
> with `which julia`.
> Using the correct path I managed to run it on the terminal as you would 
> the other scripts.
>
> I did not manage to run it clicking on the program name, but I couldn't do 
> it with a simple bash
> script also. I use mostly the shell for this.
>
> Abel Soares Siqueira
>
>

Re: [julia-users] One click execution of Julia program

[Fred]

That's true, it seems to be related to the system. 
On my system, lightweight (XFCE), a double click open any executable Julia 
program in a text editor and I don't have the choice to execute in a 
terminal.
But, if I run the program in a terminal using ./program.jl it works, 
indicating that the program could be run in one click.

Le jeudi 7 janvier 2016 02:07:09 UTC+1, ele...@gmail.com a écrit :
>
> Works for me.
>
> Note that when you run by clicking there is nowhere for the output of the 
> `println` to show, and standard input will be closed so the `readline` 
> returns EOF immediately, so it might look like it didn't run.
>
> My system offers an option to "run in terminal" when I double click which 
> gives it somewhere to show the output and to read the input from.
>
> Cheers
> Lex
>

Re: [julia-users] One click execution of Julia program

[Fred]

I solved the problem :)

In XFE (http://roland65.free.fr/xfe/) associate .jl files 
with /usr/bin/julia allows one click execution of Julia programs :)
In Thunar, it seems to be much more difficult 
: https://bbs.archlinux.org/viewtopic.php?id=194464

Le jeudi 7 janvier 2016 10:29:57 UTC+1, Fred a écrit :
>
> That's true, it seems to be related to the system. 
> On my system, lightweight (XFCE), a double click open any executable Julia 
> program in a text editor and I don't have the choice to execute in a 
> terminal.
> But, if I run the program in a terminal using ./program.jl it works, 
> indicating that the program could be run in one click.
>

Re: [julia-users] One click execution of Julia program

[Fred]

In Thunar :

 Create a custom action within Thunar which will execute your shell script 
in a terminal window. Open Thunar > Edit > Configure custom actions... > 
Add a new custom action, then:

- On the Basic Tab:
- Name = ExecuteJulia
- Description = ExecuteJulia
- Command = "xfce4-terminal -e /usr/bin/julia %f --hold". "--hold" is 
used to keep the terminal open, and can be removed if you want
- Select an icon if you want
- On the Appearance Conditions tab:
- Select "Other files"
- File Pattern (Optional) = Change to "*.jl" instead of "*"

Le jeudi 7 janvier 2016 10:51:29 UTC+1, Fred a écrit :
>
> I solved the problem :)
>
> In XFE (http://roland65.free.fr/xfe/) associate .jl files 
> with /usr/bin/julia allows one click execution of Julia programs :)
> In Thunar, it seems to be much more difficult : 
> https://bbs.archlinux.org/viewtopic.php?id=194464
>
> Le jeudi 7 janvier 2016 10:29:57 UTC+1, Fred a écrit :
>>
>> That's true, it seems to be related to the system. 
>> On my system, lightweight (XFCE), a double click open any executable 
>> Julia program in a text editor and I don't have the choice to execute in a 
>> terminal.
>> But, if I run the program in a terminal using ./program.jl it works, 
>> indicating that the program could be run in one click.
>>
>

[julia-users] Julia : install problem on ubuntu : Signing key not available

[Fred]

Hi,

When I install Julia on Xubuntu 15.10, I have some problems of 
authentication :

W: Erreur de GPG : http://ppa.launchpad.net wily InRelease : NO_PUBKEY 
CF979FFA3D3D3ACC

sudo apt-key adv --keyserver keyserver.ubuntu.com 
--recv-keys CF979FFA3D3D3ACC does not work

Trying to obtain the key 
from https://launchpad.net/~staticfloat/+archive/ubuntu/juliareleases does 
not work 

http://keyserver.ubuntu.com:11371/pks/lookup?fingerprint=on&op=index&search=0x770FD733195B3496F8F73CD4CF979FFA3D3D3ACC
 
does not work as well (blanc page)

So my question is : how to you manage to install Julia on Ubuntu without 
a GPG key problem ?

Thanks in advance !

Fred

[julia-users] How to copy a DataFrame structure in an empty DataFrame

[Fred]

Hi,

I have a DataFrame "df" and I would like to create a new DataFrame "m" with 
 the same colnames than df.  
And then, append to "m" the rows of "df" matching a complex set of 
conditions, so I need to test many conditions for each row. 

The problem I have is to create an empty dataframe "m" with the sames 
colnames of "df" and then to append the rows matching conditions to it.

df = readtable("file.tsv")

# trying to create an empty copy of df
m = DataFrame() 
names!(m, names(df)) # produce error ArgumentError: Length of nms doesn't 
match length of x
# but I did not manage to create m with the same cols number than df.


z = size(df)[1]

# for each row of df append the rows matching many conditions on many 
columns
   for i = 1:z
  if condition(df[:colname_x][i])
push!(m,i)
  end
  end

Thank you for your comments !

Re: [julia-users] How to copy a DataFrame structure in an empty DataFrame

[Fred]

Thank you very much Milan !
When I searched a solution in DataFrame documentation I did not found the 
keyword "similar". Is there an equivalent for Julia of this website 
for Perl for example ? 

julia> df = readtable("test2.tsv")
4x4 DataFrames.DataFrame
│ Row │ row1   │ x3_39 │ x2_5 │ x4_5 │
┝━┿┿━━━┿━━┿━━┥
│ 1   │ "row2" │ 3.46  │ 2.7  │ 4.8  │
│ 2   │ "row3" │ 3.58  │ 2.9  │ 4.7  │
│ 3   │ "row4" │ 3.7   │ 2.7  │ 4.9  │
│ 4   │ "row5" │ 3.81  │ 3.0  │ 5.3  │

julia> m = similar(df,0)
0x4 DataFrames.DataFrame


julia> m
0x4 DataFrames.DataFrame

julia> df[2,:]
1x4 DataFrames.DataFrame
│ Row │ row1   │ x3_39 │ x2_5 │ x4_5 │
┝━┿┿━━━┿━━┿━━┥
│ 1   │ "row3" │ 3.58  │ 2.9  │ 4.7  │

julia> append!(m, df[2,:])
1x4 DataFrames.DataFrame
│ Row │ row1   │ x3_39 │ x2_5 │ x4_5 │
┝━┿┿━━━┿━━┿━━┥
│ 1   │ "row3" │ 3.58  │ 2.9  │ 4.7  │

julia> m
1x4 DataFrames.DataFrame
│ Row │ row1   │ x3_39 │ x2_5 │ x4_5 │
┝━┿┿━━━┿━━┿━━┥
│ 1   │ "row3" │ 3.58  │ 2.9  │ 4.7  │

Re: [julia-users] How to copy a DataFrame structure in an empty DataFrame

[Fred]

Thank you Milan for this useful link. 
This website contains more informations than mine, but it does not seems to 
contains all the functions available for dataframes. For example I did not 
manage to find that I have to use append! instead of push! to add a row to 
a dataframe.

Re: [julia-users] How to copy a DataFrame structure in an empty DataFrame

[Fred]

Thank you tshort. Is there a website or email address to post requests 
concerning documentation ?

Le jeudi 24 mars 2016 19:28:44 UTC+1, tshort a écrit :
>
> Feel free to file issues for missing documentation or (even better) pull 
> requests.
>
>

[julia-users] SharedArray returns unexpected empty values

[Fred]

Hi,

I did this simple code to try to understand how shared arrays work. 
Unfortunately my shared array c[ ] is always empty :(
Here I know the size of the final array c because it is a very simple 
example, but  is it possible to create an empty shared array and push the 
results inside using push() function ?

Thank you very much for your comments !

addprocs(2)

 a = rand(8)
 b = rand(8)

 convert(SharedArray,a)
 convert(SharedArray,b)

 c = SharedArray(Float64,8)

 @everywhere function prod(x,y)
   z = x*y
   return z
 end

@parallel for i=1:10
  c[i] = prod(a[i], b[i])
end

println(c)

# result : [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0] :(

Re: [julia-users] SharedArray returns unexpected empty values

[Fred]

Thank you for your answer !

I did the following change :

@sync @parallel for i=1:8
  c[i] = prod(a[i], b[i])
end

and now I obtain a result :
[0.18576241230332016,0.23210296820071408,0.2741188848763749,0.6734257301922404,
0.4961438140080696,0.06445466347639998,0.07932590741374389,0.32153039018331936]

I hope everything is correct :)

Re: [julia-users] SharedArray returns unexpected empty values

[Fred]

I compared the speed of the parallel loop : it is 10 times slower (with 4 
cpus) than the simple loop :(

@time @sync @parallel for i=1:80
  c[i] = prod(a[i], b[i])
end

#println(c)

@time for i=1:80
  c[i] = prod(a[i], b[i])
end

0.728079 seconds (391.50 k allocations: 16.659 MB)
0.091947 seconds (4.80 M allocations: 73.240 MB, 5.67% gc time)

Re: [julia-users] SharedArray returns unexpected empty values

[Fred]

Thank you ! I will try on a more complex example ;)

[julia-users] Re: Read External file data

[Fred]

Hi,

Of course it is possible :

https://en.wikibooks.org/wiki/Introducing_Julia/Working_with_text_files

You can read data frames to :

https://dataframesjl.readthedocs.org/en/latest/io.html

[julia-users] Find the indice of the element with the nearest value of a float in an array

[Fred]

Hi,

I am looking for the most efficient (fastest) way to find the indice of the 
element with the nearest value of a float in an array.

x = [1:0.1:10]

julia> x
91-element Array{Float64,1}:
  1.0
  1.1
  1.2
  1.3
  1.4
  ⋮  
  9.4
  9.5
  9.6
  9.7
  9.8
  9.9
 10.0

It is very easy to find the indice of an exact value of x, for example 8.2

julia> find(x .== 8.2)
1-element Array{Int64,1}:
 73

But if I want the indice of the closest value of 8.22

julia> minimum(abs(x-8.22))
0.02135

julia> find(x .== minimum(abs(x-8.22)))
0-element Array{Int64,1}


Of course it is easy to do that with a loop but is it the fastest solution ?

min_i = 0
min_x = 1.0

 for i=[1:length(x)]
   e = abs(collect(x)[i] - 8.22)
   if e < min_x
   min_x = e
   min_i = i
   end
 end

println(min_x, " -> ", min_i)
0.02135 -> 73


Thanks for your comments !

Re: [julia-users] Find the indice of the element with the nearest value of a float in an array

[Fred]

Thank you very much !

I give you the results :

my loop solution :
0.000419 seconds (547 allocations: 708.797 KB)
0.02135 -> 73

 @time closest_index(x,8.22)
  0.03 seconds (4 allocations: 160 bytes)
73

@time for (i,x) in enumerate(array)...
0.000181 seconds (821 allocations: 19.953 KB)
738.20.02135

@time 
reduce((x,y)->x[2](x,abs(y-8.22)),1:length(x),x))
  0.005890 seconds (892 allocations: 24.617 KB)

Of course in my particular situation the array is sorted, so in that case I 
although think about using dichotomy

Re: [julia-users] Find the indice of the element with the nearest value of a float in an array

[Fred]

 Thank you very much Mauro !  searchsorted is the simplest solution and one 
of the fastest but it gives two indices so another comparison is needed to 
find the closest value :

@time searchsorted(x, 8.22)
  0.04 seconds (7 allocations: 240 bytes)
74:73

abs(x[73] - 8.22) > abs(x[74] - 8.22) ? i = 74 : i = 73 
73

Re: [julia-users] Find the indice of the element with the nearest value of a float in an array

[Fred]

I tested my loop monre than 2 times as it is written in my post and I have 
always the same results. The function Tim Holy posted is much faster, I 
posted the results above :)


>  Probably you are doing this wrong; it shouldn't be allocating so much 
> memory.  Is your loop using global variables?  Did you remember to time it 
> twice (since the first time you call it there is compilation overhead.) 
>  Did you try the function Tim Holy posted?
>

Re: [julia-users] Find the indice of the element with the nearest value of a float in an array

[Fred]

That's true ! But why a loop is faster in a function ? :)

>
> I seem to recall that your example loop was not in a function(?)  If so, 
> that makes it lots slower. 
>
>

Re: [julia-users] Find the indice of the element with the nearest value of a float in an array

[Fred]

Hi,

I post here my best solution taking advantage that the array is sorted. I 
expected to be a lot much faster than other solutions, but not really.
I am very impressed by the speed of searchsorted
 :

x = [1:0.1:100]
val = 8.22

function dicotomy(x, val)
  a = start(eachindex(x))
  b = length(x)
  j = length(x)
  dxbest = abs(x[a]-val)
  dx = dxbest

  while true
dx < dxbest ? dxbest = dx : 1
j = round(Int,(a+b)/2)
dx = x[j]-val
x[j]-val < 0 ? a = j : b = j
abs(a-b) < 2 && break
  end
  return a,b
end

@time dicotomy(x, 8.22)
 0.04 seconds (5 allocations: 192 bytes)
(73,74)


@time searchsorted(x, 8.22)
  0.05 seconds (7 allocations: 240 bytes)

 @time closest_index(x,8.22)
  0.027618 seconds (4 allocations: 160 bytes)
73

Re: [julia-users] Find the indice of the element with the nearest value of a float in an array

[Fred]

Hi,

I post here my best solution taking advantage that the array is sorted. I 
expected that solution to be a lot much faster than other solutions, but 
not really.
Indeed I am very impressed by the speed of searchsorted
 :

x = [1:0.1:100]
val = 8.22

function dicotomy(x, val)
  a = start(eachindex(x))
  b = length(x)
  j = length(x)
  dxbest = abs(x[a]-val)
  dx = dxbest

  while true
dx < dxbest ? dxbest = dx : 1
j = round(Int,(a+b)/2)
dx = x[j]-val
x[j]-val < 0 ? a = j : b = j
abs(a-b) < 2 && break
  end
  return a,b
end

@time dicotomy(x, 8.22)
 0.04 seconds (5 allocations: 192 bytes)
(73,74)


@time searchsorted(x, 8.22)
  0.05 seconds (7 allocations: 240 bytes)

@time closest_index(x,8.22)
  0.027618 seconds (4 allocations: 160 bytes)
73

Re: [julia-users] Find the indice of the element with the nearest value of a float in an array

[Fred]

Maybe my array is too small to see a difference, but if I increase the size 
I will lack of RAM ;)

julia> x = 1:0.1:100
1.0:0.1:1.0e6

julia> @time searchsorted(x, 8.22)
  0.045590 seconds (33.21 k allocations: 1.535 MB)
74:73

julia> @time searchsorted(x, 8.22)
  0.05 seconds (8 allocations: 288 bytes)
74:73

julia> @time searchsorted(x, 8.22)
  0.05 seconds (8 allocations: 288 bytes)
74:73

julia> @time closest_index(x,8.22)
  0.103219 seconds (4.37 k allocations: 222.884 KB)
73

julia> @time closest_index(x,8.22)
  0.095684 seconds (4 allocations: 160 bytes)
73

julia> @time dicotomy(x, 8.22)
  0.009142 seconds (3.45 k allocations: 173.973 KB)
(73,74)

julia> @time dicotomy(x, 8.22)
  0.05 seconds (5 allocations: 192 bytes)
(73,74)

julia> @time dicotomy(x, 8.22)
  0.04 seconds (5 allocations: 192 bytes)
(73,74)



Even better: get rid of the brackets around 1:0.1:10, and you'll be 
> that 
> much more impressed. 
>
> --Tim 
>
>

Re: [julia-users] Find the indice of the element with the nearest value of a float in an array

[Fred]

A huge size difference ! I have to read my array from data file so I 
suppose it is like Y and X is only for simulations ?

Le dimanche 10 avril 2016 17:50:02 UTC+2, Tim Holy a écrit :
>
> Just FYI: 
>
> julia> x = 1:0.1:100 
> 1.0:0.1:1.0e6 
>
> julia> y = collect(x);  # this is the same as y = [x;] 
>
> julia> sizeof(x) 
> 32 
>
> julia> sizeof(y) 
> 7928 
>
> --Tim 
>

[julia-users] Re: Find the indice of the element with the nearest value of a float in an array

[Fred]

Thank you for this explanation DNF ! It is the first time I use collect and 
the reason why I used it was an error message suggesting that I should use 
it :) maybe because of my first mistake to use []. 

Le dimanche 10 avril 2016 23:30:10 UTC+2, DNF a écrit :
>
> The big error you made originally is calling collect in every iteration 
> of the loop. Just deleting collect speeds things up by 100x. The lesson 
> is that you should (almost) never use collect.
>
> The other lesson is: don't do [1:0.1:10]. It makes your code slower, and 
> very soon your it will stop working correctly. Just delete [].
>

[julia-users] Creating an empty 2D array and append values to it

[Fred]

Hi !

The questions I have is so trivial that I spend a long time trying to find 
an example on the doc and on the web but I did not manage to do exactly 
what I want :

I want to create an empty 2D array  :  the function readdlm create exactly 
the kind of array I want if I read a CSV file with 2 columns. For example :

julia> m
1913x2 Array{Float64,2}:
  201.07 6597.95 
  202.092   17835.7  
  202.9449667.59 
  204.015   11983.4  
  204.8117586.52 
  205.803   1.1506





I tried to create exactly the same type of array and append values to it :

julia> a = Array{Float64,2}
Array{Float64,2}


push!(a, 10.0,3.0)
ERROR: MethodError: `push!` has no method matching push!(::Type{Array{
Float64,2}}, ::Float64)
Closest candidates are:
  push!(::Any, ::Any, ::Any)
  push!(::Any, ::Any, ::Any, ::Any...)
  push!(::Array{Any,1}, ::ANY)


 push!(a, [10.0,3.0])
ERROR: MethodError: `push!` has no method matching push!(::Type{Array{
Float64,2}}, ::Array{Float64,1})
Closest candidates are:
  push!(::Any, ::Any, ::Any)
  push!(::Any, ::Any, ::Any, ::Any...)
  push!(::Array{Any,1}, ::ANY)


append!(a, [10.0,3.0])
ERROR: MethodError: `append!` has no method matching append!(::Type{Array{
Float64,2}}, ::Array{Float64,1})
Closest candidates are:
  append!{T}(::Array{T,1}, ::AbstractArray{T,1})
  append!{T}(::StatsBase.AbstractHistogram{T,1,E}, ::AbstractArray{T,1})
  append!{T}(::StatsBase.AbstractHistogram{T,1,E}, ::AbstractArray{T,1}, ::
StatsBase.WeightVec{W,Vec<:AbstractArray{T<:Real,1}})
  ...



etc... I tried every possible syntax to append values to an empty 2D array 
and I did not find the solution and the correct syntax :)

Thank you for your help !

Re: [julia-users] Creating an empty 2D array and append values to it

[Fred]

Thank you very much Yichao ! This explain all the problems I have... But it 
still unclear :

I cannot create an empty 1D array and resize it :

julia> b = Array[]
0-element Array{Array{T,N},1}


julia> b = Array[]
0-element Array{Array{T,N},1}


julia> reshape(b,2,2)
ERROR: DimensionMismatch("new dimensions (2,2) must be consistent with 
array size 0")
 in reshape at array.jl:146
 in reshape at abstractarray.jl:215






if I create a non empty 2D array I still have problems to append values to 
it :

julia> b = [1 2; 3 4]
2x2 Array{Int64,2}:
 1  2
 3  4


julia> push!(b,[5,6])
ERROR: MethodError: `push!` has no method matching push!(::Array{Int64,2}, 
::Array{Int64,1})


julia> append!(b,[5,6])
ERROR: MethodError: `append!` has no method matching append!(::Array{Int64,2
}, ::Array{Int64,1})


even if possible, the values 1234 used to create the array will create 
mistakes in my calculations ;)

The only possibility I found is to create a non empty 1D array :


julia> c = [1 2 3 4]
1x4 Array{Int64,2}:
 1  2  3  4


An then reshape it :

julia> reshape(c,2,2)
2x2 Array{Int64,2}:
 1  3
 2  4


But that means, that the array c cannot grow in size using push!() , is 
that true ?

Re: [julia-users] Creating an empty 2D array and append values to it

[Fred]

Thank you very much Tim !

In fact I want to create an X,Y array so if I create a 1D array, I can only 
append to it (x1,y1) then (x2,y2)... (xn, yn), because I calculate x1 
before x2...

julia> d = ["x1", "y1", "x2", "y2", "x3", "y3"]
6-element Array{ASCIIString,1}:
 "x1"
 "y1"
 "x2"
 "y2"
 "x3"
 "y3"

julia> reshape(d,3,2)
3x2 Array{ASCIIString,2}:
 "x1"  "y2"
 "y1"  "x3"
 "x2"  "y3"

you see the problem ? instead I would like to have :
x1 y1
x2 y2
..
xn yn 

because I want to be able to work on columns and line ... of course another 
easy solution is to use dataframes, but I tried with arrays because the 
code should be faster... :)

Le mardi 12 avril 2016 16:27:50 UTC+2, Tim Holy a écrit :
>
> Note that in `a = Array{Float64,2}`, `a` is a *type*, not an *instance*. 
> You 
> presumably mean `a = Array(Float64, 0, 0)`. 
>
> But Yichao is right that you can't grow a 2d array, only a 1d one. 
>
> Best, 
> --Tim 
>

Re: [julia-users] Creating an empty 2D array and append values to it

[Fred]

Thank you very much Matt. In fact I will work with floats, but the idea to 
concatenate 2 columns is good !

Le mardi 12 avril 2016 16:58:21 UTC+2, Matt Bauman a écrit :
>
> It'd probably be fastest if you can pre-allocate your array:
>
> A = Array(ASCIIString, 3, 2)
> for i=1:size(A, 1)
> A[i, 1] = string('w'+i, 1)
> A[i, 2] = string('w'+i, 2)
> end
>
> If you don't know how many elements you'll have, you can use two column 
> vectors, `c1` and `c2`, and push to them independently.  Then horizontally 
> concatenate them together to get your resulting `A = [c1 c2]`.
>
> On Tuesday, April 12, 2016 at 10:38:41 AM UTC-4, Fred wrote:
>>
>> Thank you very much Tim !
>>
>> In fact I want to create an X,Y array so if I create a 1D array, I can 
>> only append to it (x1,y1) then (x2,y2)... (xn, yn), because I calculate x1 
>> before x2...
>>
>> julia> d = ["x1", "y1", "x2", "y2", "x3", "y3"]
>> 6-element Array{ASCIIString,1}:
>>  "x1"
>>  "y1"
>>  "x2"
>>  "y2"
>>  "x3"
>>  "y3"
>>
>> julia> reshape(d,3,2)
>> 3x2 Array{ASCIIString,2}:
>>  "x1"  "y2"
>>  "y1"  "x3"
>>  "x2"  "y3"
>>
>> you see the problem ? instead I would like to have :
>> x1 y1
>> x2 y2
>> ..
>> xn yn 
>>
>> because I want to be able to work on columns and line ... of course 
>> another easy solution is to use dataframes, but I tried with arrays because 
>> the code should be faster... :)
>>
>> Le mardi 12 avril 2016 16:27:50 UTC+2, Tim Holy a écrit :
>>>
>>> Note that in `a = Array{Float64,2}`, `a` is a *type*, not an *instance*. 
>>> You 
>>> presumably mean `a = Array(Float64, 0, 0)`. 
>>>
>>> But Yichao is right that you can't grow a 2d array, only a 1d one. 
>>>
>>> Best, 
>>> --Tim 
>>>
>>

Re: [julia-users] Creating an empty 2D array and append values to it

[Fred]

Thank you ! I think I will use this solution :

julia> a = []
0-element Array{Any,1}

julia> b = []
0-element Array{Any,1}

julia> push!(a, 1)
1-element Array{Any,1}:
 1

julia> push!(a, 2)
2-element Array{Any,1}:
 1
 2

julia> a
4-element Array{Any,1}:
 1
 2
 3
 4

julia> b
4-element Array{Any,1}:
 10
 20
 30
 40

julia> C = [a b]
4x2 Array{Any,2}:
 1  10
 2  20
 3  30
 4  40