[julia-users] Re: KeyError when trying to access a column of the dataframes iteratively

2016-09-18 Thread varun7rs 
Sorry about that. I was able to fix the error by changing x = frame[:type] 
to x = frame[type]

Apologies for the trouble caused

On Sunday, 18 September 2016 22:45:27 UTC+2, varu...@gmail.com wrote:
>
>
> Hi,
>
> I seem to have run into a problem here. I firstly have a function as 
> follows:
>
> function gen(type, frame1, frame2)
>
>   x = frame1[:type]
>   z = frame2[:type]
>   y = []
>  
>   for i in length(frame1)
>
> push!(y, x[i] * z[i])
>
>   end
>
> end
>
> Now, when I call:   ans = gen(I, frame1, frame2), I get a keyError saying 
> type not found.
>
> sample dataframe
>
> dataframe1
>
> Row I  II III IV V
> 10.2   0.4   0.6   0.7   0.4
> 2 .
> 3
> .
> .
> .
> Can you please let me know where I'm making a mistake and how I can fix 
> it? Thanks guys
>

Re: [julia-users] 0.5rc4: TypeError: non-boolean (Bool) used in boolean context

2016-09-18 Thread Cedric St-Jean 
Thank you, loading all the modules up-front solved the problem. I'm 
surprised that the compiler can make false assumptions about types without 
triggering segfaults left and right.

On Sunday, September 18, 2016 at 6:51:22 PM UTC-4, Yichao Yu wrote:
>
> On Sun, Sep 18, 2016 at 6:29 PM, Cedric St-Jean  > wrote: 
> > I just updated my codebase to 0.5, and encountered a strange bug. I have 
> two 
> > tests, test 1 and 2, both wrapped in their own modules so as not to 
> > interact. Both tests run fine from a fresh Julia session, but running 
> test 1 
> > before test 2 yields "TypeError: non-boolean (Bool) used in boolean 
> > context". Normally, the type in () is something like `Int64`, or 
> `String`; 
> > Bool is non-sensical The error happens on an `if` inside a function, 
> and 
> > printing thus: 
> > 
> > @show is_eoe 
> > @show isa(is_eoe, Bool) 
> > if is_eoe 
> > ... 
> > 
> > returns 
> > 
> > is_eoe = false 
> > isa(is_eoe,Bool) = true 
> > 
> > 
> > Is that a known issue? The codebase is unfortunately on the large side, 
> and 
> > trimming it down will take a while. It's not the test 1 
> definitions/imports 
> > that cause the problems; it's running code. Presumably it compiles 
> something 
> > wrong, and that corrupts test 2, somehow? 
>
> https://github.com/JuliaLang/julia/issues/18313 
>
> > 
> > Cédric 
>

[julia-users] Re: accessing globals from keyword arg defaults

2016-09-18 Thread Steven G. Johnson 


On Sunday, September 18, 2016 at 5:31:19 PM UTC-4, Marius Millea wrote:
>
> I'd like to access global variables from the default values of keywords 
> arguments, e.g.:
>
> x = 3
> function f(;x=x) #<- this default value of x here should refer to x in 
> the global scope which is 3
>

I think this should be fixed, but in the meantime it works if you just give 
the keyword argument a different name, e.g. f(; y=x) =

[julia-users] Re: accessing globals from keyword arg defaults

2016-09-18 Thread Steven G. Johnson 
See also https://github.com/JuliaLang/julia/issues/9535

Re: [julia-users] 0.5rc4: TypeError: non-boolean (Bool) used in boolean context

2016-09-18 Thread Yichao Yu 
On Sun, Sep 18, 2016 at 6:29 PM, Cedric St-Jean  wrote:
> I just updated my codebase to 0.5, and encountered a strange bug. I have two
> tests, test 1 and 2, both wrapped in their own modules so as not to
> interact. Both tests run fine from a fresh Julia session, but running test 1
> before test 2 yields "TypeError: non-boolean (Bool) used in boolean
> context". Normally, the type in () is something like `Int64`, or `String`;
> Bool is non-sensical The error happens on an `if` inside a function, and
> printing thus:
>
> @show is_eoe
> @show isa(is_eoe, Bool)
> if is_eoe
> ...
>
> returns
>
> is_eoe = false
> isa(is_eoe,Bool) = true
>
>
> Is that a known issue? The codebase is unfortunately on the large side, and
> trimming it down will take a while. It's not the test 1 definitions/imports
> that cause the problems; it's running code. Presumably it compiles something
> wrong, and that corrupts test 2, somehow?

https://github.com/JuliaLang/julia/issues/18313

>
> Cédric

[julia-users] 0.5rc4: TypeError: non-boolean (Bool) used in boolean context

2016-09-18 Thread Cedric St-Jean 
I just updated my codebase to 0.5, and encountered a strange bug. I have 
two tests, test 1 and 2, both wrapped in their own modules so as not to 
interact. Both tests run fine from a fresh Julia session, but running test 
1 before test 2 yields "TypeError: non-boolean (Bool) used in boolean 
context". Normally, the type in () is something like `Int64`, or `String`; 
Bool is non-sensical The error happens on an `if` inside a function, 
and printing thus:

@show is_eoe
@show isa(is_eoe, Bool)
if is_eoe
...

returns 

is_eoe = false
isa(is_eoe,Bool) = true


Is that a known issue? The codebase is unfortunately on the large side, and 
trimming it down will take a while. It's not the test 1 definitions/imports 
that cause the problems; it's running code. Presumably it compiles 
something wrong, and that corrupts test 2, somehow? 

Cédric

Re: [julia-users] accessing globals from keyword arg defaults

2016-09-18 Thread Erik Schnetter 
As a work-around, you can define a function `getx() = (global x; x)`, and
use it to access the global variable x.

-erik

On Sun, Sep 18, 2016 at 5:31 PM, Marius Millea 
wrote:

> I'd like to access global variables from the default values of keywords
> arguments, e.g.:
>
> x = 3
> function f(;x=x) #<- this default value of x here should refer to x in
> the global scope which is 3
>...
> end
>
> Is there any way to do this? I had guessed the following might work but it
> doesn't:
>
> function f(;x=(global x; x))
>...
> end
>



-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

[julia-users] accessing globals from keyword arg defaults

2016-09-18 Thread Marius Millea 
I'd like to access global variables from the default values of keywords 
arguments, e.g.:

x = 3
function f(;x=x) #<- this default value of x here should refer to x in the 
global scope which is 3
   ...
end

Is there any way to do this? I had guessed the following might work but it 
doesn't:

function f(;x=(global x; x)) 
   ...
end

[julia-users] KeyError when trying to access a column of the dataframes iteratively

2016-09-18 Thread varun7rs 

Hi,

I seem to have run into a problem here. I firstly have a function as 
follows:

function gen(type, frame1, frame2)

  x = frame1[:type]
  z = frame2[:type]
  y = []
 
  for i in length(frame1)

push!(y, x[i] * z[i])

  end

end

Now, when I call:   ans = gen(I, frame1, frame2), I get a keyError saying 
type not found.

sample dataframe

dataframe1

Row I  II III IV V
10.2   0.4   0.6   0.7   0.4
2 .
3
.
.
.
Can you please let me know where I'm making a mistake and how I can fix it? 
Thanks guys

Re: [julia-users] Re: julia installation broken

2016-09-18 Thread Tony Kelman 
This sounds like it may have been a download issue or a corrupt repo, 
and/or a bug in any fallback code that Pkg might contain to attempt to deal 
with problems in cloning. If you can reproduce it reliably, would be 
valuable to report the sequence of steps you used to get into the error 
state.


On Sunday, September 18, 2016 at 6:38:52 AM UTC-7, Michael Borregaard wrote:
>
> Thanks a lot, Jeffrey! Incredibly, this worked.
>
> Best,
> Michael
>
> On Fri, Sep 16, 2016 at 6:48 PM, Jeffrey Sarnoff  > wrote:
>
>> I have had this happen, too.  My suggestion is  grounded in persistence 
>> rather than deep mastery of Julia's structural and file relationships.
>> So someone else may have a more studied approach with some explaining.  
>> Until then,
>>
>> quit() Julia.
>> To have an new install be really clean, first cd  to /.julia/ 
>> and delete the directory `v0.5` -- if that is overkill because you do not 
>> want to redo many Pkg.add()s then cd v0.5 and delete the METADATA directory 
>> the REQUIRE file and the META_BRANCH file and the entry for whatever 
>> packages you had added just before this difficulty (including those that 
>> have hiccuped on precompile).   
>> now delete and then reget the most current binary for your system 
>> julia-0.5.0-rc4-osx10.7+.dmg 
>>
>> 
>> install
>> Then start julia and quit().  Then start Julia and do Pkg.update() then 
>> quit().
>> Then start julia and do Pkg.add() for 1 package you want then quit(). 
>> then start julia and do using  (so it precompiles) and quit().
>> now try using that package.
>>
>>
>>
>>
>>
>>
>> On Friday, September 16, 2016 at 4:43:06 AM UTC-4, Michael Borregaard 
>> wrote:
>>>
>>> Hi, Is this the right forum for troubleshooting / issues with my julia 
>>> install?
>>>
>>> I have kept having problems with packages giving error on precompile, 
>>> rendering julia unusable. I then reinstalled julia-0.5-rc4, and removed my 
>>> .julia folder from my home dir to do a fresh install. Now, Julia fails with 
>>> an error message when I do Pkg.update():
>>>
>>> julia> Pkg.update()
>>> INFO: Initializing package repository /Users/michael/.julia/v0.5
>>> INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
>>> ERROR: SystemError (with /Users/michael/.julia/v0.5/METADATA): rmdir: 
>>> Directory not empty
>>>  in #systemerror#51 at ./error.jl:34 [inlined]
>>>  in (::Base.#kw##systemerror)(::Array{Any,1}, ::Base.#systemerror, 
>>> ::Symbol, ::Bool) at ./:0
>>>  in #rm#7(::Bool, ::Bool, ::Function, ::String) at ./file.jl:125
>>>  in #rm#7(::Bool, ::Bool, ::Function, ::String) at 
>>> /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
>>>  in (::Base.Filesystem.#kw##rm)(::Array{Any,1}, ::Base.Filesystem.#rm, 
>>> ::String) at ./:0
>>>  in init(::String, ::String) at ./pkg/dir.jl:62
>>>  in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, 
>>> ::Vararg{Any,N}) at ./pkg/dir.jl:28
>>>  in update() at ./pkg/pkg.jl:210
>>>
>>> I assums that a completely clean install should work. Are there keys to 
>>> delete etc that will give me a properly clean install? I am on Mac OS X 
>>> (newest version).
>>> Thanks!
>>>
>>
>

Re: [julia-users] Re: Tutorial Julia language brazilian portuguese

2016-09-18 Thread Phelipe Wesley 
Não sabia da existência, tinha criado pois não conhecia nenhum, vou
informar aos que já estão cadastrados no novo para irem para o do gitter. :D

No dia 18 de setembro de 2016 às 08:16, Waldir Pimenta <
waldir.pime...@gmail.com> escreveu:

> Já existe um grupo/canal do gitter: https://gitter.im/JuliaLangPt/julia
> --- eu sugeria não fragmentarmos a comunidade, que é bem pequena, com um
> novo canal de comuncação (até porque esse canal no gitter está ligado a uma
> organização do github: https://github.com/JuliaLangPt)
>
> On Tuesday, September 13, 2016 at 4:36:40 PM UTC+1, Phelipe Wesley wrote:
>>
>> Criei o grupo no slack, para entrar é só acessar a url e um convite será
>> enviado para o email.
>>
>> https://still-dawn-96640.herokuapp.com/
>>
>> No dia 12 de setembro de 2016 às 18:59, 
>> escreveu:
>>
>>> olá felipe. É uma boa ideia. Se você criar eu compartilho com o pessoal
>>> da unb.
>>>
>>> Hello Felipe. It's a great idea. If you marry I share with the staff of
>>> unb.
>>>
>>> Em segunda-feira, 12 de setembro de 2016 17:54:50 UTC-3, Phelipe Wesley
>>> escreveu:

 O que acham de criarmos um grupo Julia-Brasil no slack?

>>>
>>

Re: [julia-users] Re: LZMA decompression with Julia

2016-09-18 Thread Yichao Yu 
On Sun, Sep 18, 2016 at 9:50 AM, Femto Trader  wrote:
> I'm still blocked
> https://github.com/yuyichao/LibArchive.jl/issues/2
> Any help is welcome.
>

As I said, you shouldn't require a IOStream, it basically means that
you can't support anything other than a raw file (which would have
been find if that's the only thing you want to support).

You should accept any objects that you can read from (or at least any
IO's). The LibArchive io interface is as close as possible to the
IOStream interface as I can make it so you should just use it directly
instead of trying to make it an IOStream, which is impossible.

>
> Le samedi 17 septembre 2016 18:54:48 UTC+2, Femto Trader a écrit :
>>
>> Thanks I will have a look
>>
>> Le samedi 17 septembre 2016 16:18:34 UTC+2, Tony Kelman a écrit :
>>>
>>> Your best bet is probably https://github.com/yuyichao/LibArchive.jl
>>>
>>>
>>> On Saturday, September 17, 2016 at 7:14:56 AM UTC-7, Femto Trader wrote:

 Hello,

 I'd like to read a LZMA compressed file with Julia.
 I haven't found such a library.
 Maybe I missed it ?

 Any help is welcome.

 Kind regards

[julia-users] Re: LZMA decompression with Julia

2016-09-18 Thread Femto Trader 
I'm still blocked
https://github.com/yuyichao/LibArchive.jl/issues/2
Any help is welcome.

Le samedi 17 septembre 2016 18:54:48 UTC+2, Femto Trader a écrit :
>
> Thanks I will have a look
>
> Le samedi 17 septembre 2016 16:18:34 UTC+2, Tony Kelman a écrit :
>>
>> Your best bet is probably https://github.com/yuyichao/LibArchive.jl
>>
>>
>> On Saturday, September 17, 2016 at 7:14:56 AM UTC-7, Femto Trader wrote:
>>>
>>> Hello,
>>>
>>> I'd like to read a LZMA compressed file with Julia.
>>> I haven't found such a library.
>>> Maybe I missed it ?
>>>
>>> Any help is welcome.
>>>
>>> Kind regards
>>>
>>

Re: [julia-users] Does Julia 0.5 leak memory?

2016-09-18 Thread Yichao Yu 
On Sun, Sep 18, 2016 at 9:36 AM, K leo  wrote:
> I am also wondering what information I should look into.
>
> On Sunday, September 18, 2016 at 9:30:00 PM UTC+8, Yichao Yu wrote:
>>
>>
>> Impossible to tell without any information provided.

At least show what your code looks like.

Re: [julia-users] Read a stuctured binary file with big endian unsigned integers (4 bytes) and big endian floats (4 bytes)

2016-09-18 Thread Femto Trader 
I think it's ok now...
https://github.com/femtotrader/DataReaders.jl/issues/14

but I'm still blocked because of LZMA compression
see https://github.com/yuyichao/LibArchive.jl/issues/2
and my 
question https://groups.google.com/forum/#!topic/julia-users/G9Pqe5svS3c

Any help will be great.

Le dimanche 18 septembre 2016 15:00:03 UTC+2, Tim Holy a écrit :
>
> See ntoh and hton. Perhaps even better, see StrPack.jl. 
>
> --Tim 
>
> On Sunday, September 18, 2016 1:13:43 AM CDT Femto Trader wrote: 
> > Hello, 
> > 
> > I'd like to read this file 
> > http://www.dukascopy.com/datafeed/EURUSD/2016/02/14/20h_ticks.bi5 
> > using Julia. 
> > 
> > It's a LZMA compressed file. 
> > 
> > I can decompressed it using 
> > cp 20h_ticks.bi5 20h_ticks.xz 
> > xz --decompress --format=lzma 20h_ticks.xz 
> > 
> > Now, I have a 20h_ticks binary file. 
> > 
> > It's a stuctured binary file with array of records 
> >   Date unsigned integer 4 bytes 
> >   Ask  unsigned integer 4 bytes 
> >   Bid  unsigned integer 4 bytes 
> >   AskVolume float 4 bytes 
> >   BidVolume float 4 bytes 
> > 
> > 
> > Using Python I'm able to read it and get a Pandas DataFrame 
> > 
> > import numpy as np 
> > import pandas as pd 
> > import datetime 
> > symb = "EURUSD" 
> > dt_chunk = datetime.datetime(2016, 2, 14) 
> > record_dtype = np.dtype([ 
> > ('Date', '>u4'), 
> > ('Ask', '>u4'), 
> > ('Bid', '>u4'), 
> > ('AskVolume', '>f4'), 
> > ('BidVolume', '>f4'), 
> > ]) 
> > 
> > data = np.fromfile("20h_ticks", dtype=record_dtype) 
> > columns = ["Date", "Ask", "Bid", "AskVolume", "BidVolume"] 
> > df = pd.DataFrame(data, columns=columns) 
> > if symb[3:] == "JPY": 
> > p_digits = 3 
> > else: 
> > p_digits = 5 
> > for p in ["Ask", "Bid"]: 
> > df[p] = df[p] / 10**p_digits 
> > df["Date"] = dt_chunk + pd.to_timedelta(df["Date"], unit="ms") 
> > df = df.set_index("Date") 
> > 
> > I'd like to do the same using Julia 
> > 
> > I did 
> > 
> > symb = "EURUSD" 
> > day_chunk = Date(2016, 2, 14) 
> > h_chunk = 20 
> > dt_chunk = DateTime(day_chunk) + Base.Dates.Hour(h_chunk) 
> > filename = @sprintf "%02dh_ticks" h_chunk 
> > println(filename) 
> > 
> > immutable TickRecordType 
> >   Date::UInt32 
> >   Ask::UInt32 
> >   Bid::UInt32 
> >   AskVolume::Float32 
> >   BidVolume::Float32 
> > end 
> > 
> > f = open(filename) 
> > 
> > # ... 
> > 
> > close(f) 
> > 
> > but I'm blocked now ... 
> > 
> > Any help will be great. 
> > 
> > Kind regards 
>
>
>

[julia-users] Re: What is the best way to element-wise right shift an array?

2016-09-18 Thread K leo 
OK, ran a test.  And the difference is pretty dramatic.  Look:

  | | |_| | | | (_| |  |  Version 0.5.0-rc4+0 (2016-09-09 01:43 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/   |  x86_64-pc-linux-gnu

julia> include("testArray.jl")

julia> testShift()
using: A=[0; A[1:end-1]]
elapsed time: *5.341734653* seconds
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0] 9
using unshift! and deleteat!
elapsed time: *0.00066514* seconds
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0] 9

function testShift()
>
> A=[1:10;]
>
> println("using: A=[0; A[1:end-1]]")
>
> tic()
>
> for i=1:1
>
> A=[0; A[1:end-1]]
>
> end
>
> toc()
>
> println(A[1:20], " ", A[10009])
>
> 
>
> B=[1:10;]
>
> println("using unshift! and deleteat!")
>
> tic()
>
> for i=1:1
>
> unshift!(B, 0)
>
> deleteat!(B, length(A))
>
> end
>
> toc()
>
> println(B[1:20], " ", B[10009])
>
> nothing
>
> end
>
>
>
On Sunday, September 18, 2016 at 9:17:11 PM UTC+8, David P. Sanders wrote:
>
> You should also benchmark the simple for loop. Please report back with the 
> results.

Re: [julia-users] Re: julia installation broken

2016-09-18 Thread Michael Krabbe Borregaard 
Thanks a lot, Jeffrey! Incredibly, this worked.

Best,
Michael

On Fri, Sep 16, 2016 at 6:48 PM, Jeffrey Sarnoff 
wrote:

> I have had this happen, too.  My suggestion is  grounded in persistence
> rather than deep mastery of Julia's structural and file relationships.
> So someone else may have a more studied approach with some explaining.
> Until then,
>
> quit() Julia.
> To have an new install be really clean, first cd  to /.julia/
> and delete the directory `v0.5` -- if that is overkill because you do not
> want to redo many Pkg.add()s then cd v0.5 and delete the METADATA directory
> the REQUIRE file and the META_BRANCH file and the entry for whatever
> packages you had added just before this difficulty (including those that
> have hiccuped on precompile).
> now delete and then reget the most current binary for your system
> julia-0.5.0-rc4-osx10.7+.dmg
> 
> install
> Then start julia and quit().  Then start Julia and do Pkg.update() then
> quit().
> Then start julia and do Pkg.add() for 1 package you want then quit(). then
> start julia and do using  (so it precompiles) and quit().
> now try using that package.
>
>
>
>
>
>
> On Friday, September 16, 2016 at 4:43:06 AM UTC-4, Michael Borregaard
> wrote:
>>
>> Hi, Is this the right forum for troubleshooting / issues with my julia
>> install?
>>
>> I have kept having problems with packages giving error on precompile,
>> rendering julia unusable. I then reinstalled julia-0.5-rc4, and removed my
>> .julia folder from my home dir to do a fresh install. Now, Julia fails with
>> an error message when I do Pkg.update():
>>
>> julia> Pkg.update()
>> INFO: Initializing package repository /Users/michael/.julia/v0.5
>> INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
>> ERROR: SystemError (with /Users/michael/.julia/v0.5/METADATA): rmdir:
>> Directory not empty
>>  in #systemerror#51 at ./error.jl:34 [inlined]
>>  in (::Base.#kw##systemerror)(::Array{Any,1}, ::Base.#systemerror,
>> ::Symbol, ::Bool) at ./:0
>>  in #rm#7(::Bool, ::Bool, ::Function, ::String) at ./file.jl:125
>>  in #rm#7(::Bool, ::Bool, ::Function, ::String) at
>> /Applications/Julia-0.5.app/Contents/Resources/julia/lib/jul
>> ia/sys.dylib:?
>>  in (::Base.Filesystem.#kw##rm)(::Array{Any,1}, ::Base.Filesystem.#rm,
>> ::String) at ./:0
>>  in init(::String, ::String) at ./pkg/dir.jl:62
>>  in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String,
>> ::Vararg{Any,N}) at ./pkg/dir.jl:28
>>  in update() at ./pkg/pkg.jl:210
>>
>> I assums that a completely clean install should work. Are there keys to
>> delete etc that will give me a properly clean install? I am on Mac OS X
>> (newest version).
>> Thanks!
>>
>

Re: [julia-users] Does Julia 0.5 leak memory?

2016-09-18 Thread K leo 
I am also wondering what information I should look into.

On Sunday, September 18, 2016 at 9:30:00 PM UTC+8, Yichao Yu wrote:
>
>
> Impossible to tell without any information provided. 
>

Re: [julia-users] Does Julia 0.5 leak memory?

2016-09-18 Thread Yichao Yu 
On Sun, Sep 18, 2016 at 8:53 AM, K leo  wrote:
> I run Julia through repl.  The procedure is simple: include("myfile.jl"),
> then run myfunction() (I might do a Ctrl-C to interrupt the function), edit
> something in myfile.jl, then repeat.  Initially, julia processes normally
> take less than 10% RAM, then after some time, one main Julia process takes
> over 70% of the 4GB RAM.  Even myfunction finishes running with real idling,
> it still takes that much RAM.  My code uses nearly no global objects, and
> myfunction ends with nothing as the last statement.
>
> myfunction isn't new though I keep modifying it from time to time.  Julia
> 0.4.6 does not appear to have this issue.
>
> Any thoughts on what might be the culprit?
>
>   | | |_| | | | (_| |  |  Version 0.5.0-rc4+0 (2016-09-09 01:43 UTC)
>  _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
> |__/   |  x86_64-pc-linux-gnu
>

Impossible to tell without any information provided.

[julia-users] What is the best way to element-wise right shift an array?

2016-09-18 Thread David P. Sanders 
You should also benchmark the simple for loop. Please report back with the 
results.

Re: [julia-users] Read a stuctured binary file with big endian unsigned integers (4 bytes) and big endian floats (4 bytes)

2016-09-18 Thread Tim Holy 
See ntoh and hton. Perhaps even better, see StrPack.jl.

--Tim

On Sunday, September 18, 2016 1:13:43 AM CDT Femto Trader wrote:
> Hello,
> 
> I'd like to read this file
> http://www.dukascopy.com/datafeed/EURUSD/2016/02/14/20h_ticks.bi5
> using Julia.
> 
> It's a LZMA compressed file.
> 
> I can decompressed it using
> cp 20h_ticks.bi5 20h_ticks.xz
> xz --decompress --format=lzma 20h_ticks.xz
> 
> Now, I have a 20h_ticks binary file.
> 
> It's a stuctured binary file with array of records
>   Date unsigned integer 4 bytes
>   Ask  unsigned integer 4 bytes
>   Bid  unsigned integer 4 bytes
>   AskVolume float 4 bytes
>   BidVolume float 4 bytes
> 
> 
> Using Python I'm able to read it and get a Pandas DataFrame
> 
> import numpy as np
> import pandas as pd
> import datetime
> symb = "EURUSD"
> dt_chunk = datetime.datetime(2016, 2, 14)
> record_dtype = np.dtype([
> ('Date', '>u4'),
> ('Ask', '>u4'),
> ('Bid', '>u4'),
> ('AskVolume', '>f4'),
> ('BidVolume', '>f4'),
> ])
> 
> data = np.fromfile("20h_ticks", dtype=record_dtype)
> columns = ["Date", "Ask", "Bid", "AskVolume", "BidVolume"]
> df = pd.DataFrame(data, columns=columns)
> if symb[3:] == "JPY":
> p_digits = 3
> else:
> p_digits = 5
> for p in ["Ask", "Bid"]:
> df[p] = df[p] / 10**p_digits
> df["Date"] = dt_chunk + pd.to_timedelta(df["Date"], unit="ms")
> df = df.set_index("Date")
> 
> I'd like to do the same using Julia
> 
> I did
> 
> symb = "EURUSD"
> day_chunk = Date(2016, 2, 14)
> h_chunk = 20
> dt_chunk = DateTime(day_chunk) + Base.Dates.Hour(h_chunk)
> filename = @sprintf "%02dh_ticks" h_chunk
> println(filename)
> 
> immutable TickRecordType
>   Date::UInt32
>   Ask::UInt32
>   Bid::UInt32
>   AskVolume::Float32
>   BidVolume::Float32
> end
> 
> f = open(filename)
> 
> # ...
> 
> close(f)
> 
> but I'm blocked now ...
> 
> Any help will be great.
> 
> Kind regards

[julia-users] Does Julia 0.5 leak memory?

2016-09-18 Thread K leo 
I run Julia through repl.  The procedure is simple: include("myfile.jl"), 
then run myfunction() (I might do a Ctrl-C to interrupt the function), edit 
something in myfile.jl, then repeat.  Initially, julia processes normally 
take less than 10% RAM, then after some time, one main Julia process takes 
over 70% of the 4GB RAM.  Even myfunction finishes running with real 
idling, it still takes that much RAM.  My code uses nearly no global 
objects, and myfunction ends with nothing as the last statement.

myfunction isn't new though I keep modifying it from time to time.  Julia 
0.4.6 does not appear to have this issue.

Any thoughts on what might be the culprit?

  | | |_| | | | (_| |  |  Version 0.5.0-rc4+0 (2016-09-09 01:43 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/   |  x86_64-pc-linux-gnu

Re: [julia-users] Re: Tutorial Julia language brazilian portuguese

2016-09-18 Thread Waldir Pimenta 
Já existe um grupo/canal do gitter: https://gitter.im/JuliaLangPt/julia --- 
eu sugeria não fragmentarmos a comunidade, que é bem pequena, com um novo 
canal de comuncação (até porque esse canal no gitter está ligado a uma 
organização do github: https://github.com/JuliaLangPt)

On Tuesday, September 13, 2016 at 4:36:40 PM UTC+1, Phelipe Wesley wrote:
>
> Criei o grupo no slack, para entrar é só acessar a url e um convite será 
> enviado para o email.
>
> https://still-dawn-96640.herokuapp.com/
>
> No dia 12 de setembro de 2016 às 18:59,  > escreveu:
>
>> olá felipe. É uma boa ideia. Se você criar eu compartilho com o pessoal 
>> da unb.
>>
>> Hello Felipe. It's a great idea. If you marry I share with the staff of 
>> unb.
>>
>> Em segunda-feira, 12 de setembro de 2016 17:54:50 UTC-3, Phelipe Wesley 
>> escreveu:
>>>
>>> O que acham de criarmos um grupo Julia-Brasil no slack?
>>>
>>
>

[julia-users] Re: Running octave scrips from julia

2016-09-18 Thread Waldir Pimenta 
Sorry, that was on me: https://github.com/JuliaPy/PyCall.jl/issues/320

Still, since this thread wasn't resolved, it may be a good thing if this 
gets exposed to more eyes, since other people may have (or have had) the 
same problem, and could comment confirming its resolution, or that it still 
occurs.

On Friday, September 9, 2016 at 11:22:44 PM UTC+1, Chris Rackauckas wrote:
>
> I guess not >1 year, but getting close to a year.
>
> On Friday, September 9, 2016 at 4:32:56 AM UTC-7, Steven G. Johnson wrote:
>>
>>
>>
>> On Friday, February 6, 2015 at 2:03:23 PM UTC-5, astromono wrote:
>>>
>>>   in pyinitialize at /home/rober/.julia/v0.4/PyCall/src/pyinit.jl:245
>>>
>>
>>  I think you must have pinned PyCall at some ancient version, because the 
>> current pyinit.jl file only has 115 lines.
>>
>

Re: [julia-users] What is the best way to element-wise right shift an array?

2016-09-18 Thread Tim Holy 
On Sunday, September 18, 2016 2:55:46 AM CDT K leo wrote:
> I have been using simply A=[0; A[1:end-1]], but found it to be somehow
> quite expensive.  I saw that there is unshift! but it has to be followed up
> with deleteat! to make the array the same size, i.e. there need to be two
> operations.  So how can I get a better performance doing the shift?

If A will always have the same length, you might use CircularBuffer. If A will 
have some predictable maximum size, you could use CircularDeque. Both of these 
are in https://github.com/JuliaLang/DataStructures.jl

--Tim

[julia-users] Re: What is the best way to element-wise right shift an array?

2016-09-18 Thread Kristoffer Carlsson 
Just because you are calling two functions doesn't mean it is slow. Have 
you benchmarked?


On Sunday, September 18, 2016 at 11:55:47 AM UTC+2, K leo wrote:
>
> I have been using simply A=[0; A[1:end-1]], but found it to be somehow 
> quite expensive.  I saw that there is unshift! but it has to be followed up 
> with deleteat! to make the array the same size, i.e. there need to be two 
> operations.  So how can I get a better performance doing the shift?
>

[julia-users] What is the best way to element-wise right shift an array?

2016-09-18 Thread K leo 
I have been using simply A=[0; A[1:end-1]], but found it to be somehow 
quite expensive.  I saw that there is unshift! but it has to be followed up 
with deleteat! to make the array the same size, i.e. there need to be two 
operations.  So how can I get a better performance doing the shift?

[julia-users] Read a stuctured binary file with big endian unsigned integers (4 bytes) and big endian floats (4 bytes)

2016-09-18 Thread Femto Trader 
Hello,

I'd like to read this file
http://www.dukascopy.com/datafeed/EURUSD/2016/02/14/20h_ticks.bi5
using Julia.

It's a LZMA compressed file.

I can decompressed it using
cp 20h_ticks.bi5 20h_ticks.xz
xz --decompress --format=lzma 20h_ticks.xz

Now, I have a 20h_ticks binary file.

It's a stuctured binary file with array of records 
  Date unsigned integer 4 bytes
  Ask  unsigned integer 4 bytes
  Bid  unsigned integer 4 bytes
  AskVolume float 4 bytes
  BidVolume float 4 bytes


Using Python I'm able to read it and get a Pandas DataFrame

import numpy as np
import pandas as pd
import datetime
symb = "EURUSD"
dt_chunk = datetime.datetime(2016, 2, 14)
record_dtype = np.dtype([
('Date', '>u4'),
('Ask', '>u4'),
('Bid', '>u4'),
('AskVolume', '>f4'),
('BidVolume', '>f4'),
])

data = np.fromfile("20h_ticks", dtype=record_dtype)
columns = ["Date", "Ask", "Bid", "AskVolume", "BidVolume"]
df = pd.DataFrame(data, columns=columns)
if symb[3:] == "JPY":
p_digits = 3
else:
p_digits = 5
for p in ["Ask", "Bid"]:
df[p] = df[p] / 10**p_digits
df["Date"] = dt_chunk + pd.to_timedelta(df["Date"], unit="ms")
df = df.set_index("Date")

I'd like to do the same using Julia

I did

symb = "EURUSD"
day_chunk = Date(2016, 2, 14)
h_chunk = 20
dt_chunk = DateTime(day_chunk) + Base.Dates.Hour(h_chunk)
filename = @sprintf "%02dh_ticks" h_chunk
println(filename)

immutable TickRecordType
  Date::UInt32
  Ask::UInt32
  Bid::UInt32
  AskVolume::Float32
  BidVolume::Float32
end

f = open(filename)

# ...

close(f)

but I'm blocked now ...

Any help will be great.

Kind regards