Re: [julia-users] 900mb csv loading in Julia failed: memory comparison vs python pandas and R

Jacob I do run into the same issue as Grey. the step ds = DataStreams.DataTable(f); gets stuck. I also tried this with a smaller file (150MB) which I have. This file is read by readtable in 15s. But the DataTable function freezes. I use 0.4 on Windows 7. I note that your code did work on a tiny

Re: [julia-users] 900mb csv loading in Julia failed: memory comparison vs python pandas and R

Even after letting it run for more than 2 hours it didn't finish, so I ended the process. Now compiling Julia 0.4 in a google cloud instance. I will try it there, where it has higher memory than my local machine. On Wednesday, October 14, 2015 at 10:24:16 AM UTC+5:30, Grey Marsh wrote: > > I am

[julia-users] Re: Julia Docker images

I've also been working on a set of containers on docker hub for source builds of dependencies and potential CI uses. Will see if anything interesting comes out of it. Travis lets you use custom docker images as a new beta feature, though there may be a potential issue with UDP tests.

Re: [julia-users] 900mb csv loading in Julia failed: memory comparison vs python pandas and R

I am using Julia 0.4 for this purpose, if that's what is meant by "0.4 only". On Wednesday, October 14, 2015 at 9:53:09 AM UTC+5:30, Jacob Quinn wrote: > > Oh yes, I forgot to mention that the CSV/DataStreams code is 0.4 only. > Definitely interested to hear about any results/experiences though

Re: [julia-users] 900mb csv loading in Julia failed: memory comparison vs python pandas and R

Oh yes, I forgot to mention that the CSV/DataStreams code is 0.4 only. Definitely interested to hear about any results/experiences though. -Jacob On Tue, Oct 13, 2015 at 10:11 PM, Yichao Yu wrote: > On Wed, Oct 14, 2015 at 12:02 AM, Grey Marsh wrote: > > @Jacob, I tried your approach. Somehow

Re: [julia-users] 900mb csv loading in Julia failed: memory comparison vs python pandas and R

On Wed, Oct 14, 2015 at 12:02 AM, Grey Marsh wrote: > @Jacob, I tried your approach. Somehow it got stuck in the "@time ds = > DataStreams.DataTable(f)" line. After 15 minutes running, julia is using > ~500mb and 1 cpu core with no sign of end. The memory use has been almost > same for the whole d

Re: [julia-users] 900mb csv loading in Julia failed: memory comparison vs python pandas and R

@Jacob, I tried your approach. Somehow it got stuck in the "@time ds = DataStreams.DataTable(f)" line. After 15 minutes running, julia is using ~500mb and 1 cpu core with no sign of end. The memory use has been almost same for the whole duration of 15 minutes. I'm letting it run, hoping that it

[julia-users] Re: Markdown.parse question

Whether it renders as $ or $$ is inferred from the position, if it's inline it uses $ if it's a block $$. julia> Markdown.latex(Markdown.parse("""\$\\sin(x)\$""")) "\$\$\\sin(x)\$\$" julia> Markdown.latex(Markdown.parse("""inline \$\\sin(x)\$""")) "inline \$\\sin(x)\$\n" https://github.com/Ju

Re: [julia-users] How to correctly do pivoted QR in Julia 0.4+?

On Tue, Oct 13, 2015 at 5:48 PM, Victor Minden wrote: > I posted this originally as an issue on the GitHub page but was redirected > here, which seems like a great resource. Also, seems like this is more just > my misunderstanding than an issue with the code, so this should be more > appropriate.

[julia-users] Markdown.parse question

With v0.4, is there a way to have Markdown parse latex and tell the difference between inline math and display math? In particular, this yields two identical pieces: ``` julia> macro L_mstr(x) x end julia> Markdown.parse(L""" $\sin(x)$ $$\sin(x)$$""", flavor=:julia).content 2-

[julia-users] julia newb seeks critique of a "Defstruct" clone macro

this is day3 of julia so pardon dumb questions i had some trouble finding tutorials on julia macros. i've read the textbooks but i suspect there is somewhere else to look (i say that since i would have thought that the following would exist in standard Julia, but i could not find it). so, two

[julia-users] How to correctly do pivoted QR in Julia 0.4+?

I posted this originally as an issue on the GitHub page but was redirected here, which seems like a great resource. Also, seems like this is more just my misunderstanding than an issue with the code, so this should be more appropriate. With Julia 0.4+, it seems that using F = qr(A,pivot=true)

[julia-users] Azure interface for Julia

Hello, Is there any Azure interface for Julia like following interface for EC2? https://github.com/amitmurthy/AWS.jl Anyways, we appretiated any help on how we could develop an Azure interface for Julia. Thanks! -- *Raphael Pereira Ribeiro* *Instituto de Computação - IC/UFAL* *Graduando em Ci

[julia-users] Re: Julia Docker images

There's an official Julia docker repository now: https://hub.docker.com/_/julia/ There is a PR to uptdate it to 0.4: https://github.com/docker-library/official-images/pull/1121/files I'm working on creating Dockerfiles for versions of Julia that support Gallium (the debugger) and Cxx (the C++

[julia-users] Re: ANN: A potential new Discourse-based Julia forum

Also just realized there's extensive support for oneboxing (smart inline expansion of links) http://julia.malmaud.com/t/testing-oneboxing/3205/1 On Tuesday, October 13, 2015 at 8:27:37 PM UTC-4, Jonathan Malmaud wrote: > > I installed a mathjax plugin to Discourse. Try it out! > http://julia.ma

[julia-users] Re: ANN: A potential new Discourse-based Julia forum

I installed a mathjax plugin to Discourse. Try it out! http://julia.malmaud.com/t/testing-mathjax-plugin/3203/1 On Tuesday, October 13, 2015 at 4:51:04 PM UTC-4, feza wrote: > > Wow this looks great. Much better than google groups which is rather > annoying in many respects. Looking forward to u

Re: [julia-users] Parameter type

On Tue, Oct 13, 2015 at 8:08 PM, wrote: > Hi, I would like to know the difference between: I assume this is a dup of https://groups.google.com/forum/?fromgroups=#!topic/julia-users/WIVyaMMqfyo > > function f1(::Int64) > > and > > function f2(::Type{Int64}) > > > when to use which ? > > Thanks

[julia-users] Parameter type

Hi, I would like to know the difference between: function f1(::Int64) and function f2(::Type{Int64}) when to use which ? Thanks

[julia-users] Escher image update

I have some widgets that run imagemagick updating images in `assets/`, and the only way I can get those to update with `image` is to change their names... So if I just do this: ```julia run(`convert a.jpg assets/a.jpg`) image("assets/a.jpg") ``` the image doesn't get updated. But this crazine

[julia-users] Re: Function push! adds objects where it should not

Thank you to both of you, this makes sense. However I'm a bit surprised that this code was working in 0.3.11, I'm certain of that, so the fill! function was using different objects each time? I'd have thought that such an important thing as evaluation of function arguments would not change from

Re: [julia-users] 900mb csv loading in Julia failed: memory comparison vs python pandas and R

On Tue, Oct 13, 2015 at 4:21 PM, Grey Marsh wrote: > I was using 0.3.10. How would gc come in the picture? I mean gc would be > called only after the file has been read into memory completely. nyway let > me try the v0.4. Well, there are also intermediate objects that needs to be allocated. See

[julia-users] Re: ANN: NBInclude.jl --- include() for IJulia notebooks

On Tuesday, October 13, 2015 at 5:04:04 PM UTC-4, Kristoffer Carlsson wrote: > > General programming in notebooks is a very interesting concept. Too bad > git diff isn't that great with it. There is this however: > https://github.com/tarmstrong/nbdiff There are also various json-diff tools.

[julia-users] Re: Function push! adds objects where it should not

Function arguments are evaluated once. Rewrite fill!(instance.ol, Tuple{Int64, Array{Float64}}[]) to x = Tuple{Int64, Array{Float64}}[] fill!(instance.ol, x) and you see that you are filling the instance with the exact same object. On Tuesday, October 13, 2015 at 6:52:15 PM UTC+2, ami...@gm

[julia-users] Re: ANN: NBInclude.jl --- include() for IJulia notebooks

General programming in notebooks is a very interesting concept. Too bad git diff isn't that great with it. There is this however: https://github.com/tarmstrong/nbdiff

Re: [julia-users] 900mb csv loading in Julia failed: memory comparison vs python pandas and R

I'm hesitant to suggest, but if you're in a bind, I have an experimental package for fast CSV reading. The API has stabilized somewhat over the last week and I'm planning a more broad release soon, but I'd still consider it alpha mode. That said, if anyone's willing to give it a drive, you just nee

[julia-users] Re: ANN: A potential new Discourse-based Julia forum

Wow this looks great. Much better than google groups which is rather annoying in many respects. Looking forward to using this sometime in the future. Do you think mathjax support for latex equations would be useful for a Julia forum? On Saturday, September 19, 2015 at 8:16:36 PM UTC-4, Jonatha

Re: [julia-users] 900mb csv loading in Julia failed: memory comparison vs python pandas and R

Finally was able to load it, but the process consumes a ton of memory. julia> @time train = readtable("./test.csv"); 124.575362 seconds (376.11 M allocations: 13.438 GB, 10.77% gc time) On Tuesday, October 13, 2015 at 4:34:05 PM UTC-4, feza wrote: > > Same here on a 12gb ram machine > >

Re: [julia-users] 900mb csv loading in Julia failed: memory comparison vs python pandas and R

Same here on a 12gb ram machine _ _ _ _(_)_ | A fresh approach to technical computing (_) | (_) (_)| Documentation: http://docs.julialang.org _ _ _| |_ __ _ | Type "?help" for help. | | | | | | |/ _` | | | | |_| | | | (_| | | Version 0.5.0-d

Re: [julia-users] 900mb csv loading in Julia failed: memory comparison vs python pandas and R

I was using 0.3.10. How would gc come in the picture? I mean gc would be called only after the file has been read into memory completely. nyway let me try the v0. On Wednesday, October 14, 2015 at 1:17:58 AM UTC+5:30, Yichao Yu wrote: > > > On Oct 13, 2015 2:47 PM, "Grey Marsh" > > wrote: > >

Re: [julia-users] 900mb csv loading in Julia failed: memory comparison vs python pandas and R

I was using 0.3.10. How would gc come in the picture? I mean gc would be called only after the file has been read into memory completely. nyway let me try the v0.4. On Wednesday, October 14, 2015 at 1:17:58 AM UTC+5:30, Yichao Yu wrote: > > > On Oct 13, 2015 2:47 PM, "Grey Marsh" > > wrote: >

Re: [julia-users] Re: non-responsive REPL with PyCall

Note that PyCall supports passing Julia I/O streams to Python. So, if you can just monkey-patch the Python library to use a Julia I/O stream to talk to the spectrometer, then it will yield to Julia's event loops.

Re: [julia-users] 900mb csv loading in Julia failed: memory comparison vs python pandas and R

On Oct 13, 2015 2:47 PM, "Grey Marsh" wrote: Which julia version are you using. There's sime gc tweak on 0.4 for that. > > I was trying to load the training dataset from springleaf marketing response on Kaggle. The csv is 921 mb, has 145321 row and 1934 columns. My machine has 8 gb ram and julia

[julia-users] 900mb csv loading in Julia failed: memory comparison vs python pandas and R

I was trying to load the training dataset from springleaf marketing response on Kaggle. The csv is 921 mb, has 145321 row and 1934 columns. My machine has 8 gb ram and julia ate 5.8gb+ memory after that I stopped julia as there was barely

[julia-users] Re: precompile error for julia debugger

Tony, Thanks. I put your fixes in my local files. They work perfectly. The bug is totally fixed. K On Tuesday, October 13, 2015 at 6:46:45 PM UTC+1, Tony Kelman wrote: > > You can always make these changes to your local copies of the packages to > fix things right away. Just be sure to undo

Re: [julia-users] Re: The Julia Community Standards

Like Tomas I fully support both the standards and the attempts at enforcement, and also didn't feel like further derailing another thread to mention that. But I wanted to reinforce his point that the number of silent supports is larger than it appears.

[julia-users] Which public key is Julia 0.4.0 Pkg.Git trying to use on Windows?

Does it still work if you try with an rc now? Check your github settings to see if all your keys are still current and valid. Sometimes github revokes things if they've been inactive for a while.

[julia-users] Issue with profiler?

This is likely an issue for the ProfileView.jl package which was recently changed over to use Gtk. Be sure to provide a self contained example that shows the problem if you can.

[julia-users] Re: precompile error for julia debugger

You can always make these changes to your local copies of the packages to fix things right away. Just be sure to undo them later once tagged versions of the packages that include the fixes are available so you can update them. Ellen, exactly what did you try to run? What does versioninfo(true) s

[julia-users] Re: Function push! adds objects where it should not

perhaps, fill!(instance.ol, Tuple{Int64, Array{Float64}}[]) uses the same address to fill every entry in `instance.ol` and the behavior observed follows. maybe, onstance.ol = [Tuple{Int64,Array{Float64}}[] for i=1:length(instance.ol)] would be better. On Tuesday, October 13, 2015 at 7:52

[julia-users] Re: precompile error for julia debugger

Tony, Thanks for the fixes. Hope that your commit can be merged quickly, as I can't do much at my end to fix this at the moment. Kuan On Tuesday, October 13, 2015 at 3:54:39 AM UTC+1, Tony Kelman wrote: > > I think this might be caused by a bug in Compat.jl if you had loaded that > package b

Re: [julia-users] Re: is there a way to access help from within emacs shell?

I also submitted a patch for ESS. non-base help / completion etc which were broken in Julia 0.4 should work now. On Mon, Oct 12, 2015 at 8:08 PM Deniz Yuret wrote: > Thank you all. @doc was exactly what I was looking for. > > best, > deniz > > On Mon, Oct 12, 2015 at 5:40 PM David Ellison > wro

Re: [julia-users] Re: how does one locate a macro?

Maybe we should make `@which @time` work... On Tue, Oct 13, 2015 at 8:51 PM, Deniz Yuret wrote: > I was wondering the same thing myself. I could not find anything in > Base. Here is something that works, at least in Julia 0.4, based on > base/reflection.jl:functionloc() > Use it with: `macrolo

Re: [julia-users] Re: non-responsive REPL with PyCall

Yeah, this is a fundamental problem, not only with Python but any other C library that uses blocking calls. At one point, when wrapping such a C library, I had the slightly insane notion of using LD_PRELOAD to replace all the blocki

[julia-users] Function push! adds objects where it should not

Hi all, I have this code with two types, one is an observation with a name, a time, and a vector of observed values. Another is a system where observations are stored, it comprises a list of names nl which is an index of all the observation names, an observation list ol which is a vector of tup

[julia-users] Function push! adds objects where it should not

Hi all, I have this code with two types, one is an observation with a name, a time, and a vector of observed values. Another is a system where observations are stored, it comprises a list of names nl which is an index of all the observation names, an observation list ol which is a vector of tup

[julia-users] Re: precompile error for julia debugger

I'm getting the same error, and I'm not using Compat. Are you saying I should try to load this? The only package I'm using also is PyPlot. Please help! On Monday, October 12, 2015 at 9:54:39 PM UTC-5, Tony Kelman wrote: > > I think this might be caused by a bug in Compat.jl if you had loaded tha

[julia-users] Re: ANN: NBInclude.jl --- include() for IJulia notebooks

Once Jupyter has a UI for cell metadata tags (https://github.com/jupyter/notebook/issues/601), it will be easy to include things like unit tests and examples in the notebook which are ignored by `nbinclude`. I'd also like to have a `doc` tag so that you can create a formatted markdown cell an

[julia-users] Issue with profiler?

I figured I'd post this here before going to github, in case it's a fault on my end rather than with Julia itself. I'm trying to use the profiler, and for the most part it seems to work... somewhat. However, if I try to profile something that takes more than 5-6 seconds, I sometimes get an erro

[julia-users] Re: how does one locate a macro?

I was wondering the same thing myself. I could not find anything in Base. Here is something that works, at least in Julia 0.4, based on base/reflection.jl:functionloc() Use it with: `macroloc(:@time)` function macroloc(expr::Expr) @assert expr.head == :macrocall f=getfield(current_modu

[julia-users] Which public key is Julia 0.4.0 Pkg.Git trying to use on Windows?

Since updating to 0.4.0, I’ve had permissions problems when trying to update my package list: julia> Pkg.update() INFO: Updating METADATA... Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exist

[julia-users] Re: Embedding Julia under Windows 7

Thank you. I appreciate your efforts. Am Montag, 12. Oktober 2015 22:18:54 UTC+2 schrieb Tracy Wadleigh: > > I've submitted an issue: > > *https://github.com/JuliaLang/julia/issues/13570 > * > > > On Monday, October 12, 2015 at 1:45:24 AM UTC-4, be

[julia-users] download cache in Windows

I'm currently using download() to get dlls on Windows. This works fine, except that it has a cache and I don't know how to clear it. Even days after a file is first downloaded, and even after that file is updated at its URL, it simply copies over the cached file instead of downloading the new v

[julia-users] Re: new function to calculate cycle basis in connected graphs

The Graphs.jl package is maintained by volunteers but has no active owner. You may want to look at contributing to the https://github.com/JuliaGraphs/LightGraphs.jl package instead. On Tuesday, October 13, 2015 at 6:28:53 AM UTC-4, Michela Di Lullo wrote: > > Hello, > > I would share into the Gr

Re: [julia-users] Re: non-responsive REPL with PyCall

I'll try the remotecall option (next week), hopefully the fetch from that won't get things stuck. I imagine there are a lot of *other* Python libraries that don't use libuv and get Julia's coroutines stuck... Thanks for all the attention! Yakir Gagnon The Queensland Brain Institute (Building #79

Re: [julia-users] Re: non-responsive REPL with PyCall

I'm working on fixing up the API to Amit's PR here that allows you to call a C function in another thread. That could also potentially be used for this. On Tue, Oct 13, 2015 at 2:43 PM, Mohammed El-Beltagy < mohammed.elbelt...@gmail.com> wrote: > Sh

[julia-users] Re: ANN: NBInclude.jl --- include() for IJulia notebooks

Seconded, thank you very much!

[julia-users] new function to calculate cycle basis in connected graphs

Hello, I would share into the Graphs.jl package a new function to calculate cycle basis in connected graphs. I looked at the source codes of some functions in the package and actually they use a specific language that is not exhaustively explained anywhere (as far as I know). So my question

Re: [julia-users] Re: non-responsive REPL with PyCall

Short of doing a reimplementation, you could possibly run your python code in another process via a remotecall (as described in the manual http://julia.readthedocs.org/en/latest/manual/parallel-computing/). In that case you REPL would be responsive as the I/O is done in another process. On Mon

[julia-users] how does one locate a macro?

If want to look up a method is julia, say I am interested in the implementation of the `sine` function, I just run ```julia julia> methods(sin) # 11 methods for generic function "sin": sin(a::Complex{Float16}) at float16.jl:151 sin(z::Complex{T<:Real}) at complex.jl:548 sin(x::Float64) at math.

Re: [julia-users] convert a list of ASCIIString into Any

It looks you it would be so much easier if you just run readcvs("test.csv"). On Tuesday, October 13, 2015 at 5:41:27 AM UTC+2, masa charlie wrote: > > Hi, Jacob > > Thanks for the reply, and the way you showed to use "enumerate()" is > really nice to know of, and it has eventually given me a gr