Re: [julia-users] using or import a specific package version

[Alex Mellnik]

I'm really looking forward to this -- knowing that all my existing projects 
and tools will continue to operate in the same way after running 
Pkg.update() will be very welcome.  

On Friday, November 4, 2016 at 4:03:45 PM UTC-7, Stefan Karpinski wrote:
>
> In Pkg3, if your script uses a particular environment, which can be 
> project-local (i.e. local to the repository the script is in) then as long 
> as you don't change the version of anything in that environment, it won't 
> change. I wasn't planning on having a way of indicating a particular 
> version in the source code at the point of loading something, however.
>
> On Fri, Nov 4, 2016 at 6:30 PM, Tom Breloff  > wrote:
>
>> Not that I know of. Hopefully the next generation package manager (Pkg3) 
>> will enable this sort of conditional pinning. It should at least enable 
>> dual environments. 
>>
>>
>> On Friday, November 4, 2016, Tom Lee  
>> wrote:
>>
>>> Hi all,
>>>
>>> Is there any way to specify that "using" or "import" should load a 
>>> specific package version without pinning the package to a particular 
>>> version globally? I would like to prevent my older scripts breaking when 
>>> packages are updated while still using the latest version by default.
>>>
>>> Thanks,
>>>
>>> Tom
>>>
>>
>

[julia-users] Re: Webapp Deployment

[Alex Mellnik]

Hi Reuben,

I largely work in this space.  I can walk through a few possible 
architectures that I have used:

1)  If the data pull and processing is fairly decoupled from the display, 
it is often easiest to use Julia only on the back end.  I have a few 
systems that pull new data every hour and add it to a database.  I then use 
Julia to do a bunch of processing and analysis, then load the results back 
into a different portion of the database.  You can then build a normal 
data-driven web app using standard tools that only looks at the database, 
and doesn't need to interface with Julia anywhere.  If you don't have any 
experience building web pages, I would suggest using Angular 1 and Plotly 
for the front end, and Node/Express for the back end.  Some basic data 
manipulation can be done via SQL if you use MySQL or similar, or in the web 
app itself using things like d3 and lodash.  I don't have any publicly 
available examples of this, but I could put one together if you like.

2) If the data pull and processing is strongly coupled to the data display, 
you can call Julia directly from a server-side web application rather than 
look at cached data.  You have a few options for the server-side code.  One 
is to call Julia from Node using node-julia 
. I have a rough example of how you 
would do this here .  One risk is 
that while node-julia works, it's a bit tricky to use, and I don't know 
what Jeff's plans for the package are.  You would again use a normal 
front-end tools like Angular/React for the front end.  

Alternatively, you could write the back-end in something like Mux.jl 
 rather than in Node.  I don't do this, 
because I need to use things like https and SSPI in an enterprise 
environment, but I think it should work fine.  

3) Lastly, you could write the whole thing in Julia using something like 
Escher or Genie.jl .  These are 
both very interesting projects and represent an incredible level of work, 
but I don't think they are ready for production use yet.  

I strongly recommend the first option if possible.  It might seem like a 
bunch of different parts that all need to work together, but I think it's 
actually the easiest to set up and maintain, and lets you use the use the 
best tools in each domain.  Failing that, try the 2nd.  

If you have any questions or would like to discuss this further just let me 
know.  Cheers,

Alex


On Monday, October 31, 2016 at 9:08:01 PM UTC-7, Reuben Brooks wrote:
>
> Context: I love julia, and I've never built any kind of webapp. Most of my 
> programming experience is in Mathematica and Julia...hacking things 
> together (poorly) in Python when nothing else works.
>
> Problem: I have a script  / notebook in julia that pulls data from 
> sources, analyzes it, builds fancy plots, and has lots of nice information. 
> Now I want to build a basic webapp that will allow me to access this 
> information anywhere, anytime (will be updated regularly). 
>
> Question 1: is there a julia package that suits my needs well, or should I 
> look at using some other fronted to create the frontend? Elm intrigues me, 
> as much for the learning as for the actual solution. 
>
> Bottom line: I don't know enough about what I'm wading into to choose 
> wisely. What does the community suggest?
>

[julia-users] Re: Writing a subset DataFrame to file is 220 times slower than saving the whole DataFrame

[Alex Mellnik]

I'm not sure what's wrong with sub, but don't use it -- it's definitely 
worse than just making a copy of the subset you want to write.  

s = df[df[:rank_PV].<=r_max,:]
@time write_results(s, name, "significant", sep, h)



On Thursday, October 27, 2016 at 5:07:31 AM UTC-7, Fred wrote:
>
> Hi,
>
> In the same program,  I save in a file a DataFrame "df" and a subset of 
> this DataFrame in another file. The problem I have is that saving the 
> subset is much slower than saving the entire DataFrame : 220 times slower. 
> It is too slow and I don't what is my mistake.
>
> Thank you for your advices !
>
> in Julia 0.4.5 : 
>
> Saving the entire DataFrame
> Saving... results/Stat.csv
> 1.115944 seconds (13.78 M allocations: 319.534 MB, 2.59% gc time)
>
>
> Saving the subset of the DataFrame 
> Saving... significant/Stat.csv
> 246.099835 seconds (41.79 M allocations: 376.189 GB, 4.77% gc time)
> elapsed time: 251.581459853 seconds
>
>
> in Julia 0.5 : 
>
> Saving the entire DataFrame
> Saving... results/Stat.csv
> 1.060365 seconds (7.08 M allocations: 116.025 MB, 0.73% gc time)
>
> Saving the subset of the DataFrame 
> Saving... significant/Stat.csv
> 226.813587 seconds (37.40 M allocations: 376.268 GB, 2.42% gc time)
> elapsed time: 232.95933586 seconds
>
> 
> # my function to save the results to a file
>
> function write_results(x, name, dir, sep, h)
>   outfile = "$dir/$name"
>   println("Saving...\t", outfile)
>   writetable( outfile, x, separator = sep, header = h)
> end
>
>
> # save my DataFrame df : very fast
> @time write_results(df, name, "results", sep, h)
>
>
> # subset DataFrame s
> s = sub(df, (df[:rank_PV] .<= r_max))
>
> # save my subset DataFrame s : incredibly slow !
>
> @time write_results(s, name, "significant", sep, h)
>
>
>

Re: [julia-users] Can't overwrite some methods in 0.5.0

[Alex Mellnik]

Thanks, after 
seeing https://github.com/JuliaLang/julia/issues/265#issuecomment-243056854 
I have a better idea of what's going on and why it doesn't occur in 0.5.  

On Wednesday, October 19, 2016 at 11:59:08 PM UTC-7, Yichao Yu wrote:
>
>
>
> On Wed, Oct 19, 2016 at 10:33 PM, Alex Mellnik <a.r.m...@gmail.com 
> > wrote:
>
>> Yichao,
>>
>> I'm afraid I'm not following -- could you expand on that a bit?  Thanks,
>>
>
> https://github.com/JuliaLang/julia/issues/265
>  
>
>>
>> Alex
>>
>> On Wednesday, October 19, 2016 at 4:41:30 PM UTC-7, Yichao Yu wrote:
>>
>>> On Oct 19, 2016 7:26 PM, "Alex Mellnik" <a.r.m...@gmail.com> wrote:
>>> >
>>> > Here's my bizarre find of the day.  Most functions can be overwritten 
>>> without problems:
>>> >
>>> > function add7(i)
>>> > 7 + i
>>> > end
>>> > Out[1]:
>>> > add7 (generic function with 1 method)
>>> > In [2]:
>>> >
>>> > add7(0)
>>> > add7(0)
>>> > Out[2]:
>>> > 7
>>> > In [3]:
>>> >
>>> > function add7(i)
>>> > 9 + i
>>> > end
>>> > function add7(i)
>>> > 9 + i
>>> > end
>>> > Out[3]:
>>> > add7 (generic function with 1 method)
>>> > WARNING: Method definition add7(Any) in module Main at In[1]:2 
>>> overwritten at In[3]:2.
>>> > In [4]:
>>> >
>>> > add7(0)
>>> > Out[4]:
>>> > 9
>>> >
>>> > However, others can not:
>>> >
>>> > using DataFrames
>>> > df = DataFrame(A=[1,2,3], B=["A", "B", "C"])
>>> > println(df)
>>> > 3×2 DataFrames.DataFrame
>>> > │ Row │ A │ B   │
>>> > ├─┼───┼─┤
>>> > │ 1   │ 1 │ "A" │
>>> > │ 2   │ 2 │ "B" │
>>> > │ 3   │ 3 │ "C" │
>>> > In [3]:
>>> >
>>> > row[:A] > 2
>>> > function filter(row)
>>> > if row[:A] > 2
>>> > return 1
>>> > else
>>> > return 3
>>> > end
>>> > end  
>>> > Out[3]:
>>> > filter (generic function with 1 method)
>>> > In [4]:
>>> >
>>> > [filter(row) for row in eachrow(df)]
>>> > [filter(row) for row in eachrow(df)]
>>> > Out[4]:
>>> > 3-element Array{Int64,1}:
>>> >  3
>>> >  3
>>> >  1
>>> > In [5]:
>>> >
>>> > rand() > 0.5
>>> > function filter(row)
>>> > if row[:A] > 2
>>> > return 2
>>> > else
>>> > return 4
>>> > end
>>> > end  
>>> > WARNING: Method definition filter(Any) in module Main at In[3]:2 
>>> overwritten at In[5]:2
>>> > Out[5]:
>>> > filter (generic function with 1 method)
>>> > .
>>> > In [6]:
>>> >
>>> > [filter(row) for row in eachrow(df)]
>>> > Out[6]:
>>> > 3-element Array{Int64,1}:
>>> >  3
>>> >  3
>>> >  1
>>> >
>>> > What is it about this second example that prevents the newer method 
>>> from being used?
>>>
>>> Nothing about it but how you use it. It's inlined to the comprehension.
>>>
>>>
>

Re: [julia-users] Can't overwrite some methods in 0.5.0

[Alex Mellnik]

Yichao,

I'm afraid I'm not following -- could you expand on that a bit?  Thanks,

Alex

On Wednesday, October 19, 2016 at 4:41:30 PM UTC-7, Yichao Yu wrote:
>
> On Oct 19, 2016 7:26 PM, "Alex Mellnik" <a.r.m...@gmail.com > 
> wrote:
> >
> > Here's my bizarre find of the day.  Most functions can be overwritten 
> without problems:
> >
> > function add7(i)
> > 7 + i
> > end
> > Out[1]:
> > add7 (generic function with 1 method)
> > In [2]:
> >
> > add7(0)
> > add7(0)
> > Out[2]:
> > 7
> > In [3]:
> >
> > function add7(i)
> > 9 + i
> > end
> > function add7(i)
> > 9 + i
> > end
> > Out[3]:
> > add7 (generic function with 1 method)
> > WARNING: Method definition add7(Any) in module Main at In[1]:2 
> overwritten at In[3]:2.
> > In [4]:
> >
> > add7(0)
> > Out[4]:
> > 9
> >
> > However, others can not:
> >
> > using DataFrames
> > df = DataFrame(A=[1,2,3], B=["A", "B", "C"])
> > println(df)
> > 3×2 DataFrames.DataFrame
> > │ Row │ A │ B   │
> > ├─┼───┼─┤
> > │ 1   │ 1 │ "A" │
> > │ 2   │ 2 │ "B" │
> > │ 3   │ 3 │ "C" │
> > In [3]:
> >
> > row[:A] > 2
> > function filter(row)
> > if row[:A] > 2
> > return 1
> > else
> > return 3
> > end
> > end  
> > Out[3]:
> > filter (generic function with 1 method)
> > In [4]:
> >
> > [filter(row) for row in eachrow(df)]
> > [filter(row) for row in eachrow(df)]
> > Out[4]:
> > 3-element Array{Int64,1}:
> >  3
> >  3
> >  1
> > In [5]:
> >
> > rand() > 0.5
> > function filter(row)
> > if row[:A] > 2
> > return 2
> > else
> > return 4
> > end
> > end  
> > WARNING: Method definition filter(Any) in module Main at In[3]:2 
> overwritten at In[5]:2
> > Out[5]:
> > filter (generic function with 1 method)
> > .
> > In [6]:
> >
> > [filter(row) for row in eachrow(df)]
> > Out[6]:
> > 3-element Array{Int64,1}:
> >  3
> >  3
> >  1
> >
> > What is it about this second example that prevents the newer method from 
> being used?
>
> Nothing about it but how you use it. It's inlined to the comprehension.
>
>

[julia-users] Can't overwrite some methods in 0.5.0

[Alex Mellnik]

Here's my bizarre find of the day.  Most functions can be overwritten 
without problems:

function add7(i)
7 + i
end
Out[1]:
add7 (generic function with 1 method)
In [2]:

add7(0)
add7(0)
Out[2]:
7
In [3]:

function add7(i)
9 + i
end
function add7(i)
9 + i
end
Out[3]:
add7 (generic function with 1 method)
WARNING: Method definition add7(Any) in module Main at In[1]:2 overwritten 
at In[3]:2.
In [4]:

add7(0)
Out[4]:
9

However, others can not:

using DataFrames
df = DataFrame(A=[1,2,3], B=["A", "B", "C"])
println(df)
3×2 DataFrames.DataFrame
│ Row │ A │ B   │
├─┼───┼─┤
│ 1   │ 1 │ "A" │
│ 2   │ 2 │ "B" │
│ 3   │ 3 │ "C" │
In [3]:

row[:A] > 2
function filter(row)
if row[:A] > 2
return 1
else
return 3
end
end  
Out[3]:
filter (generic function with 1 method)
In [4]:

[filter(row) for row in eachrow(df)]
[filter(row) for row in eachrow(df)]
Out[4]:
3-element Array{Int64,1}:
 3
 3
 1
In [5]:

rand() > 0.5
function filter(row)
if row[:A] > 2
return 2
else
return 4
end
end  
WARNING: Method definition filter(Any) in module Main at In[3]:2 
overwritten at In[5]:2
Out[5]:
filter (generic function with 1 method)
.
In [6]:

[filter(row) for row in eachrow(df)]
Out[6]:
3-element Array{Int64,1}:
 3
 3
 1

What is it about this second example that prevents the newer method from 
being used?

Re: [julia-users] Filtering DataFrame with a function

[Alex Mellnik]

Hi Júlio,

If you're just interested in using an arbitrary function to filter on rows 
you can do something like:

df = DataFrame(Fish = ["Amir", "Betty", "Clyde"], Mass = [1.2, 3.3, 0.4])
filter(row) = (row[:Fish][1] != "A")&(row[:Mass]>1)
df = df[[filter(r) for r in eachrow(df)],:]

Is that what you're looking for?  If not, can you give an example of what 
you want to do?

Best,

Alex

On Wednesday, October 12, 2016 at 10:20:52 PM UTC-7, Júlio Hoffimann wrote:
>
> Thank you very Much David, these queries you showed are really nice. I 
> meant that ideally I wouldn't need to install another package for a simple 
> filter operation on the rows.
>
> -Júlio
>
> 2016-10-12 22:14 GMT-07:00 :
>
>> Were you worried about Query being not lightweight enough in terms of 
>> overhead, or in terms of syntax?
>>
>> I just added a more lightweight syntax for this scenario to Query. You 
>> can now do the following two things:
>>
>> q = @where(df, i->i.price > 30.)
>>
>> that will return a filtered iterator. You can materialize that into a 
>> DataFrame with collect(q, DataFrame).
>>
>> I also added a counting option. Turns out that is actually a LINQ query 
>> operator, and the goal is to implement all of those in Query. The syntax is 
>> simple:
>>
>> @count(df, i->i.price > 30.)
>>
>> returns the number of rows for which the filter condition is true.
>>
>> Under the hood both of these new syntax options use the normal Query 
>> machinery, this just provides a simpler syntax relative to the more 
>> elaborate things I've posted earlier. In terms of LINQ, this corresponds to 
>> the method invocation API that LINQ has. I'm still figuring out how to 
>> surface something like @count in the query expression syntax, but for now 
>> one can use it via this macro.
>>
>> All of this is on master right now, so you would have to do 
>> Pkg.checkout("Query") to get these macros.
>>
>> Best,
>> David
>>
>> On Wednesday, October 12, 2016 at 6:47:15 PM UTC-7, Júlio Hoffimann wrote:
>>>
>>> Hi David,
>>>
>>> Thank you for your elaborated answer and for writing a package for 
>>> general queries, that is great! I will keep the package in mind if I need 
>>> something more complex.
>>>
>>> I am currently looking for a lightweight solution within DataFrames, 
>>> filtering is a very common operation. Right now, I am considering 
>>> converting the DataFrame to an array and looping over the rows. I wonder if 
>>> there is a syntactic sugar for this loop.
>>>
>>> -Júlio
>>>
>>> 2016-10-12 17:48 GMT-07:00 David Anthoff :
>>>
 Hi Julio,

  

 you can use the Query package for the first part. To filter a DataFrame 
 using some arbitrary julia expression, use something like this:

  

 using DataFrames, Query, NamedTuples

  

 q = @from i in df begin

 @where 

 @select i

 end

  

 You can use any julia code in . Say your DataFrame 
 has a column called price, then you could filter like this:

  

 @where i.price > 30.

  

 The i will be a NamedTuple type, so you can access the columns either 
 by their name, or also by their index, e.g.

  

 @where i[1] > 30.

  

 if you want to filter by the first column. You can also just call some 
 function that you have defined somewhere else:

  

 @where foo(i)

  

 As long as the  returns a Bool, you should be good.

  

 If you run a query like this, q will be a standard julia iterator. 
 Right now you can’t just say length(q), although that is something I 
 should 
 probably enable at some point (I’m also looking into the VB LINQ syntax 
 that supports things like counting in the query expression itself).

  

 But you could materialize the query as an array and then look at the 
 length of that:

  

 q = @from i in df begin

 @where 

 @select i

 @collect

 end

 count = length(q)

  

 The @collect statement means that the query will return an array of a 
 NamedTuple type (you can also materialize it into a whole bunch of other 
 data structures, take a look at the documentation).

  

 Let me know if this works, or if you have any other feedback on 
 Query.jl, I’m much in need of some user feedback for the package at this 
 point. Best way for that is to open issues here 
 https://github.com/davidanthoff/Query.jl.

  

 Best,

 David

  

 *From:* julia...@googlegroups.com [mailto:julia...@googlegroups.com] *On 
 Behalf Of *Júlio Hoffimann
 *Sent:* Wednesday, October 12, 2016 5:20 PM
 *To:* julia-users 
 *Subject:* [julia-users] Filtering DataFrame with a function

  

[julia-users] Re: Changing Repositories Without Deleting METADATA

[Alex Mellnik]

I'm in a similar situation, and any public package that I work on I have to 
manually point at my fork and add the original repo as an upstream.  I 
haven't had much luck with any of the Pkg commands for things like this.  

I got very excited since I currently only can do ~10% of what I need 
through GitHub Desktop, and have to do the rest through the command line. 
 Unfortunately it looks like GitKracken doesn't have proxy support yet.  

On Monday, October 3, 2016 at 7:31:57 AM UTC-7, Chris Rackauckas wrote:
>
> Thanks for the suggestion. For some reason I never checked to see if there 
> were other GUIs, but once you mentioned it I did quite a bit of Googling. I 
> am now using GitKraken and while it will take a bit to get used to, it's 
> already improving my productivity. Thanks for the suggestion.
>
> On Sunday, October 2, 2016 at 9:31:18 PM UTC-7, Tony Kelman wrote:
>>
>> The conclusion I'd take out of that is not to use GitHub Desktop. It 
>> tries to hide too many important things like this from you. As far as git 
>> GUIs go, try SourceTree, it has a much more direct mapping between command 
>> line operations and GUI buttons.
>
>

[julia-users] Re: Is there a way to use values in a DataFrame directly in computation?

[Alex Mellnik]

This is why, IMHO, Nullables are a mess at the moment.  You have to either 
get(df[1,:A]) or otherwise extract the actual value, since very few things 
handle Nullables out of the box.  

On Monday, October 3, 2016 at 8:21:39 AM UTC-7, Min-Woong Sohn wrote:
>
> I am using DataFrames from master branch (with NullableArrays as the 
> default) and was wondering how the following should be done:
>
> df = DataFrame()
> df[:A] = NullableArray([1,2,3])
>
> The following are not allowed or return wrong values:
>
> df[1,:A] == 1   # false
> df[1,:A] > 1 # MethodError: no method matching isless(::Int64, 
> ::Nullable{Int64})
> df[3,:A] + 1 # MethodError: no method matching +(::Nullable{Int64}, 
> ::Int64)
>
> How should I get around these issues? Does anybody know if there is a plan 
> to support these kinds of computations directly?
>
>  
>
>
>
>

Re: [julia-users] Set Data Structure

[Alex Mellnik]

Jared,

You might be interested in what I consider the most useful Julia function 
that's not actually in 
Base: 
http://stackoverflow.com/questions/29661315/vectorized-in-function-in-julia 
 

vectorin(2, s)

0-dimensional Array{Bool,0}:
true


vectorin(s, 2)

3-element Array{Bool,1}:
  true
 false
 false

although perhaps I'm misunderstanding what you want to do here.  -A

On Monday, August 29, 2016 at 12:59:32 PM UTC-7, Jared Crean wrote:
>
> Here is an oddity:
>
> julia> s
> Set([2,3,1])
>
> julia> in(s, 2)
> false
>
> julia> in(2, s)
> true
>
> I would have though the first use of in would be an error because asking 
> if a set is contained in a number is not defined.  Is there some other 
> interpretation of the operation?
>
>
> On Monday, August 29, 2016 at 3:27:30 PM UTC-4, Jared Crean wrote:
>>
>> Ah, yes.  That's it.
>>
>>   Thanks,
>> Jared Crean
>>
>> On Monday, August 29, 2016 at 3:11:02 PM UTC-4, Erik Schnetter wrote:
>>>
>>> Jared
>>>
>>> Are you looking for the function `in`?
>>>
>>> -erik
>>>
>>> On Mon, Aug 29, 2016 at 3:06 PM, Jared Crean  wrote:
>>>
 I'm looking for a data structure that allows O(1) querying if a value 
 is contained in the data structure, and reasonably fast construction of 
 the 
 data structure given that the initial size is unknown (although this 
 criteria is not that strict).  I was looking at the Set in base, but I 
 can't find a way to test if a Set contains contains a particular value.  I 
 also don't know about the efficiency of appending to a Set.  Any 
 suggestions or information about Set would be appreciated.

 Jared Crean

>>>
>>>
>>>
>>> -- 
>>> Erik Schnetter  
>>> http://www.perimeterinstitute.ca/personal/eschnetter/
>>>
>>

Re: [julia-users] Blob detection and size measurement in Julia?

[Alex Mellnik]

Hi Tim,

Thanks for the update.  After pulling the master I was able to detect some 
blobs depending on what I used for the sigmas.  However, I realized the 
problem of actually tracking the areas of the blobs between frames was 
quite a bit more complicated than I originally thought, and I've punted it 
back to my brother.  Best,

Alex

On Saturday, August 27, 2016 at 4:05:30 AM UTC-7, Tim Holy wrote:
>
> Good catch. Looks like the edge-handling in `findlocalmaxima` needed to be 
> a 
> bit more refined---it was discarding results from the first and last 
> sigma- 
> values supplied by the user. 
>
> I may have fixed this in https://github.com/timholy/Images.jl/commit/ 
> 7336f35c824b15de9e4d0def8e739bdeb6ed3b3d 
> <https://github.com/timholy/Images.jl/commit/7336f35c824b15de9e4d0def8e739bdeb6ed3b3d>,
>  
> can you do `Pkg.checkout("Images")` 
> and test? 
>
> Best, 
> --Tim 
>
> On Friday, August 26, 2016 8:11:44 PM CDT Alex Mellnik wrote: 
> > Hi, 
> > 
> > I'm attempting to measure the size and position of roughly spherical, 
> > well-defined objects in images using Julia.  I don't have any previous 
> > experience working with images and was wondering if anyone could point 
> me 
> > toward the appropriate library/function. 
> > 
> > I know that there's a blob_LoG 
> > <
> http://timholy.github.io/Images.jl/stable/function_reference/#Images.blob_L 
> > oG> in Images.jl which appears to do roughly what I'm interested in, but 
> I 
> > may be mistaken and it looks my images will need pre-processing as I 
> > haven't yet been able to get a non-null result.  There's also the new 
> > ImageFeatures.jl and bindings for OpenCV, but neither have much in the 
> way 
> > of documentation yet. 
> > 
> > Thanks for any suggestions you can provide, 
> > 
> > Alex 
>
>
>

[julia-users] Blob detection and size measurement in Julia?

[Alex Mellnik]

Hi,

I'm attempting to measure the size and position of roughly spherical, 
well-defined objects in images using Julia.  I don't have any previous 
experience working with images and was wondering if anyone could point me 
toward the appropriate library/function.  

I know that there's a blob_LoG 
 
in Images.jl which appears to do roughly what I'm interested in, but I may 
be mistaken and it looks my images will need pre-processing as I haven't 
yet been able to get a non-null result.  There's also the new 
ImageFeatures.jl and bindings for OpenCV, but neither have much in the way 
of documentation yet. 

Thanks for any suggestions you can provide,

Alex

[julia-users] Re: web scraping with Julia

[Alex Mellnik]

Ivan,

What sort of contents do you need to get?  For simple things Requests.jl 
may work, but for more complex scrapes I would suggest using Selenium.  You 
can install the Python package and call it with something like 

using PyCall
@pyimport selenium.webdriver as webdriver
driver = webdriver.Firefox()
driver["get"]("http://something.net/blah/index.js;)
contents = string(driver["page_source"])
driver["quit"]()

This is a pretty simple example, but you should have a lot more flexibility 
going forward.  

-Alex


On Sunday, July 31, 2016 at 3:23:07 PM UTC-7, Ivan Pandžić wrote:
>
> Hey, guys, I have already searched a lot online, but can't seem to get 
> anything working. I have tried LibCURL.jl, Requests.jl and HTTPClient.jl. 
> Basically, what I need is something that allows me to get the contents of a 
> website. What's the best way?
>
> Thanks!
>
> Ivan
>

[julia-users] Portland Julia users?

[Alex Mellnik]

Hello all,

I know there's a few of us around, although probably not as many as in 
Seattle.  Is there any interest in a quarterly meetup or similar?  Best,

Alex

[julia-users] Re: testing intrument Automation

[Alex Mellnik]

Hi Yared,

This should be possible, but it could be less than ideal in some instances. 
 What exactly are you hoping to automate?  

Like Isaiah, I don't know of a specific GPIB library for Julia, but if you 
are using the NI drivers you can call them directly from Julia 
(see 
http://docs.julialang.org/en/release-0.4/manual/calling-c-and-fortran-code/ 
and there's a bit of specific information about the GPIB libraries 
here: 
http://alex.mellnik.net/application-notes/application-notesusing-nis-gpib-drivers-in-qt/).
 
 

-A

On Saturday, July 2, 2016 at 5:39:07 PM UTC-7, Yared Melese wrote:
>
> Can Julia be used as instrument test Automation? Send commands via GPIB to 
> different instruments
>
> If so is there Visa library for it? 
>
> Thanks 
> Yared
>

Re: [julia-users] DateTime conversion in DataFrames

[Alex Mellnik]

To expand slightly on what Milan said, while you can pass DateTime an array 
of strings and a format code, in some cases the conversion function you 
want to use may not accept arrays.  In this case, you can do the same thing 
with array comprehension:

df1 = DataFrame(V1 = DateTime[DateTime(d, "m/d/y H:M") for d in ["4/5/2002 
04:20", "4/5/2002 04:25"]])



On Thursday, May 26, 2016 at 6:55:42 AM UTC-7, Milan Bouchet-Valat wrote:
>
> Le jeudi 26 mai 2016 à 06:15 -0700, akrun a écrit : 
> > I am using the DataFrames package.  I find it difficult to convert to 
> > DateTime.  
> > 
> >   println(DateTime("4/5/2002 04:20", "m/d/y H:M"))  
> > gives output 
> > 
> >   2002-04-05T04:20:00 
> > 
> > However, if I try 
> > 
> >   df1 = DataFrame(V1 = ["4/5/2002 04:20", "4/5/2002 04:25"]) 
> >   println(DateTime(df1[:V1]))  
> > gives 
> > 
> >  ArgumentError: Delimiter mismatch. Couldn't find first 
> > delimiter, "-", in date string 
> >  in parse at dates/io.jl:152 
> >   
> > 
> > Is there any workaround? 
> This isn't specific to data frames. You also get this with 
> V1 = ["4/5/2002 04:20", "4/5/2002 04:25"] 
> DateTime(V1) 
>
> Anyway, you need to pass the format as in your first example: 
> DateTime(V1, "m/d/y H:M")DateTime(df1[:V1], "m/d/y H:M") 
>
>
> Regards 
>

[julia-users] Re: Questions regarding to SQLite

[Alex Mellnik]

Yes to both!  

For the first one, you want to use Data.stream! to get a Data.Table from 
the result set, and then convert it to a DataFrame.  For the second you 
need to do the opposite.  I generally work with DataFrames so I wrote two 
convenience functions for doing this, which should give you an idea of how 
to go about it:

function queryToDF(db, query)
source = SQLite.Source(db, query)
return DataFrame(Data.stream!(source, Data.Table))
end

function dfToDB(db, df, table)
sink = SQLite.Sink(Data.Table(df), db, table)
Data.stream!(Data.Table(df), sink)
end

Cheers -A

On Wednesday, May 25, 2016 at 2:54:26 AM UTC-7, SHORE SHEN wrote:
>
> Hello
>
> Im trying out the sqlite package in julia, I got the following 2 questions:
>
> 1, the query will result in a type of SQLite.ResultSet, if i can output 
> dataframe or dataarry type?
>
> 2, if i have a dataframe or dataarray type, would i be able to put it into 
> the database table?
>
> thanks a lot!
>

[julia-users] Re: Error installing hydrogen for Atom

[Alex Mellnik]

I had this exact same problem.  What fixed it for me was using the nuclear 
option to force everything in the build to use python 2.7:  I temporarily 
removed 3.5 from my path and replaced it with 2.7.  

I had previously tried setting the PYTHON environment variable to the 2.7 
executable and telling npm to use 2.7, but had no luck.  

I've since restored 3.5 to my path, and can launch and use Hydrogen 
successfully.  It's pretty magical.  

-A

On Friday, April 8, 2016 at 8:38:28 AM UTC-7, Achu wrote:
>
> That's unfortunate. I was excited about hydrogen.
>
> On Friday, 8 April 2016 00:14:02 UTC-5, Jeffrey Sarnoff wrote:
>>
>> I have experienced this too.  Hydrogen has never been a load-and-go 
>> proposition for Julia on Windows.  Before this, there were other hang-ups 
>> and some have had their roots in something an earlier release of Atom on 
>> Win was doing or not doing.  I have downloaded a version of Juno/LT that 
>> works on Win7.  It is less of an advanced editor today, but it coordinates 
>> easily with Julia for type/eval/edit/eval/save.
>>
>> On Thursday, April 7, 2016 at 9:58:20 PM UTC-4, Achu wrote:
>>>
>>> Was trying to install hydrogen for Atom when I got this error. Tried it 
>>> with every available version of ZMQ and no luck. Anyone know why?
>>>
>>> Installing hydrogen to C:\Users\achug\.atom\packages failed 
>>>
>>> > zmq@2.14.0 install 
>>> C:\Users\achug\AppData\Local\Temp\apm-install-dir-11637-16296-1xp74zg\node_modules\Hydrogen\node_modules\jmp\node_modules\zmq
>>> > node-gyp rebuild
>>>
>>>
>>> C:\Users\achug\AppData\Local\Temp\apm-install-dir-11637-16296-1xp74zg\node_modules\Hydrogen\node_modules\jmp\node_modules\zmq>if
>>>  
>>> not defined npm_config_node_gyp (node 
>>> "C:\Users\achug\AppData\Local\atom\app-1.6.2\resources\app\apm\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js"
>>>  
>>> rebuild )  else (node  rebuild )
>>>
>>> gypnpm ERR! Windows_NT 6.2.9200
>>> npm ERR! argv 
>>> "C:\\Users\\achug\\AppData\\Local\\atom\\app-1.6.2\\resources\\app\\apm\\bin\\node.exe"
>>>  
>>> "C:\\Users\\achug\\AppData\\Local\\atom\\app-1.6.2\\resources\\app\\apm\\node_modules\\npm\\bin\\npm-cli.js"
>>>  
>>> "--globalconfig" "C:\\Users\\achug\\.atom\\.apm\\.apmrc" "--userconfig" 
>>> "C:\\Users\\achug\\.atom\\.apmrc" "install" 
>>> "C:\\Users\\achug\\AppData\\Local\\Temp\\d-11637-16296-10fuf5t\\package.tgz"
>>>  
>>> "--target=0.34.5" "--arch=ia32"
>>> npm ERR! node v0.10.40
>>> npm ERR! npm  v2.13.3
>>> npm ERR! code ELIFECYCLE
>>>
>>> npm ERR! zmq@2.14.0 install: `node-gyp rebuild`
>>> npm ERR! Exit status 1
>>> npm ERR!
>>> npm ERR! Failed at the zmq@2.14.0 install script 'node-gyp rebuild'.
>>> npm ERR! This is most likely a problem with the zmq package,
>>> npm ERR! not with npm itself.
>>> npm ERR! Tell the author that this fails on your system:
>>> npm ERR! node-gyp rebuild
>>> npm ERR! You can get their info via:
>>> npm ERR! npm owner ls zmq
>>> npm ERR! There is likely additional logging output above.
>>>
>>

Re: [julia-users] Julia SQL

[Alex Mellnik]

To expand on what Jacob said, you can read a DataFrame into an in-memory 
SQLite table and then run SQL commands on that.  However, unless you really 
need to use SQL there's probably a way to do the same operation faster (and 
with less code) using native DataFrame operations.  If you can provide some 
examples we're happy to help translate them.  -A

On Sunday, May 8, 2016 at 9:01:07 AM UTC-7, Jacob Quinn wrote:
>
> Also checkout the SQLite.jl package. It provides methods for reading CSV 
> files into an SQLite table and then running SQLite SQL commands on those 
> tables. You can then export the SQLite to a CSV or Data.Table/DataFrame.
>
> -Jacob
> On May 8, 2016 4:32 AM, "Tero Frondelius"  > wrote:
>
>> Maybe this thread is relevant:
>> https://groups.google.com/forum/m/#!topic/julia-users/QjxiCO-Lv-0
>
>

[julia-users] Syntax highlighting for julia symbols in Notepad++?

[Alex Mellnik]

There's a Notepad++ configuration file provided in the distribution here 
. 
 One limitation is that it doesn't highlight symbols (like *:something*) 
and always treats : as an operator.  IJulia on the other hand is smart 
enough to highlight them with a custom style.  Is there a way to do this in 
Notepad++?  I haven't been able to get it to work without catching at least 
one trailing character.  -A

[julia-users] Re: Using @pyimport inside a module

[Alex Mellnik]

Hi Cedric,

Thanks for looking into this.  I just tried it again today and it's working 
-- I don't know what changed -- I think the only thing I have differently 
is that I restarted my jupyter process.  Not sure, but thanks again,

Alex  

On Wednesday, April 6, 2016 at 3:38:16 PM UTC-7, Cedric St-Jean wrote:
>
> Hi Alex, it looks like you're doing everything right. Just to make sure: 
> does  "TheModuleInJulia" have a final `end` statement? I just tried this 
> code in a notebook:
>
> module TheModuleInJulia
> using PyCall
> @pyimport numpy
> export geteverything 
>
> function geteverything()
> return collect(numpy.eye(10))
> end
> end
>
> TheModuleInJulia.geteverything()
>
> It worked for me. Does it work for you?
>
> On Wednesday, April 6, 2016 at 12:18:09 PM UTC-4, Alex Mellnik wrote:
>>
>> I rely on a small Python module which I am currently calling directly 
>> with @pyimport in a notebook:
>>
>> using PyCall
>> @pyimport datetime
>> @pyimport TheModule
>>
>> cursor = TheModule.connect() 
>>
>>
>> This works fine, but now I would like to wrap the Python module with some 
>> other functions in a julia package.  I tried 
>>
>> module TheModuleInJulia
>>
>> using PyCall
>> @pyimport TheModule
>>
>> export geteverything 
>>
>>
>> function geteverything()
>>
>> return collect(TheModule.connect())
>>
>> end
>>
>>
>> and several varaitions thereof.  In my main file I can
>>
>> using PyCall, TheModule
>>
>>
>> which works fine, but when I  
>>
>> geteverything()
>>
>>
>> I get an error saying that TheModule is not defined.  What's the correct 
>> way to import and use the python module within the julia module? I didn't 
>> see anything about this in the docs for PyCall. Thanks -A
>>
>

[julia-users] Using @pyimport inside a module

[Alex Mellnik]

I rely on a small Python module which I am currently calling directly with 
@pyimport in a notebook:

using PyCall
@pyimport datetime
@pyimport TheModule

cursor = TheModule.connect() 


This works fine, but now I would like to wrap the Python module with some 
other functions in a julia package.  I tried 

module TheModuleInJulia

using PyCall
@pyimport TheModule

export geteverything 


function geteverything()

return collect(TheModule.connect())

end


and several varaitions thereof.  In my main file I can

using PyCall, TheModule


which works fine, but when I  

geteverything()


I get an error saying that TheModule is not defined.  What's the correct 
way to import and use the python module within the julia module? I didn't 
see anything about this in the docs for PyCall. Thanks -A

[julia-users] Re: ccall, dlopen when dll contains a class

[Alex Mellnik]

Sorry to reopen an old thread.  I'm trying to do something similar and 
haven't been able to figure it out so far.  Isaiah, I noticed that your 
COMCall.jl library is not on GitHub anymore.  Is this code still floating 
around anywhere or is there a different suggested route?

In my case, I have a dll (written in c# I think) which is currently being 
called from Python using win32com 
.
 
 I would like to call the dll directly from Julia without going through 
PyCall (although this works).  I don't have the source for that dll but a 
simple example is:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace SimpleLibrary
{
public class MyClass
{
public static double MySum(double a, double b)
{
return a + b;
}

public static double NoArgs()
{
return 1.4;
}

}
}

Naively trying to call these with ccal like 

ccall( ("SimpleLibrary.MyClass.MySum", "SimplyLibrary"), Float32, (1.2, 
3.1))

gives a "wrong number of arguments" error.  Is there any easy (or failing 
that, hard) way to do this?

[julia-users] Re: Remove Gadfly gridlines?

[Alex Mellnik]

You are probably overwriting the theme later and the last one overwrites 
that setting.  The following example works for me:




-A


On Monday, February 1, 2016 at 5:58:13 AM UTC-8, Jon Norberg wrote:
>
> I have searched and tried a few things but cannot remove the background 
> grids in Gadfly. Its probably simple and I am missing something 
> obvious...Any suggestions would be appreciated.
>
> layer(x=E,y=wetness(E,10.0), Geom.line,Theme(default_color=a[1], 
> line_width=2pt, grid_color=colorant"white")
>
> also tried grid_line_width=0pt
>
> but they still show up ( I save it as SVG, but also output in jupiter 
> shows them)
>
>

[julia-users] Every module recompiles every time?

[Alex Mellnik]

At some point since the release of 0.4.3 every module that I use started 
always recompiling whenever I first use it in a new kernel.  If I'm using 
something with lots of dependencies like Gadfly this can lead to 
several-minute startup times.  What determines when modules are recompiled 
and how can I avoid this?  

Two possible things which may be related -- I had a computer crash while 
modules were recompiling around the same time, and I have several packages 
which are either pinned to their master or on my own development branch. 
 Thanks  -A

[julia-users] Re: Running a julia script on the web

[Alex Mellnik]

A fairly radical approach would be to use Escher.jl (http://escher-jl.org/) 
as your web server and write all the code in julia.  Another method would 
be to write a simple webserver with Mux.jl 
(https://github.com/JuliaWeb/Mux.jl).  If you can use it, I think the best 
method would be to have a more conventional web site that can interact with 
Julia through Node.js (https://www.npmjs.com/package/node-julia).  

(In the unlikely case that you're my high school classmate of the same 
name, hi!)

[julia-users] Re: Proposal: NoveltyColors.jl

[Alex Mellnik]

On a related note, I've been thinking that it would be nice to include the 
results of the xkcd color survey  in 
Colors.jl.  Right now it has the CSS/SVG and X11 colors which is great for 
standardization, but sometimes you want to be able to get a RGB value 
corresponding to fairly specific and easy-to-remember color names (mocha, 
cerulean blue, etc).  I was originally going to stick it in a different 
package, but there might be a nice way to separate these names in Colors.jl

-A

On Tuesday, November 24, 2015 at 2:08:35 PM UTC-8, Randy Zwitch wrote:
>
> Since the Julia ecosystem is getting bigger, I figured I'd propose this 
> here first and see what people think is the right way forward (instead of 
> wasting people's time at METADATA)
>
> In the R community, they've created two packages of novelty color schemes: 
> Wes 
> Anderson  and Beyonce 
> . While humorous, these color palettes 
> are interesting to me and I'd like to make them available in Vega.jl (and 
> Julia more broadly). Should I:
>
> 1) Not do it at allbecause this is a serious, scientific community!
> 2) Do two separate packages, mimicking R
> 3) Create a single NoveltyColors.jl package, in case there are other 
> palettes that come up in the future
> 4) Make a feature request at Colors.jl (really not my favorite choice, 
> since there is so much cited research behind the palettes)
>
> I neglected to mention ColorBrewer.jl (which Vega.jl uses), since 
> ColorBrewer is a known entity in the plotting community.
>
> What do people think? Note, I'm not looking for anyone to do the work 
> (I'll do it), just looking for packaging input.
>

[julia-users] Re: Masters/PhD programs to continue work on Julia

[Alex Mellnik]

Agreed.  I would also note that Julia might be used in a wide range of 
graduate fields, from linguistics to economics to computational physics to 
math, so if you're only interested in working in Julia you needn't limit 
yourself to Comp Sci/Eng unless that's the field you wanted to go into 
anyways.  

On Friday, November 13, 2015 at 9:17:53 AM UTC-8, Steven G. Johnson wrote:
>
> On Thursday, November 12, 2015 at 11:43:38 AM UTC-5, Rohit Thankachan 
> wrote:
>>
>> I am planning on applying to masters and/or PhD programs that start in 
>> Fall 2016. I am a JSoC 2015 student and have been doing work related to 3D 
>> visualizations from Julia - ThreeJS.jl 
>>  and Compose3D.jl 
>> . I am hoping to continue 
>> working on these projects along with newer projects in Julia as part of a 
>> master's or PhD program. Does anyone know of any programs that would allow 
>> me to do this? Pointers would be appreciated too. :)
>>
>
> As a general rule, you should apply to PhD engineering programs with the 
> expectation that you can choose your general area of research (e.g. 
> visualization), but your specific projects will be chosen by your advisor 
> in the beginning.   You can generally spend a certain fraction of your time 
> on personal projects, and as you become a more senior graduate student you 
> often have more input in steering your research direction, but your advisor 
> will typically hire you with projects already in mind for you to work on.
>
> (Also, as a general tip for admissions: by all means, express an interest 
> in specific areas of research, but in your application statement you should 
> appear flexible in what specific problems you want to work on.   If you 
> appear too determined to work on a specific problem, no one will want to 
> admit you except in the unlikely event that your problem exactly coincides 
> with a project already in their research program/grants.)
>

[julia-users] Re: Custom parsing of csv files in DataFrames

[Alex Mellnik]

I think it may be easiest to do this conversion in your own program after 
using readtable:

using DataFrames
#Make some canned data instead of reading from a table
df = DataFrame(datestring = ["12 Apr 1996", "05 Aug 2015"], moneystring = 
["\$12.75", "\$0.69"])
df[:date] = Date()
df[:money] = 0.0
dateformat = Dates.DateFormat("dd uuu ")
#map! and not map to preserve types
map!(d->DateTime(d,dateformat), df[:date], df[:datestring])
map!(x->float(lstrip(x,'$')), df[:money], df[:moneystring])
df