[julia-users] Re: Cxx.jl: Creating a templated class

[Jan Strube]

Is this in any way related to https://github.com/Keno/Cxx.jl/issues/212 ?
Any pointer what this means?
ERROR: LoadError: Creating LLVM constants for type `T` not implemented yet


On Monday, July 4, 2016 at 10:09:34 PM UTC-7, Jan Strube wrote:
>
> What's the right way to create a templated class on the C++ side with Cxx?
>
> I have an LCCollection 
> ,
>  
> and I would like to create a CellIDDecoder 
> ,
>  
> which needs a template argument.
>
> A simple 
>cxx"""UTIL::CellIDDecoder($(collection.coll)) x;
>   std::cout << x << std::endl;
> results in 
>
> ERROR: LoadError: Creating LLVM constants for type `T` not implemented yet
>  in llvmconst(::Any) at /home/stru821/.julia/v0.5/Cxx/src/cxxstr.jl:22
>  in ArgCleanup(::Cxx.ClangCompiler, ::Cxx.CppPtr{Cxx.CxxQualType{Cxx.
> CppBaseType{Symbol("EVENT::LCCollection")},(false,false,false)},(false,
> false,false)}, ::Cxx.CppPtr{Cxx.CxxQualType{Cxx.CppBaseType{Symbol(
> "clang::VarDecl")},(false,false,false)},(false,false,false)}) at /home/
> stru821/.julia/v0.5/Cxx/src/cxxstr.jl:192
>  in macro expansion at /home/stru821/.julia/v0.5/Cxx/src/cxxstr.jl:634 [
> inlined]
>  in getCellIDDecoder(::LCIO.LCCollection{Cxx.CppPtr{Cxx.CxxQualType{Cxx.
> CppBaseType{Symbol("EVENT::SimCalorimeterHit")},(false,false,false)},(
> false,false,false)}}) at /home/stru821/.julia/v0.5/LCIO/src/CaloHit.jl:29
>  in testReader(::String) at /home/stru821/.julia/v0.5/LCIO/examples/
> testLCIOCxx.jl:14
>  in include_from_node1(::String) at ./loading.jl:426
>  in process_options(::Base.JLOptions) at ./client.jl:266
>  in _start() at ./client.jl:322
> while loading /home/stru821/.julia/v0.5/LCIO/examples/testLCIOCxx.jl, in 
> expression starting on line 22
>
> I have also tried to return the result of 
> icxx"""UTIL::CellIDDecoder($(collection.coll));
>
> But that replies with an even stranger message:
> :2:45: error: no matching constructor for initialization of 
> 'UTIL::CellIDDecoder'
> UTIL::CellIDDecoder < SimCalorimeterHit > ( __juliavar1 ) ;
> ^
> /scratch/software/lcio-test/include/UTIL/CellIDDecoder.h:36:3: note: 
> candidate constructor not viable: requires single argument 'encoder_str', 
> but no arguments were provided
>   CellIDDecoder( const std::string& encoder_str ) : _oldHit(0) {
>   ^
> /scratch/software/lcio-test/include/UTIL/CellIDDecoder.h:47:5: note: 
> candidate constructor not viable: requires single argument 'col', but no 
> arguments were provided
> CellIDDecoder( const EVENT::LCCollection* col ) : _oldHit(0) {
> ^
>
> The code clearly has a constructor argument, so I don't understand why the 
> compiler things it doesn't...
>
> If somebody knows how to construct such a templated class, please share.
> Many thanks.
>

Re: [julia-users] list kw args

[Leonardo]

Thanks, I've found function:
kwarg_decl(sig::ANY, kwtype::DataType)
in develop version (not stable version 0.4.6)

But I cannot understand (also following code flow) what is kwtype 
parameter and how kwarg_decl can be called.


And finally, can this code run on current stable version?

Many thanks again

Leonardo



Il 06/07/2016 15:32, Isaiah Norton ha scritto:

The implementation is not trivial to follow, but have a look at
`kwarg_decl` in `base/methodshow.jl`.

This might be a reasonable thing to have as a helper function, if you
want to open a GitHub issue to discuss.

(Be warned: you will pay a severe performance penalty if you try to use
reflection for some kind of runtime meta-dispatch)

On Wed, Jul 6, 2016 at 1:41 AM, Leonardo > wrote:

There is a way to list all keyword arguments defined by a function f?
(a sort of reflection?)

e.g.;

/having
/function f(i::Int; a::AbstractString=0, b::Float64=0.0)
  ...
end

/how to return a, b from f?
/

Many thanks in advance

Leonardo

[julia-users] Pkg "threadsafe" or interprocess mutex package

[David Anthoff]

Hi,

 

I have a script that is using some packages. The script is set up such that
if the package it is trying to use is not installed, it will do a
``Pkg.add`` to add the package before ``using`` it.

 

This script is triggered by a third process, and it might be triggered
multiple times simultaneously, i.e. multiple instances of julia might run
the same script at the same time. Therefore the second julia instance might
issue the same ``Pkg.add`` command as the first instance, while the first
instance is not done with the ``Pkg.add`` yet.

 

I assume the package manager per se is not set up to handle that situation
gracefully?

 

If not, is there some package that allows me to have something like a
mutex/lock between my julia processes, so that I can guard the code section
that fiddles with packages and make sure only one julia instance at a time
runs that code?

 

Thanks,

David

 

--

David Anthoff

University of California, Berkeley

 

http://www.david-anthoff.com

 

Re: [julia-users] R code to Julia Performance issues with pmap

[Michael Krabbe Borregaard]

Ah, forgot that the system.time is a bit tricky when running parallel cores

[julia-users] Re: Julia community stewards

[Jameson]

I have +o privileges on IRC. I'm (almost) never on, but I can give ops if 
you PM me and have an account registered with NickServ.


On Tuesday, July 5, 2016 at 9:48:07 AM UTC-4, Chris Rackauckas wrote:
>
> I think in general we need some active mods for IRC/Gitter. Some people 
> with the ability to edit something to get rid of accidental mentions (sorry 
> @parallel) and ban spammers. It's not too bad, and most issues are 
> dealt with people being reasonable, but I can see future moderation needed. 
> Who would we talk to about this?
>
> On Friday, July 1, 2016 at 9:57:20 AM UTC-7, Steven G. Johnson wrote:
>>
>>
>>
>> On Thursday, June 30, 2016 at 12:20:10 AM UTC-4, Lyndon White wrote:
>>>
>>> As such, can we new get a formal response from one of these Stewards to 
>>> Bridging the IRC and Gitter  (
>>> https://groups.google.com/forum/#!topic/julia-users/ImKYzqHXA90)
>>>
>>
>> To be clear, the role of the Stewards is to respond to 
>> Community-Standards violations (in the hopefully rare cases where informal 
>> feedback from other community members is insufficient).   Technical 
>> maintenance of community forums is not their responsibility.
>>
>

[julia-users] Re: How to fix writemime deprecation?

[Jameson]

Correct. This still needs a doc update (I added #16763 to the v0.5.x 
milestone). The two-argument form `show(io, x)` should produce a concise, 
single line description that formats reasonably into any unknown context.

The three-argument mime-format aware version `show(io, mime, x)` can be 
additionally overloaded to produce a complete, reactive, expanded 
representation.


On Saturday, July 2, 2016 at 8:38:51 AM UTC-4, Fengyang Wang wrote:
>
> It is no longer true that
>
> show(io, x)
>
> should be overriden for MIME text/plain. Instead, the MIME version is for 
> multiline display, and the two-argument form for single-line display. See 
> https://github.com/JuliaLang/julia/pull/17113.
>
>
> On Saturday, July 2, 2016 at 8:06:28 AM UTC-4, Steven G. Johnson wrote:
>>
>> On Saturday, July 2, 2016 at 5:11:16 AM UTC-4, Sheehan Olver wrote:
>>>
>>> What has replaced writemime?  I'm trying to fix the 0.5 deprecation 
>>> warnings.
>>>
>>
>> writemime(io, mime, x) has been renamed to show(io, mime, x), and show is 
>> now the correct method to override to change how an object is displayed 
>> with various mime types.  If mime == text/plain, you can just do show(io, 
>> x) , and you should override the 2-argument show to change an object's text 
>> output.
>>
>> For 0.4 compatibility, you can use Compat and o
>>
>> @compat Base.show(io::IO, mime::MIME"", x::MyType) = 
>>
>> and it will rewrite to a writemime definition on older versions.
>>
>

Re: [julia-users] R code to Julia Performance issues with pmap

[jascisen]

This is not entirely true in R:

Details:

 ‘proc.time’ returns five elements for backwards compatibility, but
 its ‘print’ method prints a named vector of length 3.  The first
 two entries are the total user and system CPU times of the current
 R process and any child processes on which it has waited, and the
 third entry is the ‘real’ elapsed time since the process was
 started.


On Wednesday, July 6, 2016 at 5:11:04 PM UTC+2, Michael Borregaard wrote:
>
> I am not seeing your speed-up in R? elapsed is less time, but user 
> significantly more, and it is the sum that counts.
> When executing in parallel the language needs to copy the data to the 
> workers. If the matrices are large, that takes longer than the speedup of 
> the parallel execution. See what happens with a smaller matrix and then 
> repeating the operation on the workers.
>
>

Re: [julia-users] R code to Julia Performance issues with pmap

[Michael Borregaard]

I am not seeing your speed-up in R? elapsed is less time, but user 
significantly more, and it is the sum that counts.
When executing in parallel the language needs to copy the data to the 
workers. If the matrices are large, that takes longer than the speedup of 
the parallel execution. See what happens with a smaller matrix and then 
repeating the operation on the workers.

Re: [julia-users] Re: MYSQL/MARIADB Big INSERT

[Páll Haraldsson]

On þri  5.júl 2016 23:49, Andrey Stepnov wrote: 

Hi Palli! 

Thanks for reply. 

Here is code with transactions example: 
|| 
|mysql_query(con, "SET autocommit = 0") 

Generally you do not have to change this, rely on the default setting of 
the database(?).

mysql_query(con, "START TRANSACTION;") 

Yes, non-standard in MySQL.. One of the reasons to use some 
beginTransaction(). 

mysql_query(con, "SET autocommit = 1") 

[And you would then not change again. When your connection closes, setting 
are usually lost. I'm sure there are other ways to change globally for all 
users.] 


 > Then do your INSERTs (prepared statments can be better here for 
security but not speed, while can also help for speed in queries), e.g 
in a loop. 
Yes, you right! It works with small data but if I have: 

And "works" with big data.. 

while 

it's extremely slow. 

at least in MySQL (yes, also 4x faster in one statement in PostgreSQL, I 
see from the link I provide below. I've only used COPY, never equivalent(?) 
multi-INSERT that is actually there I see, and COPY only outside of the 
application). 

Solution: string with non-safe big-insert: 


[It can also be made safe, look into escape functions. They work, just more 
brittle to work with, as you have to take care for each instruction. That 
is why prepared statements are better (and some abstraction layers 
auto-prepare for you) and can be also faster in other cases.] 


That confirms this isn't a Julia issue. I'm not an expert on MySQL. I'll go 
into little more detail, but this belongs in some MySQL (or generic db) 
forum. 

I'm glad you're using Julia, less glad that you're not using PostgreSQL. 
That might also have helped, as it is a superior database (MySQL had a few 
things going for it against PostgreSQL, none I think any more, it had 
clustering but not the other, maybe theirs (or really from a third party) 
is better for special cases of clustering (or not)). 


All databases will be slow if they need to commit each INSERT (or say 
UPDATE) to stable storage (disk, less slowdown for SSDs..). You could 
monitor if that is happening (with strace, in Linux/Unix), maybe by slowing 
down your loop. 

All databases will be faster if you really manage to commit more rows at a 
time, in a transaction (each instruction at least is one), but they 
probably also have non-standard ways. 

PostgreSQL has COPY (similar to MySQL's multi-INSERT, neither are 
standardized, so it doesn't need their way, except for compatibility for 
MySQL code 
https://kaiv.wordpress.com/2007/07/19/faster-insert-for-multiple-rows/ ) as 
it's faster than the non-default that pg_dump exports (for compatibility 
with other databases; you can fairly easily migrate to and from it). 
[Informix db has non-standard dbexport and dbimport commands, then also 
HPL, high-performance loader, that I never used..] 


or even 

https://dev.mysql.com/doc/refman/5.5/en/load-data.html 

"LOAD DATA [LOW_PRIORITY | CONCURRENT]" 

might be more similar to COPY. 

https://dev.mysql.com/doc/refman/5.5/en/insert-delayed.html 

"INSERT DELAYED ... 

The DELAYED option for the INSERT statement is a MySQL extension to 
standard SQL that is very useful if you have clients that cannot or need 
not wait for the INSERT to complete." 

Might also be what you're after. 


Possibly, you need to set isolations level(?), if other users are seeing 
your INSERTs trickle in, it's a clue that they are also going to disk. They 
should see them all or none, if transactions are working. 


MySQL, was really bad, while you could declare FOREIGN KEYs (and probably 
open transactions), they where just ignored! It might still be happening. 
Not sure what the default is. Transactions and foreign keys etc. are only 
defined in the sane engines, e.g. InnoDB. I didn't follow what came later, 
those probably are sane also. 

-- 
Palli.

Re: [julia-users] More on Macros

[Isaiah Norton]

No worries; if you are missing S-expressions too much, you might enjoy
playing with https://github.com/swadey/LispSyntax.jl

On Wed, Jul 6, 2016 at 12:41 AM, Rangarajan Krishnamoorthy <
ranga@gmail.com> wrote:

> That was very helpful. Thanks for your patience.
>
> -Rangarajan
>
>
> On Wednesday, July 6, 2016 at 9:39:56 AM UTC+5:30, Isaiah wrote:
>>
>> Is there a clear language spec for macros, or is it still evolving? I
>>> understand that relying on implementation to learn a language feature can
>>> be risky.
>>>
>>
>> There is no formal spec that I am aware of. However, I don't remember any
>> substantive changes except for this one in 0.5-dev:
>>
>> https://github.com/JuliaLang/julia/pull/14563
>>
>>
>> (which is fairly minor: macro-methods can now dispatch on number of
>> arguments IIUC)
>>
>> There's one major proposal related to hygiene; no final decision on
>> whether/when it will be accepted:
>>
>> https://github.com/JuliaLang/julia/pull/6910 (rebased in #10940)
>>
>>
>> When I call retMacro(), I get an error saying "Macro definition not
>>> allowed in local scope". What I tried to do in my earlier example (pre
>>> post) is to hide the macro definition in a string and then parse and
>>> evaluate it to get the same effect as the above function. Obviously the
>>> behaviour is different.
>>>
>>
>> Your earlier example works because `eval` always operates at
>> module-global scope. Macro definitions, type declarations, and a handful of
>> other things, are only allowed at global scope.
>>
>>
>>> Does it mean every macro is its own type?
>>>
>>
>> Yes (and further, on 0.5-dev all functions are internally represented as
>> types)
>>
>>
>>> Should it not be a predefined type like "macro" (just as there is type
>>> for functions)? If macros are "pre-processed away", then there can't be an
>>> independent existence for them?
>>>
>>
>> Here's some more detail that will hopefully clarify. Macro invocations
>> are represented as :macrocall expressions:
>>
>> julia> parse("@foo(1, 2)") |> dump
>> Expr
>>   head: Symbol macrocall
>>   args: Array{Any}((3,))
>> 1: Symbol @foo
>> 2: Int64 1
>> 3: Int64 2
>>   typ: Any
>>
>>
>> The first step of lowering is to walk the AST to find all :macrocall
>> expressions, call the corresponding macro-function -- exactly the function
>> returned by `eval(Symbol("@foo"))` -- with the appropriate arguments, and
>> paste the result expression into the AST. (see
>> http://docs.julialang.org/en/latest/devdocs/julia/ for some more details)
>>
>> Finally, in 0.5-dev, if "returnMacro()" works as expected, then calling
>>> myMacro("1+1") should return 2 (and not the parsed structure), isn't it?
>>> The reason is does an additional evaluation?
>>>
>>
>> No, `myMacro` is just a handle to the macro-function, exactly the same as
>> `myMacro(x) = parse(x)`. There is no evaluation of the return expression
>> when directly calling a macro-function.
>>
>>
>> As I had mentioned in my first post, these questions are based on my
>>> experience with Lisp and I am trying to digest the differences, if any.
>>>
>>> -Rangarajan
>>>
>>> On Tuesday, July 5, 2016 at 9:35:52 PM UTC+5:30, Isaiah wrote:
>>>
 Anonymous functions are a bit incomplete / not-fully-integrated in 0.4,
 and I think this falls out as a consequence.

 You can simulate the result you want with:

 julia> function returnMacro()
  eval(parse("""
  macro myMacro(anexpr)
parse(anexpr)
  end
  """))
  eval(symbol("@myMacro"))
end

 julia> mymacro = returnMacro()
 (anonymous function)

 julia> mymacro("1+1")
 :(1 + 1)

 However, there have been significant improvements to anonymous
 functions in the nightly (0.5-dev), and the result is what you expect:

 julia> function returnMacro()
  eval(parse("""
  macro myMacro(anexpr)
parse(anexpr)
  end
  """))
end
 returnMacro (generic function with 1 method)

 julia> mymacro = returnMacro()
 @myMacro (macro with 1 method)

 julia> typeof(mymacro)
 #@myMacro

 julia> mymacro("1+1")
 :(1 + 1)

 That said, `mymacro` here is purely a function handle -- totally
 identical to writing `mymacro(expr) = parse(expr)`. There is no special
 behavior.

 (are you trying to do something specific?)

 On Tue, Jul 5, 2016 at 11:04 AM, Rangarajan Krishnamoorthy <
 rang...@gmail.com> wrote:

> Consider this function:
>
> function returnMacro()
>   eval(parse("""
>   macro myMacro(anexpr)
> parse(anexpr)
>   end
>   """))
> end
>
> I assign the function's result to a variable:
> mymacro = returnMacro()
>
> Then, typeof(mymacro) returns "Void". Is this correct? Because macros
> are parse time entities, wouldn't it be better for 

Re: [julia-users] list kw args

[Isaiah Norton]

The implementation is not trivial to follow, but have a look at
`kwarg_decl` in `base/methodshow.jl`.

This might be a reasonable thing to have as a helper function, if you want
to open a GitHub issue to discuss.

(Be warned: you will pay a severe performance penalty if you try to use
reflection for some kind of runtime meta-dispatch)

On Wed, Jul 6, 2016 at 1:41 AM, Leonardo  wrote:

> There is a way to list all keyword arguments defined by a function f?
> (a sort of reflection?)
>
> e.g.;
>
>
> *having*function f(i::Int; a::AbstractString=0, b::Float64=0.0)
>   ...
> end
>
>
> *how to return a, b from f?*
>
> Many thanks in advance
>
> Leonardo
>
>

Re: [julia-users] R code to Julia Performance issues with pmap

[jascisen]

Yes, but I am not using BLAS or FFT transforms so it si a bit surprising 
that I am not getting any speed improvements

On Wednesday, July 6, 2016 at 2:17:49 AM UTC+2, Stefan Karpinski wrote:
>
> Similar question and answer: 
> http://stackoverflow.com/questions/38075163/julia-uses-only-20-30-of-my-cpu-what-should-i-do/38075939
> .
>
> On Tue, Jul 5, 2016 at 11:26 AM,  wrote:
>
>> I am a complete newcomer to Julia and trying to port some of my R code to 
>> it;
>> Basically I have rewritten the following R code in Julia:
>>
>> library(parallel)
>> 
>> eps_1<-rnorm(100)
>> eps_2<-rnorm(100)
>> 
>> large_matrix<-ifelse(cbind(eps_1,eps_2)>0,1,0)
>> matrix_to_compare = expand.grid(c(0,1),c(0,1))
>> indices<-seq(1,100,4)
>> large_matrix<-lapply(indices,function(i)(large_matrix[i:(i+3),]))
>> 
>> function_compare<-function(x){
>>   which((rowSums(x==matrix_to_compare)==2) %in% TRUE)
>> }
>> 
>> > system.time(lapply(large_matrix,function_compare))
>>user  system elapsed 
>>  38.812   0.024  38.828 
>> > system.time(mclapply(large_matrix,function_compare,mc.cores=11))
>>user  system elapsed 
>>  63.128   1.648   6.108 
>>
>> As one can notice I am getting significant speed-up when going from one 
>> core to 11. Now I am trying to do the same in Julia:
>>
>> using Distributions;
>> @everywhere using Iterators;
>> d = Normal();
>> 
>> eps_1 = rand(d,100);
>> eps_2 = rand(d,100);
>> 
>> #Define cluster:
>> addprocs(11);
>> 
>> #Create a large matrix:
>> large_matrix = hcat(eps_1,eps_2).>=0;
>> indices = collect(1:4:100)
>> 
>> #Split large matrix:
>> large_matrix = [large_matrix[i:(i+3),:] for i in indices];
>> 
>> #Define the function to apply:
>> @everywhere function function_split(x)
>> matrix_to_compare = 
>> transpose(reinterpret(Int,collect(product([0,1],[0,1])),(2,4)));
>> matrix_to_compare = matrix_to_compare.>0;
>> find(sum(x.==matrix_to_compare,2).==2)
>> end
>> 
>> @time map(function_split,large_matrix )
>> @time pmap(function_split,large_matrix )
>>  5.167820 seconds (22.00 M allocations: 2.899 GB, 12.83% gc time)
>>  18.569198 seconds (40.34 M allocations: 2.082 GB, 5.71% gc time)
>>
>> I somehow do not understand why parallel map function does not work for 
>> me. Maybe somebody can point me to a correct solution. 
>>
>
>

Re: [julia-users] Re: Master list of constants

[Waldir Pimenta]

While we're at it: since all constants in that list are represented by 
their symbols, with the names serving as aliases, would it be a good idea 
to define Catalan's constant 
 using the G symbol? If 
that's too common a character which we wouldn't want to pollute, we could 
use 퐺 (mathematical italic capital g, U+1D43A). Current behavior would be 
unchanged since "catalan" would be added as an alias. Thoughts?

On Wednesday, July 6, 2016 at 12:42:57 AM UTC+1, Stefan Karpinski wrote:
>
> The master list is basically this:
>
>
> https://github.com/JuliaLang/julia/blob/7b3f5296c54d0362ab90c2299b4c9255aca171a8/base/irrationals.jl#L127-L131
>
> On Tue, Jul 5, 2016 at 9:42 AM, esproff  
> wrote:
>
>> Ok great!  Thanks Kenta and Lyndon for your quick responses!
>>
>>
>> On Tuesday, July 5, 2016 at 4:23:47 AM UTC-7, Lyndon White wrote:
>>>
>>> You can use the `names` function to search a module for all constants it 
>>> defines.
>>> By evaluating the names to get the values, then checking if they are 
>>> Irrational (formerly known as MathConst?)
>>> By doing this, I conclude that the  ASCII name for the Euler-Mascheroni 
>>> constant (γ) is eulergamma
>>>
>>>
>>>  
>>>
>>>
>>> julia> for name_sym in names(Base)
>>>   value = eval(name_sym)
>>>   if typeof(value)<: Irrational
>>> println(name_sym, "\t", value)
>>>   end
>>>end
>>>
>>>
>>> catalan catalan = 0.9159655941772...
>>> e   e = 2.7182818284590...
>>> eu  e = 2.7182818284590...
>>> eulergamma  γ = 0.5772156649015...
>>> golden  φ = 1.6180339887498...
>>> pi  π = 3.1415926535897...
>>> γ   γ = 0.5772156649015...
>>> π   π = 3.1415926535897...
>>> φ   φ = 1.6180339887498...
>>>
>>>
>>>
>>>
>

[julia-users] Advice for parallel computing

[Dupont]

Hi,

I have an array of composite type

A = Array{MyType}(N)

that have been initialized somewhere else.

Currently, I am doing this operation

for i=1:N
doit!(A[i])
end

I would like to perform this operation in parallel (threads if possible on 
a single computer) but SharedArray does not accept composite types.

Thank you for your suggestions,

R

[julia-users] JuliaCon 2016 Escher.jl examples

[Jānis Erdmanis]

Hello everybody,

I am impressed by a recent Escher.jl JuliaCon presentation - youtube 
.
 
I would like to try and tinker on these talk codes myself, is it possible 
now? 

[julia-users] JuliaCon 2016 Escher.jl examples

[Jānis Erdmanis]

Hello everybody,

I am impressed by a recent Escher.jl JuliaCon presentation - youtube 
.
 
I would like to try and tinker on these talk codes myself, is it possible 
now?