Re: [julia-users] Weird timing issue
Thanks Iain! I knew I'd seen something like *findnz* in the documentation somewhere but I couldn't find it again... -- Sean On Thursday, December 11, 2014 11:38:24 PM UTC-6, Iain Dunning wrote: > > Or even simpler: > > coords(A::SparseMatrixCSC) = collect(zip(findn(A)...)) > > On Fri, Dec 12, 2014 at 12:22 AM, Iain Dunning > wrote: >> >> Also, you can pretty much just do this with inbuilt functions: >> >> function coord(A::SparseMatrixCSC) >> rows, cols, vals = findnz(A) >> collect(zip(rows,cols)) >> end >> >> >> On Friday, December 12, 2014 12:14:22 AM UTC-5, Iain Dunning wrote: >>> >>> I'm imagine its something like the following pattern: >>> >>> Run 1: generate X garbage >>> Run 2: generate X garbage, for total 2X garbage, which is over >>> threshold, reduce back to 0 >>> Run 3: generate X garbage >>> Run 4: generate X garbage, for total 2X garbage, which is over >>> threshold, reduce back to 0 >>> and so on >>> >>> On Friday, December 12, 2014 12:09:19 AM UTC-5, Sean McBane wrote: >>>> >>>> Alright. I am curious now as to what causes this behavior; hopefully >>>> someone will offer an explanation. >>>> >>>> I'll be sure to from now on. >>>> >>>> -- Sean >>>> >>>> On Thursday, December 11, 2014 11:07:07 PM UTC-6, John Myles White >>>> wrote: >>>>> >>>>> This is just how the GC works. Someone who's done more work on the GC >>>>> can give you more context about why the GC runs for the length of time it >>>>> runs for at each specific moment that it starts going. >>>>> >>>>> As a favor to me, can you please make sure that you quote the entire >>>>> e-mail thread you're responding to? I find responding to e-mails without >>>>> context to be pretty jarring. >>>>> >>>>> -- John >>>>> >>>>> On Dec 12, 2014, at 12:04 AM, Sean McBane wrote: >>>>> >>>>> > Right, I know I'm allocating it and discarding memory. However, if >>>>> the GC cleans up at deterministic points in time, as you point out in >>>>> your >>>>> first reply, why is timing erratic? And why the regular pattern in >>>>> timing? >>>>> It's always faster one call, slower one call, faster one call, slower one >>>>> call... >>>>> >>>>> > > -- > *Iain Dunning* > PhD Candidate > <http://orc.scripts.mit.edu/people/student.php?name=idunning> / MIT > Operations Research Center <http://web.mit.edu/orc/www/> > http://iaindunning.com / http://juliaopt.org >
Re: [julia-users] Weird timing issue
Alright. I am curious now as to what causes this behavior; hopefully someone will offer an explanation. I'll be sure to from now on. -- Sean On Thursday, December 11, 2014 11:07:07 PM UTC-6, John Myles White wrote: > > This is just how the GC works. Someone who's done more work on the GC can > give you more context about why the GC runs for the length of time it runs > for at each specific moment that it starts going. > > As a favor to me, can you please make sure that you quote the entire > e-mail thread you're responding to? I find responding to e-mails without > context to be pretty jarring. > > -- John > > On Dec 12, 2014, at 12:04 AM, Sean McBane > > wrote: > > > Right, I know I'm allocating it and discarding memory. However, if the > GC cleans up at deterministic points in time, as you point out in your > first reply, why is timing erratic? And why the regular pattern in timing? > It's always faster one call, slower one call, faster one call, slower one > call... > >
Re: [julia-users] Weird timing issue
Right, I know I'm allocating it and discarding memory. However, if the GC cleans up at *deterministic* points in time, as you point out in your first reply, why is timing erratic? And why the regular pattern in timing? It's always faster one call, slower one call, faster one call, slower one call...
[julia-users] Re: Weird timing issue
Oh yeah, and one more note - I'm using the dev build (0.4.0) for access to the *rowvals* and *nzrange* functions.
Re: [julia-users] Weird timing issue
function getIJValues(A::SparseMatrixCSC) m,n = size(A) rowcoords = rowvals(A) coordinates = [] for j = 1:n append!(coordinates, [(rowcoords[i],j) for i in nzrange(A,j)]) end return sort(coordinates) end I've never formally studied any computer science or programming so I don't have a great grasp on what goes on underneath this, but it seems to me like the only thing the garbage collector should need to do would be free the memory taken up by the list comprehension inside the loop at the end of the loop. And perhaps this could be redone in a more efficient manner, but inlining it like that seemed most natural. But what's strange is that called twice in a row, with exactly the same input, it takes twice as long the second time as the first. Otherwise, I certainly wouldn't be surprised if this method is inherently inefficient, since I only picked up the language yesterday. -- Sean
[julia-users] Re: Weird timing issue
As a note, the amount of memory allocated in both cases is the same. The difference seems to be only caused by garbage collection. On Thursday, December 11, 2014 10:43:03 PM UTC-6, Sean McBane wrote: > > Hi all, > > So, I'm starting to define a couple of routines to be used in iterative > solvers, optimized for sparse matrices. I wrote a routine that returns me a > list of tuples containing the (i,j) coordinates for the non-zero values > from a sparse matrix and was testing it for timing, but observed a > performance issue I don't understand. > > Testing using a 100x100 sparse matrix from a sample finite > difference problem, containing about 500 non-zero values, @time shows a > time of ~20s the first time I load the module and execute the function, and > about 3-5% of that is gc. I'm sure I can speed that up later, but the > really odd thing is that the NEXT run, it takes ~40s and about 50% of that > is garbage collection. The next one after that is back to 20s, and it keeps > going back and forth every time. This seems really weird to me. > > Anyway, to confirm that I wasn't going crazy I wrote a loop and timed this > a hundred times and collected results, and it keeps following the same > pattern. See attached 'times.txt' with the numbers. Any ideas what could be > causing this behavior? > > Thanks, > > -- Sean >
[julia-users] Weird timing issue
Hi all, So, I'm starting to define a couple of routines to be used in iterative solvers, optimized for sparse matrices. I wrote a routine that returns me a list of tuples containing the (i,j) coordinates for the non-zero values from a sparse matrix and was testing it for timing, but observed a performance issue I don't understand. Testing using a 100x100 sparse matrix from a sample finite difference problem, containing about 500 non-zero values, @time shows a time of ~20s the first time I load the module and execute the function, and about 3-5% of that is gc. I'm sure I can speed that up later, but the really odd thing is that the NEXT run, it takes ~40s and about 50% of that is garbage collection. The next one after that is back to 20s, and it keeps going back and forth every time. This seems really weird to me. Anyway, to confirm that I wasn't going crazy I wrote a loop and timed this a hundred times and collected results, and it keeps following the same pattern. See attached 'times.txt' with the numbers. Any ideas what could be causing this behavior? Thanks, -- Sean
Re: [julia-users] Unexpected append! behavior
Ah, I see the error in my thinking now. Knowing what the ! signifies makes
it make a lot more sense.
Thanks guys for putting up with a newbie. This was probably one of those 1
in the morning questions that I should have waited to look at the next day
before asking for help; it seems obvious now.
-- Sean
On Thursday, December 11, 2014 3:26:12 AM UTC-6, Mike Innes wrote:
>
> Think of append!(X, Y) as equivalent to X = vcat(X, Y). You called append!
> twice, so X gets Y appended twice.
>
> julia> X = [1,2]; Y = [3,4];
>
> julia> X = vcat(X,Y)
> [1, 2, 3, 4]
>
> In your example you went ahead and did this again:
>
> julia> X = (X = vcat(X, Y))
> [1, 2, 3, 4, 3, 4]
>
> But if you reset X, Y via the first statement and *then* call X =
> append!(X, Y), it works as you would expect.
>
> julia> X = [1,2]; Y = [3,4];
>
> julia> X = append!(X, Y) # same as X = (X = vcat(X, Y))
> [1, 2, 3, 4]
>
> On 11 December 2014 at 07:51, Alex Ames > wrote:
>
>> Functions that end with an exclamation point modify their arguments, but
>> they can return values just like any other function. For example:
>>
>> julia> x = [1,2]; y = [3, 4]
>> 2-element Array{Int64,1}:
>> 3
>> 4
>>
>> julia> append!(x,y)
>> 4-element Array{Int64,1}:
>> 1
>> 2
>> 3
>> 4
>>
>> julia> z = append!(x,y)
>> 6-element Array{Int64,1}:
>> 1
>> 2
>> 3
>> 4
>> 3
>> 4
>>
>> julia> z
>> 6-element Array{Int64,1}:
>> 1
>> 2
>> 3
>> 4
>> 3
>> 4
>>
>> julia> x
>> 6-element Array{Int64,1}:
>> 1
>> 2
>> 3
>> 4
>> 3
>> 4
>>
>> The append! function takes two arrays, appends the second to the first,
>> then returns the values now contained by the first array. No recursion
>> craziness required.
>>
>> On Thursday, December 11, 2014 1:11:50 AM UTC-6, Sean McBane wrote:
>>>
>>> Ivar is correct; I was running in the Windows command prompt and
>>> couldn't copy and paste so I copied it by hand and made an error.
>>>
>>> Ok, so I understand that append!(X,Y) is modifying X in place. But I
>>> still do not get where the output for the second case, where the result of
>>> append!(X,Y) is assigned back into X is what it is. It would make sense to
>>> me if this resulted in a recursion with Y forever getting appended to X,
>>> but as it is I don't understand.
>>>
>>> Thanks.
>>>
>>> -- Sean
>>>
>>> On Thursday, December 11, 2014 12:42:45 AM UTC-6, Ivar Nesje wrote:
>>>>
>>>> I assume the first line should be
>>>>
>>>> > X = [1,2]; Y = [3,4];
>>>>
>>>> Then the results you get makes sense. The thing is that julia has
>>>> mutable arrays, and the ! at the end of append! indicates that it is a
>>>> function that mutates it's argument.
>>>
>>>
>
Re: [julia-users] Unexpected append! behavior
Ivar is correct; I was running in the Windows command prompt and couldn't copy and paste so I copied it by hand and made an error. Ok, so I understand that append!(X,Y) is modifying X in place. But I still do not get where the output for the second case, where the result of append!(X,Y) is assigned back into X is what it is. It would make sense to me if this resulted in a recursion with Y forever getting appended to X, but as it is I don't understand. Thanks. -- Sean On Thursday, December 11, 2014 12:42:45 AM UTC-6, Ivar Nesje wrote: > > I assume the first line should be > > > X = [1,2]; Y = [3,4]; > > Then the results you get makes sense. The thing is that julia has mutable > arrays, and the ! at the end of append! indicates that it is a function > that mutates it's argument.
[julia-users] Unexpected append! behavior
Hi all, I'm an engineering student with interests in numerical simulation of, well, just about anything, and stumbled across Julia and have been experimenting a bit. I have no idea if this question has been addressed before, but I'm seeing a behavior that's not quite what I'd expect from the append! method. At the prompt, if I type > X = [1,2]; Y = [1,2]; > append!(X,Y) 4-element Array(Int64,1): 1 2 3 4 This is the output that I expect to see. But if I try to store this result back in X: > X = append!(X,Y) 6-element Array(Int64,1): 1 2 3 4 3 4 This is counterintuitive to me. It looks as though the append is recursing and adding Y onto the end of the new X obtained from appending... but only once. If I'm causing a recursion by this call, why doesn't it continue? Thanks to anyone who can enlighten me. -- Sean
