Baunsgaard commented on pull request #1303:
URL: https://github.com/apache/systemds/pull/1303#issuecomment-858012541
> I think the way the WTree is added to the instructions seems reasonable if
the CompressionCPInstruction needs the entire tree, but is this really the case?
Yes, but that is assuming that i know at compile time what information is
needed in the runtime of an arbitrary matrix.
currently we don't have access to this information at compile time, and we
probably should not.
> Isn't it possible to add the information about compression contained in
the tree and add it to the different instructions, so instead of putting all
the information into a single CompressionCPInstruction string (which would have
variable length), the information should be placed in the relevant instruction
strings so that each of the strings contains only the compression information
needed for its instruction execution (resulting in an instruction string of
fixed length).
Since the Compression instruction is one instruction i don't see how i
should create multiple instruction strings? Maybe I'm missing the question.
> Maybe this is because I do not fully understand what you mean by "means of
separating different loops into sub tree nodes in the structure". Couldn't the
nested structures be represented by adding to the different instructions
instead of adding all of it to the same instruction?
To clarify the loops:
Lets say i read in a matrix
```
X = read("...")
```
This matrix is used in a nested for loop at different locations
```
d = mean(X)
for(a in ...){
...
c = max(X)
Xm = X + 2
for(b in ...){
...
B = t(Xm) %*% X
...
}}
```
for optimizing compression for workflow i need to know
1. what functions are called on X and how many times(at least an
approximation), mean once, max in forloop, compressed multiplication in forloop
forloop.
2. what derived compressed matrices is constructed such as Xm (Still
compressed).
3. what operations is performed on the derived matrices
4. how many decompressing operations there is
all this can be encapsulated in the "AWTree" object, and is hard to add to
the instruction string since this string then would have to support all this
complexity.
The information cannot just be extracted at compile time because i need
access to the actual data that is being compressed to optimize for the given
instructions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
