[GitHub] incubator-quickstep pull request #346: QUICKSTEP-124: Add a python script to...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-quickstep/pull/346


---

[GitHub] incubator-quickstep issue #346: QUICKSTEP-124: Add a python script to auto f...

[zuyu]

Github user zuyu commented on the issue:

https://github.com/apache/incubator-quickstep/pull/346
  
LGTM!


---

[GitHub] incubator-quickstep pull request #346: Add a python script to auto fix CMake...

[jianqiao]

GitHub user jianqiao opened a pull request:

https://github.com/apache/incubator-quickstep/pull/346

Add a python script to auto fix CMakeLists files

This PR adds a script that intends to help improve developer productivity 
by automatically fixing `CMakeLists.txt` files for the Quickstep project (with 
best effort).

The script will do the following things:
- Scan the repo's subdirectories and collect `#include` information from 
all source code files.
- Parse existing `CMakeLists.txt` files and convert all "recognized" 
commands into proper intermediate representations -- the "unrecognized" part 
will be kept as "verbal" lines.
- Resolve subdirectories, targets and link dependencies. Add / delete / 
update the corresponding entries.
- Convert the intermediate representations back to `CMakeLists.txt` files.

**NOTE:** Currently the script is at its initial stage and will not update 
tests or conditional targets (i.e. those within cmake `if` commands). It is 
likely to work well if you just create/delete some files or add/remove some 
`#include`'s -- otherwise additional manual fixes may need to be done after 
applying the script.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jianqiao/incubator-quickstep 
autofix-cmake-tool

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-quickstep/pull/346.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #346


commit 73d796dee760a03a91d55cb0fe4d8f073f831237
Author: Jianqiao Zhu 
Date:   2018-04-27T22:28:51Z

Add a python script to auto fix CMakeLists files




---

[GitHub] incubator-quickstep pull request #343: Fix all CMakeLists.txt for automated ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-quickstep/pull/343


---

[GitHub] incubator-quickstep issue #343: Fix all CMakeLists.txt for automated process...

[zuyu]

Github user zuyu commented on the issue:

https://github.com/apache/incubator-quickstep/pull/343
  
LGTM!


---

Re: Eviction Policy Algorithm

[Dylan Bacon]

Navneet,

Thank you for the warning. A cursory search didn't turn up any patent on 
the algorithms. As this paper is 14 years old I'm not sure if contacting 
the original authors would be fruitful, but unless there's a better way 
to determine this I'll have to try if this is an approach I consider 
further.


Harshad,

You're correct, it might be an easier initial approach to improve our 
existing algorithms and attempt to parallelize serial parts instead of 
rip the system out and try again with a far more complex algorithm. I'll 
look at the current LRU-k approach we take and see if any 
parallelization is possible, or working with mutexes could be improved.


As for the concerns, I do not believe that the current LRU-k 
implementation notes block size or temporary/permanent blocks. It seems 
strictly based on time. Evicting hash tables would take new structures 
build to detect, evaluate, and evict said tables I believe. It's 
something to consider but attention might be better spent on the mutex 
situation and general algorithm efficiency.



On 4/27/18 2:39 PM, Navneet Potti wrote:

Caution! That paper is from IBM. A lot of the caching algorithms developed there 
(including one  I worked on, which 
handles the problem of blocks being of different sizes) come with patent restrictions.



On Apr 27, 2018, at 2:21 PM, Harshad Deshmukh  wrote:

Hi Dylan,


That's a good find! Thanks for sharing the paper.


I have dealt with the eviction policy code a little bit. Would you mind 
describing the issues first? As far as I know, the referencing and 
de-referencing of blocks is serial and sometimes causes performance 
bottlenecks. We may need to analyze if the bottlenecks are due to the eviction 
algorithm or synchronization involved in the eviction policy code. I tried the 
following fix 
https://github.com/hbdeshmukh/incubator-quickstep/commit/4b32336a1b89f70d1c17ffe7956ae6f78e691d7c
 to make the synchronization a bit fine grained and it helped a bit. Please 
feel free to adopt this fix.


As a separate discussion, we have a complicated buffer management problem for 
following reasons. I am not sure if the current LRU-k eviction implementation 
addresses these concerns:

  1.  Blocks could be of different sizes and have different lifetimes (stored 
vs temporary). Therefore the consequences of evicting different blocks could be 
different.
  2.  We currently do not evict join hash tables. When memory is really scarce, 
the buffer pool should evict hash tables.


How much attention should we pay to buffer management is more of a strategic 
decision, considering that our focus is on in-memory settings.


Thanks,

Harshad


From: Dylan Bacon 
Sent: Friday, April 27, 2018 2:07:14 PM
To: dev@quickstep.incubator.apache.org
Subject: Eviction Policy Algorithm

Attached screenshot of an algorithm that seems worthwhile to implement
in Quickstep for an eviction policy. The language and structure would
need to be adapted from cache pages to our database blocks, I'm not sure
how straightforward that would be. Sending this to the dev email for
broader discussion as I initially sent to individuals. Thoughts and
input are welcome.

Source paper:
http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=C660AD720FD52C67A063B270F4B6DDF4?doi=10.1.1.105.6057&rep=rep1&type=pdf

--
Regards,

Dylan Bacon
University of Wisconsin - Madison
Department of Computer Sciences
dba...@wisc.edu

Re: Eviction Policy Algorithm

[Navneet Potti]

Caution! That paper is from IBM. A lot of the caching algorithms developed 
there (including one  I worked on, 
which handles the problem of blocks being of different sizes) come with patent 
restrictions.


> On Apr 27, 2018, at 2:21 PM, Harshad Deshmukh  wrote:
> 
> Hi Dylan,
> 
> 
> That's a good find! Thanks for sharing the paper.
> 
> 
> I have dealt with the eviction policy code a little bit. Would you mind 
> describing the issues first? As far as I know, the referencing and 
> de-referencing of blocks is serial and sometimes causes performance 
> bottlenecks. We may need to analyze if the bottlenecks are due to the 
> eviction algorithm or synchronization involved in the eviction policy code. I 
> tried the following fix 
> https://github.com/hbdeshmukh/incubator-quickstep/commit/4b32336a1b89f70d1c17ffe7956ae6f78e691d7c
>  to make the synchronization a bit fine grained and it helped a bit. Please 
> feel free to adopt this fix.
> 
> 
> As a separate discussion, we have a complicated buffer management problem for 
> following reasons. I am not sure if the current LRU-k eviction implementation 
> addresses these concerns:
> 
>  1.  Blocks could be of different sizes and have different lifetimes (stored 
> vs temporary). Therefore the consequences of evicting different blocks could 
> be different.
>  2.  We currently do not evict join hash tables. When memory is really 
> scarce, the buffer pool should evict hash tables.
> 
> 
> How much attention should we pay to buffer management is more of a strategic 
> decision, considering that our focus is on in-memory settings.
> 
> 
> Thanks,
> 
> Harshad
> 
> 
> From: Dylan Bacon 
> Sent: Friday, April 27, 2018 2:07:14 PM
> To: dev@quickstep.incubator.apache.org
> Subject: Eviction Policy Algorithm
> 
> Attached screenshot of an algorithm that seems worthwhile to implement
> in Quickstep for an eviction policy. The language and structure would
> need to be adapted from cache pages to our database blocks, I'm not sure
> how straightforward that would be. Sending this to the dev email for
> broader discussion as I initially sent to individuals. Thoughts and
> input are welcome.
> 
> Source paper:
> http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=C660AD720FD52C67A063B270F4B6DDF4?doi=10.1.1.105.6057&rep=rep1&type=pdf
> 
> --
> Regards,
> 
> Dylan Bacon
> University of Wisconsin - Madison
> Department of Computer Sciences
> dba...@wisc.edu
> 

Re: Eviction Policy Algorithm

[Harshad Deshmukh]

Hi Dylan,


That's a good find! Thanks for sharing the paper.


I have dealt with the eviction policy code a little bit. Would you mind 
describing the issues first? As far as I know, the referencing and 
de-referencing of blocks is serial and sometimes causes performance 
bottlenecks. We may need to analyze if the bottlenecks are due to the eviction 
algorithm or synchronization involved in the eviction policy code. I tried the 
following fix 
https://github.com/hbdeshmukh/incubator-quickstep/commit/4b32336a1b89f70d1c17ffe7956ae6f78e691d7c
 to make the synchronization a bit fine grained and it helped a bit. Please 
feel free to adopt this fix.


As a separate discussion, we have a complicated buffer management problem for 
following reasons. I am not sure if the current LRU-k eviction implementation 
addresses these concerns:

  1.  Blocks could be of different sizes and have different lifetimes (stored 
vs temporary). Therefore the consequences of evicting different blocks could be 
different.
  2.  We currently do not evict join hash tables. When memory is really scarce, 
the buffer pool should evict hash tables.


How much attention should we pay to buffer management is more of a strategic 
decision, considering that our focus is on in-memory settings.


Thanks,

Harshad


From: Dylan Bacon 
Sent: Friday, April 27, 2018 2:07:14 PM
To: dev@quickstep.incubator.apache.org
Subject: Eviction Policy Algorithm

Attached screenshot of an algorithm that seems worthwhile to implement
in Quickstep for an eviction policy. The language and structure would
need to be adapted from cache pages to our database blocks, I'm not sure
how straightforward that would be. Sending this to the dev email for
broader discussion as I initially sent to individuals. Thoughts and
input are welcome.

Source paper:
http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=C660AD720FD52C67A063B270F4B6DDF4?doi=10.1.1.105.6057&rep=rep1&type=pdf

--
Regards,

Dylan Bacon
University of Wisconsin - Madison
Department of Computer Sciences
dba...@wisc.edu

Eviction Policy Algorithm

[Dylan Bacon]

Attached screenshot of an algorithm that seems worthwhile to implement 
in Quickstep for an eviction policy. The language and structure would 
need to be adapted from cache pages to our database blocks, I'm not sure 
how straightforward that would be. Sending this to the dev email for 
broader discussion as I initially sent to individuals. Thoughts and 
input are welcome.


Source paper: 
http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=C660AD720FD52C67A063B270F4B6DDF4?doi=10.1.1.105.6057&rep=rep1&type=pdf


--
Regards,

Dylan Bacon
University of Wisconsin - Madison
Department of Computer Sciences
dba...@wisc.edu