Hi Julian, Thanks for a lot for the prompt response and support. I will try running the test with JMH and will let you know the feedback.
I wish you all have a prosperous new year. Thanks and Regards Lekshmi B.G Email: lekshmib...@gmail.com On Mon, Dec 31, 2018 at 10:38 AM Julian Feinauer < j.feina...@pragmaticminds.de> wrote: > Hi Lekshmi, > > your activity sounds very interesting. > One important thing to note is that Performance testing in Java is always > tricky due to JIT and "warmup" phase of the JVM. Thus it is generally > recommended to do these tests with JMH ( > https://openjdk.java.net/projects/code-tools/jmh/). > > I would assume that the time for sql2rel reduces drastically (perhaps one > or two orders) when run with JMH. > > Best > Julian > > Am 30.12.18, 23:12 schrieb "Lekshmi" <lekshmib...@gmail.com>: > > Hello Folks, > > For my research activities, I was trying to perform a benchmark > comparison > between calcite with other database systems. As an initial step, I was > trying to do it for *Calcite* and *PostgresSql*. So, I thought TPCH > queries > were the right thing to start with. I tried running the TpchTest ( > > https://github.com/apache/calcite/blob/master/plus/src/test/java/org/apache/calcite/adapter/tpch/TpchTest.java > ) > by adding the *CalciteTimingTracer* in the junit tests to determine the > execution time. While doing so, I could see that the execution time in > calcite is significantly higher compared to postgresSql. On further > investigation, I could see that we generate the required datas > required for > these queries(which comes around 150,000 for some tables) and I was > under > an impression that most of the time was spend on the data generation > and > that the query execution could be faster. So, I modified the relevant > schema class ( > > https://github.com/apache/calcite/blob/master/plus/src/main/java/org/apache/calcite/adapter/tpch/TpchSchema.java > ) > to perform the data generation and query execution separately. Then, I > traced the time took for just query execution. Even, then there was a > significant difference from that of PostgresSql. > > I, also enabled the *log4j.rootLogger* to *TRACE * to find the time > spend > for sql2rel and optimization phases of the class Prepare > < > > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/Prepare.java > >. > And, to my surprise, I could see that calcite takes a time of 355ms for > sql2rel and 352ms for optimization for the junit test *testQuery01*. > On the > other side, the same query gave a planning time of 0.163ms in Postgres. > > I would like to know, if this is the right way to test the performance > of > TPCH queries using apache calcite. Can anyone let me know if there > exist > any better ways to do it. > > And, while searching through JIRA, I could find a ticket > https://issues.apache.org/jira/browse/CALCITE-2169 which was created > by > Edmon Begoli for performing a comparative performance study of the > calcite > framework. I think, its related to my current problem. I have no idea > regarding the status of the ticket. It would be really great if someone > could help me with some information on it. > > Also, now coming to the personal preference, I would like to continue > my > research in calcite due to its simplicity and extensibility. But, if I > fail to give a good case study in favour of Calcite, I am afraid that I > could loose an opportunity to work with calcite. > > Thanks and Regards > > Lekshmi B.G > Email: lekshmib...@gmail.com > > >
