I agree that lack of documentation is a problem. However, I can’t see an obvious way to fix it.
The fact is, we just don’t receive contributions of documentation. I don’t believe we have ever received contributions from someone who is a technical writer by profession. Nor has Calcite ever been a paid commercial offering by any company. So, what documentation we have we squeeze out of the development flow. (Yes, I’m the a$$hole who refused to merge your PR because I didn’t think your commit message was good enough.) Let’s look at the current development flow: * People submit PRs with some new classes and methods, and those classes and methods must have javadoc; * The PRs must have a Jira case that contains an adequate amount of specification (good summary and description) and a commit comment that matches the summary. * The release notes consist of the commit messages plus a link to the Jira case. * There are discussions in Jira cases, and on the dev list. * For a few types of change (adapters, SQL syntax changes, new SQL functions, model syntax) there is a prescribed place to put documentation. That documentation is very terse (e.g. one line to describe a SQL function). Can we squeeze more out of the current development flow? Should we find ways to get the current documentation indexed better? Or should we find new places to put documentation (e.g. encourage a series of blog posts)? On the bright side, I see more people coming forward to review and merge PRs. Thank you to those people. And, thank you to all of on the dev list. You make this a rewarding, productive, positive, kind, respectful community. Julian > On Nov 6, 2023, at 1:55 PM, Francis Chuang <francischu...@apache.org> wrote: > > Thank you, Stamatis, for being chair and leading the project for 2023. > > I also agree that it's been a great year for Calcite as we've seen 3 releases > for Calcite so far. We have also added a significant amount of committers. > > There have also been more reviewers for PRs and participants on the mailing > lists, which is an improvement to issues identified in previous years. > > I also agree that the documentation for Calcite on the website can be a bit > confusing and sparse, however, I think this can be remedied by reworking the > information architecture of the website, so that information is presented in > a more bite-sized and digestible manner. > > I am also excited about the possibility of automating the release process > [1]. I have not had time to look into it further, but hopefully, we can have > some discussions about this as a community next year. I believe implementing > release automation will significantly reduce the amount of mechanical work to > produce a release and can potentially increase our release velocity. > > I look forward to another exciting year with a new chair next year and I am > excited to see further developments for the project and the community. > > Francis > > [1] https://lists.apache.org/thread/qkstx4o9qw90fxzqcfnp69w33h7vw2yg > > On 6/11/2023 8:24 pm, Stamatis Zampetakis wrote: >> Thanks everyone for sharing your thoughts. >> Indeed documentation is a common pain point especially with those >> starting off with Calcite. However, over the past few years there are >> more and more people publishing blogs, talks, and tutorials around >> Calcite. I find it very helpful and I would definitely like to see >> more improvements around this area. >> As far as it concerns testing we can always do better and personally I >> am a big fan of extending test coverage. There are many good ideas >> around improvements in our JIRA backlog and I hope to see many of >> those implemented over the next year. >> Note that the PMC nomination period has ended. The PMC will proceed >> with the vote on existing candidates in the following days. >> Best, >> Stamatis >> On Thu, Oct 26, 2023 at 8:42 PM Ran Tao <chucheng...@gmail.com> wrote: >>> >>> Really, totally agree with Mihai & Hongyu about documentation. >>> >>> In fact, I have a similar problem. >>> The learning curve of calcite is indeed steep, most of my learning comes >>> from source code. >>> >>> Fortunately, we still have a relatively rich javadoc in the source code. >>> If there is no javadoc, I mostly rely on experience to guess a certain >>> field or a certain logic. >>> There are relatively few people around me who know about calcite. >>> >>> Of course, now I try to ask questions on the mailing list, >>> but considering the asynchronous situation of the mailing list and the fact >>> that new-beginners may not be willing or comfortable to ask questions by >>> email, it may not always work. >>> >>> If we have some documentation from simple to in-depth, >>> such as how to build a simple function, a rule, from parser to optimizer, >>> how to combine some basic components to complete the optimization of a >>> large SQL or even run it. >>> Slowly increase the depth so that it will be more friendly to new-beginners. >>> >>> I can’t imagine that all new people would go directly to the source code to >>> see it. >>> There is still a certain threshold for this. >>> I hope that in the future we can have some documents that are progressive >>> from shallow to deep, >>> no matter what page or form they are provided. >>> >>> hope the community will get better and better in the new year. >>> >>> >>> Best Regards, >>> Ran Tao >>> https://github.com/chucheng92 >>> >>> >>> Hongyu Guo <guohongyu...@gmail.com> 于2023年10月27日周五 00:05写道： >>> >>>> Hi devs, >>>> >>>> There is no doubt that Calcite boasts powerful capabilities, but the curve >>>> of acquiring and learning calcite is quite steep. After studying Calcite >>>> for a year, I have only gained a limited understanding of some of its >>>> modules. Mihai is about to contribute several documents related to calcite, >>>> which is truly exciting. Calcite is a treasure, and I hope there will be >>>> more developer documentation next year to enable developers to quickly >>>> grasp how to use calcite and how it works. >>>> >>>> Secondly, Calcite supports a wide array of SQL syntax, but there are still >>>> a few syntaxes that it does not cover. I hope that Calcite can be equipped >>>> with more comprehensive SQL syntax parsing capabilities, which will be the >>>> selling point that convinces more developers to opt for Calcite. I am >>>> anticipating that Calcite will support more syntax next year, both standard >>>> and non-standard, and I will continue to contribute in this direction. >>>> >>>> Best, >>>> Hongyu Guo >>>> >>>> On Tue, Oct 24, 2023 at 3:16 AM <mbu...@gmail.com> wrote: >>>> >>>>> Hello all, >>>>> >>>>> Calcite is great! However, I think that there are several areas where the >>>>> project could do better: >>>>> - testing >>>>> - documentation >>>>> - legacy code cleanup >>>>> >>>>> (1) Testing: Most of my work in Calcite so far has been about testing. I >>>>> have contributed the SqlLogicTest suite, which uncovers many bugs. >>>> However, >>>>> I have not filed issues for any of these bugs, and I have not fixed any >>>> of >>>>> these bugs either, since I have been very busy with other bugs I found >>>>> using other testing methods. I also have written a new test fixture used >>>>> for SqlOperatorTest tests, which finds lots of bugs, but I haven't sent a >>>>> PR for this one since there are too many failures. I was hoping other >>>>> people would be looking at the SLT bugs. (One other test fixture I >>>> proposed >>>>> was merged, though, and I hope it will prevent quite a few bugs too: >>>>> https://github.com/apache/calcite/pull/3433). >>>>> >>>>> I think testing could be improved a lot by getting Calcite to run more >>>>> tests from other open-source projects (e.g., Postgres, or MySql). For >>>>> example, it would be great to have a script to convert Postgres regress >>>>> tests >>>>> >>>> https://github.com/postgres/postgres/tree/master/src/test/regress/expectedto >>>>> to Quidem tests. These could be slow tests. >>>>> >>>>> (2) Documentation; Calcite has a steep learning curve. I think that >>>> better >>>>> documentation could help attract more developers and users. I would like >>>> to >>>>> propose adding a blog to the Calcite project web page. If every committer >>>>> writes two blog posts per year about some Calcite technical aspect, then >>>>> the project can quickly accumulate a lot of helpful information. The dev >>>>> mailing list does not seem to be well indexed by search engines; we could >>>>> even take some answers for questions on the mailing list and turn them >>>> into >>>>> short articles. I have written a blog post about several Calcite IRs, >>>> which >>>>> I plan to post in the following days on our company website (I will >>>>> actually send a request for review on this list when I do), but it would >>>> be >>>>> great to cross-post such documents on the Calcite project blog. >>>>> >>>>> (3) While poking around I have found examples of data structure that are >>>>> probably confusing. Some of them have been known to be confusing for a >>>> long >>>>> time: e,g: RexLiteral.typeName. >>>>> >>>> https://github.com/apache/calcite/blob/c83ac69111fd9e75af5e3615af29a72284667a4a/core/src/main/java/org/apache/calcite/rex/RexLiteral.java#L214 >>>>> A comment from 2006 indicates that this field should not exist. >>>>> >>>>> Another example is https://github.com/apache/calcite/pull/3411, where I >>>>> tried to fix up some problems with the type families. Such PRs are >>>>> difficult to accept, in the short term they may increase the code entropy >>>>> (like this one does, by creating deprecated functions and keeping the old >>>>> ones too), and it is very hard to make sure that they don't break >>>> anything >>>>> for users, and their value is questionable if people do not actually >>>> remove >>>>> all uses that should be deprecated. This requires a sustained effort to >>>>> cleanup; contributors could periodically self-assign a small cleanup >>>> task. >>>>> >>>>> Thank you, >>>>> Mihai >>>>> >>>>> -----Original Message----- >>>>> From: Stamatis Zampetakis >>>>> Sent: Sunday, October 22, 2023 9:00 AM >>>>> To: dev@calcite.apache.org >>>>> Subject: [DISCUSS] State of the project 2023 >>>>> >>>>> Hello community and happy birthday Calcite, >>>>> >>>>> Eight years ago (22 October 2015) Calcite graduated as a top-level Apache >>>>> project [1]. At that point, the community decided to have an annual >>>> “state >>>>> of the project” discussion, and we arrived at that time of the year. >>>>> >>>>> We have had three Calcite releases (1.33.0 to 1.35.0) so far in 2023 [2], >>>>> with probably one more coming before the end of the year. With roughly >>>> 300 >>>>> commits coming in, it is evident that the community did a great job in >>>>> fixing numerous bugs, landing new features, and bringing notable >>>>> improvements and optimizations to the project. It is worth highlighting >>>> the >>>>> collective effort that was done for improving the BigQuery dialect where >>>>> more than 60 new SQL functions were added. >>>>> >>>>> Regarding the sub-project Avatica, we had a release at the beginning of >>>>> the year (1.23.0) [3] featuring some new configurations for fetchSize and >>>>> SSL key stores along with various other improvements and bug fixes. Since >>>>> the last release, there have been roughly 10 new commits in the main >>>> branch >>>>> mostly comprising dependency upgrades and build fixes. As in previous >>>> years >>>>> the activity is rather low but there are still volunteers regularly >>>>> checking and contributing to the repo. >>>>> >>>>> In terms of community, I think this has been a great year. We see more >>>> and >>>>> more people participating in email discussions, Jira tickets and Github >>>>> PRs. Our list of committers has grown with Istvan Toth, Alex Plehanov, >>>>> Jiajun Xie, Tanner Clary, Zhe Hu, Jacky Lau, Oliver Lee, TJ Banghart, Dan >>>>> Zou; and so has our PMC with Benchao Li joining the team. With nine new >>>>> committers and one new PMC member till now, it's probably the best year >>>> so >>>>> far (and it's not yet ended). Calcite grows and evolves because of (and >>>>> thanks to) its community, so I would like to thank everyone for being >>>> part >>>>> of this family and working together in a respectful and motivating >>>>> environment. >>>>> >>>>> It was nice to see our community members giving talks to conferences such >>>>> as ApacheCon East Asia, VLDB, and Community over Code presenting Calcite >>>>> and/or its application. As in previous years, we had one online Calcite >>>>> meetup, which was a great opportunity for the community to virtually meet >>>>> and share some interesting presentations, and hopefully we could add some >>>>> in-person events in the years to come. >>>>> >>>>> Over the past few years we always had problems with reviewing pull >>>>> requests with a lot of weight falling upon a few individuals. In the last >>>>> board report for (Q3 2023), the numbers were a bit more encouraging >>>> showing >>>>> more people involved in reviews. The problem is not yet solved but we are >>>>> moving in a promising direction. >>>>> >>>>> To conclude, I will repeat the questions from previous years: >>>>> 1) What else are we doing well in the project? >>>>> 2) What areas do we need to do better? >>>>> >>>>> Please take some time to share your thoughts! >>>>> >>>>> Note that this discussion is for everyone, not only for committers / PMC >>>>> members; even if you have never sent an email to the dev list before, now >>>>> is a good time to do so :) >>>>> >>>>> Finally, it has been a privilege to serve as Calcite's PMC Chair this >>>>> year. I have learnt a lot and I am very grateful for the opportunity >>>> that I >>>>> was given. Following our yearly rotation tradition, I will step down as >>>>> Chair by the end of the year, and a new one will have to be chosen. As we >>>>> discussed some time ago [4], if you have any suggestion and you would >>>> like >>>>> to put someone forward as a potential next Chair, please send an email to >>>>> priv...@calcite.apache.org. The nomination period for the new chair will >>>>> remain open for the next two weeks (till 2023-11-05). The PMC will study >>>>> all proposals, discuss, and start a vote soon after 2023-11-05. The >>>> change >>>>> will be effective some time in December once the resolution is approved >>>> by >>>>> the board. >>>>> >>>>> Best, >>>>> Stamatis >>>>> >>>>> [1] https://calcite.apache.org/news/2015/10/22/calcite-graduates/ >>>>> [2] https://calcite.apache.org/news/ >>>>> [3] https://calcite.apache.org/avatica/news/ >>>>> [4] https://lists.apache.org/thread/gplfqs4snr1b6h62cngyvb65sz41z3fk >>>>> >>>>> >>>>
