I think I'd be fine exposing this through a test Jar, but it seems to me that if we were to put it into a normal package it would turn into the situation we want to avoid. People would use it for unintended purposes and it would become a distraction.
What do you think about using the tests Jar for this? On Thu, Jan 25, 2024 at 12:48 PM Jack Ye <yezhao...@gmail.com> wrote: > Yes, sorry I did not make it clear, I also agree it is not the right > direction to invest a lot of community effort. I am more talking about > casual use cases like importing a server for unit tests outside Iceberg, > running some local debugging, etc. I think it would be valuable to provide > a server in Iceberg for that purpose, and maybe vend it as test utils. > Thoughts? > > -Jack > > On Thu, Jan 25, 2024 at 11:35 AM Ryan Blue <b...@tabular.io> wrote: > >> > I know we have the RESTCatalogAdapter and RESTCatalogSevlet for unit >> tests, and technically we have a very similar Jetty server implementation >> in TestRESTCatalog. Should we think about making those components out of >> the tests into an iceberg-rest-server module for this use case, and merge >> with the implementation that Gravitino has? >> >> I think that this would take the Iceberg project in the wrong direction. >> Iceberg has always been a library and I think it should continue to be. >> Concerns about runtime should be left to other projects that need to fit >> into existing infrastructure or skillsets of people maintaining them. The >> question of whether to use Jetty or Tomcat or whatever else is a serious >> consideration, as is how to monitor that application and send metrics. I >> think it would slow down the core purpose of Iceberg if we got distracted >> by these things. >> >> In fact, I think that this project shows that the library is getting the >> balance right: it is using `CatalogHandlers` for their intended purpose. It >> has opinions about how to run the actual HTTP service and people that agree >> can use it. Other people could use `CatalogHandlers` to build on a >> different foundation. >> >> On Thu, Jan 25, 2024 at 11:13 AM Jack Ye <yezhao...@gmail.com> wrote: >> >>> Really cool project! >>> >>> I browsed a bit of the codebase, and see this implementation of the REST >>> service backend: >>> - >>> https://github.com/datastrato/gravitino/blob/main/catalogs/catalog-lakehouse-iceberg/src/main/java/com/datastrato/gravitino/catalog/lakehouse/iceberg/IcebergRESTService.java#L39 >>> - >>> https://github.com/datastrato/gravitino/blob/main/catalogs/catalog-lakehouse-iceberg/src/main/java/com/datastrato/gravitino/catalog/lakehouse/iceberg/ops/IcebergTableOps.java#L42-L51 >>> >>> Looks like it is initializing a Jetty server that uses CatalogHandlers >>> to delegate the execution to a specific Java Catalog implementation. >>> >>> I think this is actually something that is lacking today in Iceberg, >>> which is an easy way for users to start an actual REST HTTP server. >>> >>> I know we have the RESTCatalogAdapter and RESTCatalogSevlet for unit >>> tests, and technically we have a very similar Jetty server implementation >>> in TestRESTCatalog. Should we think about making those components out of >>> the tests into an iceberg-rest-server module for this use case, and merge >>> with the implementation that Gravitino has? >>> >>> Best, >>> Jack Ye >>> >>> On Thu, Jan 25, 2024 at 10:47 AM Yufei Gu <flyrain...@gmail.com> wrote: >>> >>>> Thanks Justin for the sharing. >>>> >>>> It's pretty cool to see an open source REST catalog implementation in >>>> action. Having dabbled a bit in the early development of Gravitino myself, >>>> I'm really excited about its potential with the Iceberg REST catalog. >>>> >>>> The idea of Gravitino moving to an ASF project is promising. It’ll >>>> surely boost its visibility and open up more doors for collaboration and >>>> adoption. >>>> >>>> Looking forward to where this goes. Keep up the fantastic work! >>>> >>>> Yufei >>>> >>>> >>>> On Thu, Jan 25, 2024 at 5:55 AM Jean-Baptiste Onofré <j...@nanthrax.net> >>>> wrote: >>>> >>>>> Hi Justin, >>>>> >>>>> I talked with Junping a couple of months ago about Gravitino. Thanks >>>>> for sharing ! >>>>> >>>>> Regards >>>>> JB >>>>> >>>>> On Thu, Jan 25, 2024 at 12:15 AM Justin Mclean < >>>>> jus...@classsoftware.com> wrote: >>>>> > >>>>> > Hi, >>>>> > >>>>> > We open-sourced a new project, Gravitino, in December and have been >>>>> working on growing the community and adding new functionality. We plan to >>>>> donate the project to the ASF this year. Gravitino is a unified metadata >>>>> lake solution offering a unified approach to managing datasets from >>>>> diverse >>>>> sources and regions across multiple cloud platforms. Its core is an >>>>> Iceberg >>>>> REST catalog service implementation to manage Iceberg tables efficiently. >>>>> > >>>>> > If this sounds like something you would be interested in, then the >>>>> following resources will help: >>>>> > - Blog post: >>>>> https://datastrato.ai/blog/gravitino-iceberg-rest-catalog-service/ >>>>> > - Gravitino documentation: https://datastrato.ai/docs/0.3.1/ >>>>> > - Iceberg REST service documentation: >>>>> https://datastrato.ai/docs/0.3.1/iceberg-rest-service >>>>> > >>>>> > We welcome any feedback and suggestions you have, and as always, all >>>>> contributions are welcome. You can find the source code at >>>>> https://github.com/datastrato/gravitino. >>>>> > >>>>> > Kind Regards, >>>>> > Justin >>>>> >>>> >> >> -- >> Ryan Blue >> Tabular >> > -- Ryan Blue Tabular
