Hi Robert,
> Isn't it okay to yield a HTTP/409 in this particular case
(namespace properties) and let the user figure out the right values?
Unfortunately, I do not think it would be ok. AFAIK, current Iceberg java
code interprets 409 in this case as "namespace already exists" [1].
I'm personally
Hi Yufei,
1: I do not really know. This is a question about a specific deployment
environment.
2: I'm not sure I understand your question. Two endpoints are necessary in
cases when the server's view of the network is different from the engine's
view.
Cheers,
Dmitri.
On Thu, Jul 31, 2025 at 5:
Thanks for the explanation. Two questions:
1. Should the public endpoint used by engines still work with Polaris even
if it co-locates with MinIO server?
2. Can we set Polaris endpoint directly to the internal address in that
case? Another way to ask this question is that why do we need to keep bot
Thanks for the async task proposal. I think it's the right direction
for async light tasks. Meanwhile, we will still need other models:
1. A scalable way to execute synchronous tasks
2. A scalable way to execute heavy async tasks, e.g., table maintenance
tasks.
The delegation service[1] is a good
Hi Eric,
Unfortunately, SKIP_CREDENTIAL_SUBCOPING_INDIRECTION=true does not work for
non-AWS S3 storage because that setting effectively causes "endpoint"
settings to be ignored.
I'll look into other options for now.
Cheers,
Dmitri.
On Thu, Jul 31, 2025 at 3:21 PM Eric Maynard
wrote:
> The ex
The existence of SKIP_CREDENTIAL_SUBCOPING_INDIRECTION may be helpful
context for this thread. With that config set, we already support loading
tables without subscoped vended credentials. If there are certain
storage configurations that only work with this config set, that doesn't
seem like it wou
I'm fine with the plan although I think we should probably change step 4 to
allow both the current implementation and the new implementation to exist at
the same time with a flag for switching over to the new task implementation.
While the new implementation may be much better, it is a pretty si
Hi Dmitri,
Thanks for your reply. My proposal doesn’t affect how
polaris-runtime-defaults is “pulled in” by downstream builds.
We can, however, explore ways to make the downstream integration experience
better, but imo only *after* the merge.
Thanks,
Alex
Le jeu. 31 juil. 2025 à 19:56, Dmitri B
Hi Yufei,
The "how" in your question depends on the deployment environment, I guess.
There are a lot of variants.
If you wonder whether such a situation is possible in practice, I
believe it is. An example would be self-hosting non-AWS S3 storage and
Polaris in a way that Polaris connections go t
Hi Alex,
Unifying polaris-service-common and polaris-runtime-service sounds good to
me.
Re: config, I suppose it should not be an issue to have Quarkus (or
Smallrye) dependencies in any "service" modules.
Side note: I'd like to exclude polaris-runtime-defaults from the transitive
dependency chai
> I mentioned smallrye-config, not Quarkus, so you _can_ annotate those.
Fair point: you are absolutely right. We *could* do things the other
way around, and move the configuration classes from the
polaris-runtime-service module to the polaris-service-common module.
BUT: my main point for proposi
Hi Dimtri,
That generally makes sense to me. For awareness, could you elaborate a bit
on how the Polaris server and query engines (like Spark, Trino, etc.) might
access the same object storage (e.g., MinIO) via different DNS endpoints?
Yufei
On Thu, Jul 31, 2025 at 4:36 AM Alexandre Dutra wrot
> That's not possible because there are no Quarkus dependencies in that module,
> so you can't annotate with @WithDefault, for example.
I mentioned smallrye-config, not Quarkus, so you _can_ annotate those.
On Thu, Jul 31, 2025 at 6:34 PM Yufei Gu wrote:
>
> +1. Thanks Alex for driving this. Ot
+1. Thanks Alex for driving this. Other than the benefits discussed like
removing the duplicated config classes like
`QuarkusStorageCredentialCacheConfig`, or removing `.quarkus.` in the
package name, we can finally put classes like S3AccessConfig
and StsClientsPool into the right package.
Yufei
The project already *has* adopted Quarkus for good, so I think making the
module structure reflect that is fine. In addition to the configuration
issue, which is significant, I think cutting down on the number of modules
we publish is a noble goal.
—EM
On Fri, Aug 1, 2025 at 00:43 Alexandre Dutra
Hi Robert,
> I think we can get away by having the smallrye-config annotations on the
> "parent" interface.
That's not possible because there are no Quarkus dependencies in that
module, so you can't annotate with @WithDefault, for example.
Indeed merging those two modules would mean that we're
Hi,
not sure whether exposing the object storage credentials given to
Polaris to all clients isn't going to cause a "false impression of
security" (aka: "our credentials are vended by Polaris, so we're safe"
- nope...).
With my "evil user" hat on, I'd try to figure out the configuration
option (is
Is the issue here just the configuration types or is there more?
For the config types, I think we can get away by having the
smallrye-config annotations on the "parent" interface.
My concern is that polaris-runtime-service is rather the Quarkus specifics.
CDI is great, just Quarkus isn't the only
Having retries would be great.
However, for namespace property updates, the situation is undefined,
because there are no "prerequisites" that have to be satisfied.
Say, you have two concurrent requests:
- One requests sets a=b
- Another request removes a
- Yet another request sets a=c
Technically
Hi Dmitri,
I think your suggestion makes sense. We added something similar in
Nessie long ago, and it is definitely useful.
I left some comments in the PR.
Thanks,
Alex
On Thu, Jul 31, 2025 at 4:12 AM Dmitri Bourlatchkov wrote:
>
> Hi All,
>
> I propose to add an `endpointInternal` optional pa
Hi all,
The polaris-service-common module is a reminiscence of the times where
we were still supporting two runtimes.
I think it has now become obsolete, and could be merged into
polaris-runtime-service.
One annoyance that polaris-service-common brings is with configuration
classes that have to
21 matches
Mail list logo
