Re: [ANNOUNCE] Apache Iceberg release 1.6.0

2024-07-25 Thread Jack Ye
Thanks JB for organizing the release, and thanks everyone that has contributed! -Jack On Thu, Jul 25, 2024 at 8:07 AM Jean-Baptiste Onofré wrote: > The Apache Iceberg team is pleased to announce the release of Apache > Iceberg 1.6.0! > > Apache Iceberg is an open table format for huge analytic

Meeting time for catalog community sync

2024-07-24 Thread Jack Ye
on this. I will go ahead to change the time directly if there is a clear preference here, let's see. Best, Jack Ye

Re: Administration of Apache Iceberg Social/Marketing Channels

2024-07-24 Thread Jack Ye
s posted > 3. We publish under the Apache Iceberg YouTube channel in a new playlist > > IMO #3 would be good so we can publish under the same roof and not have > community-led videos fractured over multiple accounts. Ryan, Jack, or > anyone else with some thoughts on this? > > - J

Re: Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

2024-07-23 Thread Jack Ye
r catalog implementations still have value as either REST > back-end catalogs or as regular catalogs for many users. > > On Tue, Jul 23, 2024 at 9:11 AM Jack Ye wrote: > >> For some additional information, we also have some Iceberg HDFS users on >> EMR. Those are mainly us

Re: Dropping JDK 8 support

2024-07-23 Thread Jack Ye
>>>>>>> >>>>>>> Also as Huaxin has discovered in Spark 4.0 Support PR >>>>>>> <https://github.com/apache/iceberg/pull/10622>, looks like we may >>>>>>> have to drop Java8 first in Spark 4.0 module, due

Re: Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

2024-07-23 Thread Jack Ye
doop to extend >> hadoop_catalog? If it is removed, we won't be able to continue providing >> services to customers. So, if possible, please consider this option. >> >> Thank you all. >> >> Kind regards, >> lisoda >> >> >> >> >> >> >> At 2024-0

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

2024-07-23 Thread Jack Ye
s and provide guidance on how members >>> can grow into those roles. (Cover release manager, comitter, PMC member, >>> community member, with links as appropriate). >>> 4. Create guidelines on starting new revision to the specification >>> (i.e. when will t

Re: [ANNOUNCE] Welcoming new committers and PMC members

2024-07-23 Thread Jack Ye
Congratulations!! Best, Jack Ye On Tue, Jul 23, 2024 at 8:16 AM Ryan Blue wrote: > Congratulations, everyone! And thanks for all your contributions! > > On Tue, Jul 23, 2024 at 8:06 AM Mehul Batra > wrote: > >> Congratulations Everyone! >> >> On Tue, Jul 23,

Re: [VOTE] Release Apache Iceberg 1.6.0 RC1

2024-07-22 Thread Jack Ye
+1 (binding) Checked signature, checksum, license Ran unit and integration tests with JDK17 Ran manual tests with Spark 3.5 Best, Jack Ye On Mon, Jul 22, 2024 at 3:02 PM Dmitri Bourlatchkov wrote: > +1 (nb) > > Verified the REST Client warning about OAuth2 URL (again, jus

Re: Dropping JDK 8 support

2024-07-22 Thread Jack Ye
+1 (binding), I did not expect this to be a vote thread, but overall +1 for dropping JDK8 support. -Jack On Mon, Jul 22, 2024 at 10:30 AM Yufei Gu wrote: > +1(binding), as much as I want to drop JDK 8, still encourage everyone to > spark out about any concerns. > Yufei > > > On Mon, Jul 22,

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

2024-07-22 Thread Jack Ye
ocs.google.com/document/d/1S3igb5NqSlYE3dq_qRsP3X2gwhe54fx-Sxq5hqyOe6I/edit [4] https://iceberg.apache.org/contribute/#how-are-proposals-adopted [5] https://orc.apache.org/develop/bylaws/ -Jack On Fri, Jul 19, 2024 at 2:29 PM Jack Ye wrote: > > specifically the discussion of the standard roles > &g

Re: Building with JDK 21

2024-07-19 Thread Jack Ye
+1 for dropping JDK8 support and adding JDK21. > What does dropping Java 8 support mean to companies that are still using Java 8 for Iceberg in production? >From the AWS side, AWS Corretto JDK8 end of life is July 2026, see: https://aws.amazon.com/corretto/faqs/#support_calendar. I would suggest

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

2024-07-19 Thread Jack Ye
ed by a PMC vote. > > Ryan > > On Fri, Jul 19, 2024 at 11:00 AM Owen O'Malley > wrote: > >> I meant specifically the discussion of the standard roles (eg. users, >> committers, pmc, pmc chair) that are well covered in >> https://www.apache.org/foundation/how-it-w

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

2024-07-19 Thread Jack Ye
was not intended to be tied to committer responsibilities. Best, Jack Ye [1] https://community.apache.org/newcommitter.html [2] https://www.apache.org/foundation/voting On Fri, Jul 19, 2024 at 10:22 AM Owen O'Malley wrote: > Everyone is welcome to vote. The Iceberg PMC will have the only bindi

Re: [VOTE] Merge table spec clarifications on time travel and equality deletes

2024-07-19 Thread Jack Ye
+1 (binding) added minor comments to the time travel PR. Best, Jack Ye On Fri, Jul 19, 2024 at 8:22 AM Daniel Weeks wrote: > +1 (binding) > > Thanks, Micah. > > On Thu, Jul 18, 2024 at 8:29 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > >> +1 (non-binding

Re: Administration of Apache Iceberg Social/Marketing Channels

2024-07-18 Thread Jack Ye
Do we consider playlists links? > I guess the question is whether it looks official. Would inclusion in a > playlist be mistaken for an endorsement? Or would we get into unnecessary > debates over what is appropriate for inclusion in a playlist? > > On Thu, Jul 18, 2024 at 3:39 PM Jack Ye

Re: Administration of Apache Iceberg Social/Marketing Channels

2024-07-18 Thread Jack Ye
t;> project to decide what opinions can be posted there. >>> >>> I think that we should remove the last meetup video and only post >>> content like the community sync videos. For everything else, I think it is >>> a good idea to have a separate channel. We ca

Re: Administration of Apache Iceberg Social/Marketing Channels

2024-07-18 Thread Jack Ye
> posting/content management privileges to the Iceberg YouTube channel for these meetup recordings This sounds reasonable to me, when the meetup is recurring. So I am good with doing this for the Seattle meetup series. We have technically already done so for Bits for the community sync meeting

Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

2024-07-18 Thread Jack Ye
that. I could help raise a thread about it after this discussion is closed. Best, Jack Ye [1] https://cloud.google.com/storage/docs/object-versioning#file_restoration_behavior [2] https://cloud.google.com/storage/docs/xml-api/reference-headers#xgoogifgenerationmatch [3] https://learn.micr

Re: Iceberg namespace operation behaviors

2024-07-16 Thread Jack Ye
This is the public link: https://www.youtube.com/watch?v=JSN9T20eAuU -Jack On Tue, Jul 16, 2024 at 7:38 PM Manu Zhang wrote: > Hi Ajantha, > > It looks the catalog sync link is not publicly accessible. Can you check? > > Thanks, > Manu > > On Wed, Jul 17, 2024 at 12:42 AM Ajantha Bhat >

Re: [DISCUSS] Describing REST Server capabilities

2024-07-11 Thread Jack Ye
ead of calling server and getting a "generic 501"? -Jack On Thu, Jul 11, 2024 at 9:51 AM Jack Ye wrote: > Sorry I will take a look at the new comments later today. > > -Jack > > On Wed, Jul 10, 2024, 11:42 PM Eduard Tudenhöfner < > etudenhoef...@apache.or

Re: [DISCUSS] Describing REST Server capabilities

2024-07-11 Thread Jack Ye
>>> make more informed decisions on which endpoints to call. One could argue >>> that generally throwing a 501 on everything that isn't supported might be >>> sufficient, but that doesn't necessarily help a client in knowing which >>> version

Re: [VOTE] spec: remove the JSON spec for content file and file scan task sections

2024-07-11 Thread Jack Ye
+1 (binding) On Thu, Jul 11, 2024 at 3:37 AM Piotr Findeisen wrote: > it looks it's part of the spec that's not connected to the other parts of > the spec (like "dead code") > > +1 (non binding) > > > On Thu, 11 Jul 2024 at 08:30, Eduard Tudenhöfner > wrote: > >> +1 (non-binding) >> >> On Thu,

Re: [Discussion] Apache Iceberg Community Guideline - Initial Version

2024-07-11 Thread Jack Ye
Update: Based on the conversations on the incubator list ( https://lists.apache.org/thread/5ojxny76fr7n1y0hs2rxhr55g1fgcsln), I have updated the document title from "guidelines" back to "bylaws". Thanks for pointing this out Carl! I have also updated the document with a few suggested edits from

Re: [DISCUSS] Enable the discussion tab for iceberg github repos

2024-07-10 Thread Jack Ye
+1 for enabling it on iceberg-rust first. I am curious how search-engine friendly this is, that will be a huge plus if people can find these discussion contents. For people who have used it more like Piotr, do you know about this? I search Trino things quite a lot on Google, but rarely did I find

Re: [VOTE] Fix property names in REST spec for statistics / partition statistics

2024-07-10 Thread Jack Ye
+1 (binding) -Jack On Wed, Jul 10, 2024 at 9:00 AM Steve Zhang wrote: > +1 (non binding) > > Thanks, > Steve Zhang > > > > On Jul 10, 2024, at 1:10 AM, Jean-Baptiste Onofré wrote: > > +1 (non binding) > > >

Re: [DISCUSS] Fix property names in REST spec for statistics / partition statistics

2024-07-09 Thread Jack Ye
Those can be separate things, like sending a [ANNOUNCE] message on this > list rather than having a [VOTE] thread open for 3 days before fixing it. > > Again, I'm asking a question about how we want to handle situations like > this moving forward. As I said originally, I think it's fine to have a >

Re: [DISCUSS] Fix property names in REST spec for statistics / partition statistics

2024-07-09 Thread Jack Ye
I agree with Robert here that we need to get into the habit of doing votes on the spec changes. There are typos that could be in sections like description, that does not affect the overall spec usage at all, maybe those changes do not need an official vote. However, this is a change of an

Re: [DISCUSS] Describing REST Server capabilities

2024-07-09 Thread Jack Ye
at 9:12 AM Dmitri Bourlatchkov > wrote: > >> Re: remote signing, I agree that it does not look like a server >> capability that a client can / should discover. It is more like something >> that the server instructs / configures the client to do. >> >> Cheer

Re: [DISCUSS] Describing REST Server capabilities

2024-07-09 Thread Jack Ye
he server enforces the payload differently based on the >>> TableMetadata.format-version field. If the server does not support v3, it >>> can return unsupported at that time. >>> >>> Either way we go, the table-spec version does not need to be a >>> capability.

Re: [DISCUSS] Enable the discussion tab for iceberg github repos

2024-07-09 Thread Jack Ye
I am not familiar with the GitHub discussion feature, but could we start with GitHub Issue tags + templates to distinguish between actual issues vs this kind of questions? Why is that not sufficient? Also, if there are a lot of questions about the roadmap, I think we should discuss and make good

Re: [Vote] Deprecate oauth tokens endpoint

2024-07-08 Thread Jack Ye
+1 (binding) There are some wording aspects that others still have comments in the PR, but in general, +1 for deprecating the endpoint. Best, Jack Ye On Mon, Jul 8, 2024 at 9:22 AM Robert Stupp wrote: > +1 > > On 08.07.24 18:15, Robert Stupp wrote: > > Hi Everyone, > >

Re: [Proposal] REST Spec: Server-side Metadata Tables

2024-07-03 Thread Jack Ye
e APIs. Just a quick thought for now, I need to think a bit more about this. Best, Jack Ye On Wed, Jul 3, 2024 at 1:45 PM Yufei Gu wrote: > Hi folks, > > I'd like to discuss a new proposal to support server-side metadata tables. > > One of Iceberg's most advantageous features

Re: [DISCUSSION] Addressing security questions in the Iceberg REST specification

2024-07-02 Thread Jack Ye
> as documentation/reference of what each client (language/version) supports > and how. > > Robert > > > [1] https://projectnessie.org/nessie-latest/client_config/#oauth2-settings > > [2] > https://github.com/projectnessie/nessie/tree/main/api/client/src/m

Re: [Discussion] Apache Iceberg Community Guideline - Initial Version

2024-07-02 Thread Jack Ye
people are still out of town, so things were also delayed at this front. Let us know what you think is the best way to proceed! Best, Jack Ye On Tue, Jul 2, 2024 at 9:14 AM Daniel Weeks wrote: > Thanks Owen, I really appreciate the offer to moderate the discussion. I > think that's

Re: Support Securable Objects in Iceberg REST Catalog

2024-07-02 Thread Jack Ye
ntually there's no way around "trust" between the engine and the > catalog. Establishing "trust" in a secure way is not that easy IMO. > > > On 02.07.24 06:30, Jack Ye wrote: > > Thanks Dennis for the detailed analysis and suggestions! Here are a few > question

Re: [DISCUSS] Describing REST Server capabilities

2024-07-02 Thread Jack Ye
Spec is an interesting topic we did not discuss. Robert, how do you envision this to be used? In my mind, if a new table format v3 is launched, there are 2 approaches we can go with, taking CreateTable as an example: (1) increment the related operation version, which means that POST

Re: Re: Support Securable Objects in Iceberg REST Catalog

2024-07-01 Thread Jack Ye
and one of the main > > improvements to that catalog extension would be securable view objects. > > Admittedly, it might require further improvements to compute engines to > > implement the permissions, but having an Iceberg spec would be the first > > step. > > > > Lookin

Re: [DISCUSS] Describing REST Server capabilities

2024-07-01 Thread Jack Ye
new versions that are still in development, it's possibly the >>> easiest to not do anything special, not even announce the new version. >>> Whether we indicate some "beta version" or reserve "Integer.MAX_VALUE" >>> doesn't really help anyways - client

Re: [Discussion] Apache Iceberg Community Guideline - Initial Version

2024-07-01 Thread Jack Ye
Hi everyone, Thanks for all the comments and feedback on the document, I am working with a few commenters on some additional changes and wording, and then will carry out the vote. Best, Jack Ye On Thu, Jun 27, 2024 at 11:02 AM Jack Ye wrote: > To provide an update here, I have consolida

Re: [Discussion] Apache Iceberg Community Guideline - Initial Version

2024-06-27 Thread Jack Ye
issues. Let me know if there is anything else we see major disagreements with, and I will organize a vote after 24 hours. Best, Jack Ye On Wed, Jun 26, 2024 at 11:04 AM Jack Ye wrote: > +1 for adding to the site. > > I am putting it as a doc for now since Google doc is easier to comment (I >

Re: [DISCUSS] Describing REST Server capabilities

2024-06-27 Thread Jack Ye
2024 at 11:34 AM Jean-Baptiste Onofré > wrote: > >> Hi Jack >> >> I like Robert's proposal. Back to the topics, I think grouping with >> tags is more "flexible" (it was what we included in the REST spec >> proposal as well). >> >> Reg

Re: [DISCUSS] Describing REST Server capabilities

2024-06-26 Thread Jack Ye
a broader set of users have > a clear idea of what's supported and can make good decisions for themselves > since everyone is speaking the same standard language. > > Furthermore, we could also look at if it makes sense to make the tags more > granular if the scenario described is

Re: [Discussion] Apache Iceberg Community Guideline - Initial Version

2024-06-26 Thread Jack Ye
version first, and then we just go through all the identified topics one by one. I have a list of all topics in the original feedback collection devlist thread. Let me know what you think about these plans! Best, Jack Ye On Wed, Jun 26, 2024 at 9:04 AM Ryan Blue wrote: > +1 for add

Re: [DISCUSS] Describing REST Server capabilities

2024-06-26 Thread Jack Ye
. Best, Jack Ye On Wed, Jun 26, 2024 at 9:02 AM Daniel Weeks wrote: > I think Robert's approach is a reasonable compromise here. > > If we wanted a "per operation/endpoint" versioning, I think I'd prefer > Micah's OpenAPI spec based approach because it's more standardi

Re: Iceberg Catalog Syncs Invite

2024-06-26 Thread Jack Ye
Onofré > wrote: > >> Hi Jack >> >> Thanks ! >> Just a reminder: we have to provide regular updates on the dev mailing >> list (as we say at Apache: if it doesn't happen on the mailing list, >> it never happened). >> >> Regards >> JB >> &g

Re: Iceberg Catalog Syncs Invite

2024-06-25 Thread Jack Ye
The google group is purely for subscribing to the meeting. We do not plan to use it for discussions. -Jack On Tue, Jun 25, 2024, 10:33 PM Xuanwo wrote: > Hi, Jack Ye > > Thank you for setting up the Iceberg Catalog Sync; I'm eager to > participate. > > However, it feels

Iceberg Catalog Syncs Invite

2024-06-25 Thread Jack Ye
in the sync meetings! Best, Jack Ye

Re: [DISCUSS] Describing REST Server capabilities

2024-06-25 Thread Jack Ye
bility, I think not returning anything should mean that all API that the client understands are having version *. What do people think of it, compared to the tag approach? Best, Jack Ye On Mon, Jun 24, 2024 at 1:42 PM Micah Kornfield wrote: > I don't have strong opinions eith

[Discussion] Apache Iceberg Guidelines for Committership and PMC Membership

2024-06-25 Thread Jack Ye
, Jack Ye

Re: Feedback Collection: Bylaws in Iceberg

2024-06-25 Thread Jack Ye
these topics one by one. Feel free to keep providing comments or feedback in this thread, and please let me know any additional topics I did not cover in the list above! Best, Jack Ye On Tue, Jun 25, 2024 at 8:54 AM Honah J. wrote: > Hi everyone, > > Thanks to everyone for the valuab

[Discussion] Apache Iceberg Community Guideline - Initial Version

2024-06-25 Thread Jack Ye
hing is probably to agree upon the 2/3 majority approval for modifying the project guidelines, so we can have a consistent voting method going forward. This initial introduction of the bylaws will be voted using consensus approval. Please take a look and comment about any additional changes needed, and I will host a vote in 3 days. Best, Jack Ye

Re: Feedback Collection: Bylaws in Iceberg

2024-06-24 Thread Jack Ye
ote that >> I'm one of the individuals here. >> >> Respectfully, what does this mean, Ryan? No individual was even singled >> out here. This comes off as stifling discussion, not cool... >> >> On Mon, Jun 24, 2024, 9:08 AM Jack Ye wrote: >> >>> Sorry for

Re: Feedback Collection: Bylaws in Iceberg

2024-06-24 Thread Jack Ye
ibutions are not >>> making the expected progress. It might be helpful to create bylaws or >>> procedures to ensure these proposals and PRs receive the necessary >>> attention and are addressed promptly. This could involve setting timeframes >>> for reviews or establishin

Re: Feedback Collection: Bylaws in Iceberg

2024-06-24 Thread Jack Ye
tion" in Apache is vague, and I don't think there is a requirement to say a proposal is a part of that and stick to the vague standard process. Maybe we should re-discuss this and formally conclude a direction with everyone. Best, Jack Ye On Mon, Jun 24, 2024 at 6:19 AM Renjie Liu wrote: >

Feedback Collection: Bylaws in Iceberg

2024-06-24 Thread Jack Ye
this with everyone! Best, Jack Ye

Support Securable Objects in Iceberg REST Catalog

2024-05-30 Thread Jack Ye
to implement for managing access to objects in IRC, and (2) users that are on one IRC product do not need to be locked-in due to access management aspects. Would really appreciate any feedback on this topic and proposal! Best, Jack Ye

Re: Addressing security questions in the Iceberg REST specification

2024-05-29 Thread Jack Ye
o that based on security concerns. Best, Jack Ye On Wed, May 29, 2024 at 10:28 AM Steven Wu wrote: > Wondering if the auth endpoints can be separated out to a separate OpenAPI > spec file. Then we still have some reference for interactions with auth > server and make it clear it is

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

2024-05-28 Thread Jack Ye
gt; will be similar to managing views. >>> >>> Thanks, Ryan, Robert, and Jack, for your input. >>> We will work on publishing the draft spec (inspired by the view spec) >>> this week to facilitate further discussions. >>> >>> - Ajantha >>>

Re: Addressing security questions in the Iceberg REST specification

2024-05-28 Thread Jack Ye
cloud providers next to AWS. > > Kind regards, > Fokko > > > > Op do 23 mei 2024 om 15:49 schreef Dmitri Bourlatchkov > : > >> I think Jack makes a good point with AWS SigV4 Authentication. I suppose, >> in REST Catalog implementations that support that aut

Re: Addressing security questions in the Iceberg REST specification

2024-05-28 Thread Jack Ye
lementations that support that auth method, the > /v1/oauth/token Catalog REST endpoint is redundant. > > Cheers, > Dmitri. > > On Thu, May 23, 2024 at 9:20 AM Jack Ye wrote: > >> I do not know enough details about OAuth to make comments about this >> issue, but jus

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

2024-05-28 Thread Jack Ye
when we have the actual proposal published. Best, Jack Ye On Tue, May 28, 2024 at 1:32 AM Robert Stupp wrote: > UDFs are as engine specific and portable and "non-centralized" as views > are. The same performance concerns apply to views as well. > Iceberg should define a common b

Re: Addressing security questions in the Iceberg REST specification

2024-05-23 Thread Jack Ye
erg/blob/main/core/src/main/java/org/apache/iceberg/rest/HTTPClient.java#L72>. It would be nice if we formalize that in the spec, at least define it as a generic authorization header. Best, Jack Ye On Thu, May 23, 2024 at 2:51 AM Robert Stupp wrote: > Hi all, > > Iceberg REST implement

Re: Proposal for REST APIs for Iceberg table scans

2024-05-20 Thread Jack Ye
should see more progress pretty soon! Best, Jack Ye On Sat, May 18, 2024 at 4:05 PM Pucheng Yang wrote: > Hi all, I wonder if we have a ETA for this change? thanks > > On Wed, Jan 31, 2024 at 10:30 AM Chertara, Rahil > wrote: > >> Sure, I can look into adding this t

Re: Pagination for List APIs in the REST spec

2024-05-20 Thread Jack Ye
I believe this is already merged? https://github.com/apache/iceberg/pull/9782 Best, Jack Ye On Sat, May 18, 2024 at 4:06 PM Pucheng Yang wrote: > Hi all, is there an ETA for this? thanks > > On Wed, Dec 20, 2023 at 6:03 PM Renjie Liu > wrote: > >> I think if servers provi

Re: Materialized Views: Next Steps

2024-05-20 Thread Jack Ye
nformation is already encoded in the query and >>>> there is no need to capture it externally. >>>> >>>> For example, if the MV definition consists of table references where >>>> all of the references are bound to specific snapshot IDs or timestam

Re: [Early Feedback] Variant and Subcolumnarization Support

2024-05-16 Thread Jack Ye
forward to the proposal! Best, Jack Ye On Wed, May 15, 2024 at 9:37 AM Tyler Akidau wrote: > On Tue, May 14, 2024 at 7:58 PM Gang Wu wrote: > >> > We may need some guidance on just how many we need to look at; >> > we were planning on Spark and Trino, but weren't sure ho

Re: New committer: Renjie Liu

2024-03-11 Thread Jack Ye
Congratulations Renjie! Best, Jack Ye On Mon, Mar 11, 2024, 8:24 AM Ryan Blue wrote: > Congratulations, Renjie! Thanks for all your contributions! > > On Mon, Mar 11, 2024 at 12:52 AM Eduard Tudenhoefner > wrote: > >> Congrats Renjie! >> >> On Mon, Mar 11

Re: New committer: Bryan Keller

2024-03-05 Thread Jack Ye
Congrats Bryan! -Jack On Tue, Mar 5, 2024 at 7:33 AM Amogh Jahagirdar wrote: > Congratulations Bryan! Very well deserved, thank you for all your > contributions! > > On Tue, Mar 5, 2024 at 7:29 AM Steven Wu wrote: > >> Bryan, congratulations and thank you for your many contributions. >> >> On

Re: Materialized view integration with REST spec

2024-03-04 Thread Jack Ye
.apache.org/community/#iceberg-community-events. Might > help to fix it or share the meeting link here. > > On Fri, Mar 1, 2024 at 3:43 PM Jack Ye wrote: > >> Sounds good, let's discuss this in person! >> >> I am a bit worried that we have quite a few critical top

Re: Gravitino an Iceberg REST catalog service

2024-03-01 Thread Jack Ye
erg >> users. (Blue) >> > - Simplify plugging in your own catalog so the Iceberg project isn’t >> responsible for maintaining and testing a bunch of dialects. (Blue). >> > - Aim for a REST catalog centric future and continue to remove Iceberg >> support where it m

Re: Materialized view integration with REST spec

2024-03-01 Thread Jack Ye
it. Best, Jack Ye On Fri, Mar 1, 2024 at 12:48 PM Ryan Blue wrote: > Hey everyone, > > I think this thread has hit a point of diminishing returns and that we > still don't have a common understanding of what the options under > consideration actually are. > > Since we

Proposal for Iceberg Spec Extensions for Data Access Decision Exchange

2024-03-01 Thread Jack Ye
add support to enforce those access decisions. This would hopefully provide a good data governance story for enterprise Iceberg users. Would greatly appreciate any feedback! Best, Jack Ye

Re: Materialized view integration with REST spec

2024-03-01 Thread Jack Ye
t; I'm still strongly in favor of Option 1 (separate table and view) for > these reasons. > > -Dan > > > > On Thu, Feb 29, 2024 at 11:07 PM Jack Ye wrote: > >> > Jack, it sounds like you’re the proponent of a combined table and view >> (rather than a ne

Re: Materialized view integration with REST spec

2024-02-29 Thread Jack Ye
berg spec. Many mechanisms need to be built in a catalog to hide, protect, maintain, recycle the storage table, that can be avoided by using other approaches. I think we should reach a consensus about that and discuss further if you do not agree. Best, Jack Ye On Thu, Feb 29, 2024 at 10:53 PM

Re: Inconsistency between REST spec and table/view spec

2024-02-29 Thread Jack Ye
t;> interoperability. >>> >>> My main point is that there isn't an inherent incompatibility between >>> the REST spec and the Iceberg spec. The preservation of the storage >>> representation was discussed and intentional during the design/development >>&

Re: Inconsistency between REST spec and table/view spec

2024-02-29 Thread Jack Ye
ake sense. Putting it in a key-value store or >> RDMS could be a better option. >> >> Given that we are going to remove the storage requirement. Should we >> avoid the file path in the current design for things like view spec? A >> solution like table identifier + versi

Re: Inconsistency between REST spec and table/view spec

2024-02-29 Thread Jack Ye
quirements, it's not compliant with the spec. There's no exemption that >> says if you're using REST you don't need to follow the spec. Why do you >> think that's the case? >> >> I don't believe there's a reason to say that the REST spec needs to >>

Re: Inconsistency between REST spec and table/view spec

2024-02-29 Thread Jack Ye
could bifurcate access to Iceberg data. There are also aspects to the spec >> that reference the metadata paths (like metadata log, though it's >> optional), but would likely need to be addressed. >> >> -Dan >> >> >> >> On Thu, Feb 29, 2024

Inconsistency between REST spec and table/view spec

2024-02-29 Thread Jack Ye
same approach can be used to talk to HMS/Glue/JDBC/... while users will only interact with the RESTCatalog as the entry point. I think this can provide a good path forward overall for the catalog consolidation story, interested to know what others think. Best, Jack Ye

Re: Materialized view integration with REST spec

2024-02-28 Thread Jack Ye
ll the arguments you listed for the new metadata type still hold true, and it also "reuses existing metadata definitions" and can "fall back to simple views". -Jack On Wed, Feb 28, 2024 at 5:05 PM Jack Ye wrote: > Thanks Ryan for the help to trace back to the root question! Ju

Re: Materialized view integration with REST spec

2024-02-28 Thread Jack Ye
Thanks Ryan for the help to trace back to the root question! Just a clarification question regarding your reply before I reply further: what exactly does the option "a combination of the two (i.e. commits are combined)" mean? How is that different from "a new metadata type"? -Jack On Wed, Feb

Re: Deprecate DynamodbCatalog

2024-02-27 Thread Jack Ye
t;> +1 for deprecation and removal by 2.0 version. >>>> >>>> - Ajantha >>>> >>>> On Sat, Feb 17, 2024 at 4:23 AM Daniel Weeks wrote: >>>> >>>>> +1 as well for deprecation >>>>> >>>>> On Fri, Fe

Re: Proposal for RESTful Data Operations

2024-02-26 Thread Jack Ye
e to find out what you’d change or what > specific features of an LSM approach you’re looking for that we can’t do > today. Maybe we should set up a time to talk with a group that wants to > work on this area? > > Ryan > > On Wed, Feb 21, 2024 at 3:55 PM Jack Ye wrote: > >&

Re: Support permission concepts in REST spec

2024-02-26 Thread Jack Ye
l if the chain > of trust includes the ability to enforce constraints. Plus, constraints may > need to be known during job execution, not just at commit time, so it is > better to send them when loading a table. > > Ryan > > On Tue, Feb 20, 2024 at 1:44 PM Jack Ye wrot

Re: [VOTE] Release Apache Iceberg 1.5.0 RC3

2024-02-23 Thread Jack Ye
>> catalog. >> >> Thanks, >> >> Amogh Jahagirdar >> >> On Thu, Feb 22, 2024 at 9:10 PM Jack Ye wrote: >> >>> +1 (binding) >>> >>> Checked license, signature, checksum, build, test with Java17 >>> Ran manual test

Re: [VOTE] Release Apache Iceberg 1.5.0 RC3

2024-02-22 Thread Jack Ye
+1 (binding) Checked license, signature, checksum, build, test with Java17 Ran manual test with EMR 7.0 Spark 3.5 and Glue. Best, Jack Ye On Thu, Feb 22, 2024 at 7:58 PM Daniel Weeks wrote: > +1 (binding) > > Verified sigs/sums/license/build/test (Java 17) > > I also did manu

Re: Materialized view integration with REST spec

2024-02-21 Thread Jack Ye
Thanks everyone for the help in organizing the thoughts! I have moved the summary of everyone's comments here also to the doc that Jan linked under question 0. We can continue to have more discussions there and cast votes! Best, Jack Ye On Wed, Feb 21, 2024 at 12:14 PM Jan Kaul wrote

Re: [DISCUSS] spec: remove the file scan task JSON serialization section from table spec

2024-02-21 Thread Jack Ye
ies on the format to evolve in >> compatible ways across versions. I think that means that we don't make any >> guarantees about how it evolves and it can be safely removed since it is >> not a contract that we are committed to maintaining. >> >> Ryan >> >&

Re: Proposal for RESTful Data Operations

2024-02-21 Thread Jack Ye
simpler, but that’s not enough (yet) to convince > me that it is a good idea. > > I think there’s a strong case for append, but for deletes I don’t think we > need or want to go there. I certainly would not want to add delete > endpoints that would be misused. > > Ryan &g

Re: [DISCUSS] spec: remove the file scan task JSON serialization section from table spec

2024-02-21 Thread Jack Ye
Was there any prior discussions on devlist for adding it to the spec? Could you help link those conversations? Thanks, Jack Ye On Wed, Feb 21, 2024 at 1:05 PM Steven Wu wrote: > > In the recent PR review [1], Ryan and emkornfield has raised a question > why file scan task JSON seri

Re: Proposal for RESTful Data Operations

2024-02-20 Thread Jack Ye
have a chance to review, I would appreciate any additional feedback >> you may have. >> >> https://github.com/apache/iceberg/pull/9292 >> >> Best, >> >> Drew >> >> On Fri, Jan 12, 2024 at 3:40 PM Drew wrote: >> >>> Hi everyone,

Re: Support permission concepts in REST spec

2024-02-20 Thread Jack Ye
re detailed doc for us to review. Best, Jack Ye On Fri, Feb 16, 2024 at 10:29 AM Micah Kornfield wrote: > Hi Jack, > I think this is an interesting idea but I think there are some practical > concerns (I posted them inline). > > - general access patterns, like read-only,

Re: Table Schema History Pruning

2024-02-20 Thread Jack Ye
metadata size. +1 for reopening the issue to discuss further. We should probably also make the title more specific than "The metadata file is too large". Best, Jack Ye On Tue, Feb 20, 2024 at 9:55 AM Sung Yun (BLOOMBERG/ 120 PARK) < syu...@bloomberg.net> wrote: > Hi Ba

Re: Table Portability Proposal

2024-02-20 Thread Jack Ye
Formation. Has this option been considered? I quickly scanned through the linked doc, it seems to be not discussed, but I might have missed it. Best, Jack Ye On Tue, Feb 20, 2024 at 9:21 AM Jean-Baptiste Onofré wrote: > Hi Ryan > > Ah ok, I thought that an Iceberg rele

Re: Materialized view integration with REST spec

2024-02-20 Thread Jack Ye
a single thread instead of multiple threads going on at the same time. If we think this format is not effective, I propose that we create a new mv channel in Iceberg Slack workspace, and people interested can join and discuss all these points directly. What do we think? Best, Jack Ye On Mon, Feb 19

Re: [ANNOUNCE] Release Apache Iceberg Rust 0.2.0

2024-02-20 Thread Jack Ye
Congratulations on the first release! -Jack On Tue, Feb 20, 2024 at 2:32 AM Driesprong, Fokko wrote: > Hi all, > > The Apache Iceberg Rust community is pleased to announce that Apache > Iceberg Rust 0.2.0 has been released! > > Iceberg is a data access layer that allows users to easily and

Re: Materialized view integration with REST spec

2024-02-19 Thread Jack Ye
der directly describing the full metadata of the storage table >>> in Iceberg view, instead of pointing to a JSON file. >> >> >> Do you mean we need to add components like >> `LoadMaterializedViewResponse`, if so, I would +1 for this. >> >> *Q2: what RES

Re: [VOTE] Release Apache Iceberg Rust 0.2.0 RC1

2024-02-19 Thread Jack Ye
+1 (binding) Verified checksum, signature, license, note, ASF header Ran build and test Checked no unexpected binary files Best, Jack Ye On Mon, Feb 19, 2024 at 2:33 AM Jean-Baptiste Onofré wrote: > +1 (non binding) > > I checked: > - checksum and signature are correct >

Re: [VOTE] Release Apache PyIceberg 0.6.0rc6

2024-02-19 Thread Jack Ye
+1 (binding) Checked signature, checksum, license Ran unit and integ tests with Python 3.11 Ran manual tests with Glue catalog Best, Jack Ye On Mon, Feb 19, 2024 at 4:35 AM Fokko Driesprong wrote: > +1 (binding) > > I've checked signatures and checksums, checked the licenses, and

  1   2   3   4   >