Re: Shall we start a regular community sync up?
Same - would love to attend... On Tue, Dec 1, 2020 at 2:46 PM Russell Spitzer wrote: > Sent > > On Tue, Dec 1, 2020 at 1:43 PM Chen Song wrote: > >> Hey Ryan >> >> Could you please send me an invite list as well. We have been evaluating >> Iceberg strategically in our company and very interested in this. >> >> Best, >> Chen >> >> On Thu, Nov 19, 2020 at 7:51 AM Vivekanand Vellanki >> wrote: >> >>> Hi Ryan, >>> >>> I'd like to attend the regular community syncs. Can you send me an >>> invite? >>> >>> Thanks >>> Vivek >>> >>> On Mon, Jun 15, 2020 at 11:16 PM Edgar Rodriguez >>> wrote: >>> Hi Ryan, I'd like to attend the regular community syncs, could you send me an invite? Thanks! - Edgar On Wed, Mar 25, 2020 at 6:38 PM Ryan Blue wrote: > Will do. > > On Wed, Mar 25, 2020 at 6:36 PM Jun Ma wrote: > >> Hi Ryan, >> >> Thanks for driving the sync up meeting. Could you please add Fan Diao( >> fan.dia...@gmail.com) and myself to the invitation list? >> >> Thanks, >> Jun Ma >> >> On Mon, Mar 23, 2020 at 9:57 PM OpenInx wrote: >> >>> Hi Ryan >>> >>> I received your invitation. Some guys from our Flink teams also want >>> to join the hangouts meeting. Do we need >>> also send an extra invitation to them ? Or could them just join the >>> meeting with entering the meeting address[1] ? >>> >>> If need so, please let the following guys in: >>> 1. ykt...@gmail.com >>> 2. imj...@gmail.com >>> 3. yuzhao@gmail.com >>> >>> BTW, I've written a draft to discuss in the meeting [2], anyone >>> could enrich the topics want to discuss. >>> >>> Thanks. >>> >>> [1]. https://meet.google.com/_meet/xdx-rknm-uvm >>> [2]. >>> https://docs.google.com/document/d/1wXTHGYhc7sDhP5DxlByba0S5YguNLWwY98FAp6Tx2mw/edit# >>> >>> On Mon, Mar 23, 2020 at 5:35 AM Ryan Blue >>> wrote: >>> I invited everyone that replied to this thread and the people that were on the last invite. If you have specific topics you'd like to put on the agenda, please send them to me! On Sun, Mar 22, 2020 at 2:28 PM Ryan Blue wrote: > Let's go with Wednesday. I'll send out an invite. > > On Sun, Mar 22, 2020 at 1:36 PM John Zhuge > wrote: > >> 5-5:30 pm work for me. Prefer Wednesdays. >> >> On Sun, Mar 22, 2020 at 1:33 PM Romin Parekh < >> rominpar...@gmail.com> wrote: >> >>> Hi folks, >>> >>> Both times slots work for me next week. Can we confirm a day? >>> >>> Thanks, >>> Romin >>> >>> Sent from my iPhone >>> >>> > On Mar 20, 2020, at 11:38 PM, Jun H. >>> wrote: >>> > >>> > The schedule works for me. >>> > >>> >> On Thu, Mar 19, 2020 at 6:55 PM Junjie Chen < >>> chenjunjied...@gmail.com> wrote: >>> >> >>> >> The same time works for me as well. >>> >> >>> >>> On Fri, Mar 20, 2020 at 9:43 AM Gautam < >>> gautamkows...@gmail.com> wrote: >>> >>> >>> >>> 5 / 5:30pm any day of next week works for me. >>> >>> >>> >>> On Thu, Mar 19, 2020 at 6:07 PM 李响 >>> wrote: >>> >>> 5 or 5:30 PM (UTC-7, is it PDT now) in any day works for >>> me. Looking forward to it 8-) >>> >>> On Fri, Mar 20, 2020 at 8:17 AM RD >>> wrote: >>> > >>> > Same time works for me too! >>> > >>> > On Thu, Mar 19, 2020 at 4:45 PM Xabriel Collazo Mojica >>> wrote: >>> >> >>> >> 5pm or 5:30pm PT any day next week would work for me. >>> >> >>> >> Thanks for restoring the community sync up! >>> >> >>> >> Xabriel J Collazo Mojica | Sr Computer Scientist II | >>> Adobe >>> >> >>> >> On 3/18/20, 6:45 PM, "justin_cof...@apple.com on behalf >>> of Justin Q Coffey" >> j...@apple.com.INVALID> wrote: >>> >> >>> >>Any chance we could actually do 5:30pm PST? I'm a bit >>> of a lurker, but this roadmap is important to mine and I have a >>> daily at >>> 5pm :(. >>> >> >>> >>-Justin >>> >> >>> >>> On Mar 18, 2020, at 6:43 PM, Saisai Shao < >>> sai.sai.s...@gmail.com> wrote: >>> >>> >>> >>> 5pm PST in any day works for me. >>> >>> >>> >>> Looking forward to it. >>> >>> >>> >>> Thanks >>> >>> Saisai >>> >> >>> >> >>> >> >>> >>> >>>
Re: Shall we start a regular community sync up?
Sent On Tue, Dec 1, 2020 at 1:43 PM Chen Song wrote: > Hey Ryan > > Could you please send me an invite list as well. We have been evaluating > Iceberg strategically in our company and very interested in this. > > Best, > Chen > > On Thu, Nov 19, 2020 at 7:51 AM Vivekanand Vellanki > wrote: > >> Hi Ryan, >> >> I'd like to attend the regular community syncs. Can you send me an invite? >> >> Thanks >> Vivek >> >> On Mon, Jun 15, 2020 at 11:16 PM Edgar Rodriguez >> wrote: >> >>> Hi Ryan, >>> >>> I'd like to attend the regular community syncs, could you send me >>> an invite? >>> >>> Thanks! >>> >>> - Edgar >>> >>> On Wed, Mar 25, 2020 at 6:38 PM Ryan Blue >>> wrote: >>> Will do. On Wed, Mar 25, 2020 at 6:36 PM Jun Ma wrote: > Hi Ryan, > > Thanks for driving the sync up meeting. Could you please add Fan Diao( > fan.dia...@gmail.com) and myself to the invitation list? > > Thanks, > Jun Ma > > On Mon, Mar 23, 2020 at 9:57 PM OpenInx wrote: > >> Hi Ryan >> >> I received your invitation. Some guys from our Flink teams also want >> to join the hangouts meeting. Do we need >> also send an extra invitation to them ? Or could them just join the >> meeting with entering the meeting address[1] ? >> >> If need so, please let the following guys in: >> 1. ykt...@gmail.com >> 2. imj...@gmail.com >> 3. yuzhao@gmail.com >> >> BTW, I've written a draft to discuss in the meeting [2], anyone >> could enrich the topics want to discuss. >> >> Thanks. >> >> [1]. https://meet.google.com/_meet/xdx-rknm-uvm >> [2]. >> https://docs.google.com/document/d/1wXTHGYhc7sDhP5DxlByba0S5YguNLWwY98FAp6Tx2mw/edit# >> >> On Mon, Mar 23, 2020 at 5:35 AM Ryan Blue >> wrote: >> >>> I invited everyone that replied to this thread and the people that >>> were on the last invite. >>> >>> If you have specific topics you'd like to put on the agenda, please >>> send them to me! >>> >>> On Sun, Mar 22, 2020 at 2:28 PM Ryan Blue wrote: >>> Let's go with Wednesday. I'll send out an invite. On Sun, Mar 22, 2020 at 1:36 PM John Zhuge wrote: > 5-5:30 pm work for me. Prefer Wednesdays. > > On Sun, Mar 22, 2020 at 1:33 PM Romin Parekh < > rominpar...@gmail.com> wrote: > >> Hi folks, >> >> Both times slots work for me next week. Can we confirm a day? >> >> Thanks, >> Romin >> >> Sent from my iPhone >> >> > On Mar 20, 2020, at 11:38 PM, Jun H. >> wrote: >> > >> > The schedule works for me. >> > >> >> On Thu, Mar 19, 2020 at 6:55 PM Junjie Chen < >> chenjunjied...@gmail.com> wrote: >> >> >> >> The same time works for me as well. >> >> >> >>> On Fri, Mar 20, 2020 at 9:43 AM Gautam < >> gautamkows...@gmail.com> wrote: >> >>> >> >>> 5 / 5:30pm any day of next week works for me. >> >>> >> >>> On Thu, Mar 19, 2020 at 6:07 PM 李响 wrote: >> >> 5 or 5:30 PM (UTC-7, is it PDT now) in any day works for me. >> Looking forward to it 8-) >> >> On Fri, Mar 20, 2020 at 8:17 AM RD >> wrote: >> > >> > Same time works for me too! >> > >> > On Thu, Mar 19, 2020 at 4:45 PM Xabriel Collazo Mojica >> wrote: >> >> >> >> 5pm or 5:30pm PT any day next week would work for me. >> >> >> >> Thanks for restoring the community sync up! >> >> >> >> Xabriel J Collazo Mojica | Sr Computer Scientist II | >> Adobe >> >> >> >> On 3/18/20, 6:45 PM, "justin_cof...@apple.com on behalf >> of Justin Q Coffey" > j...@apple.com.INVALID> wrote: >> >> >> >>Any chance we could actually do 5:30pm PST? I'm a bit >> of a lurker, but this roadmap is important to mine and I have a >> daily at >> 5pm :(. >> >> >> >>-Justin >> >> >> >>> On Mar 18, 2020, at 6:43 PM, Saisai Shao < >> sai.sai.s...@gmail.com> wrote: >> >>> >> >>> 5pm PST in any day works for me. >> >>> >> >>> Looking forward to it. >> >>> >> >>> Thanks >> >>> Saisai >> >> >> >> >> >> >> >> >> -- >> >> 李响 Xiang Li >> >> 手机 cellphone ：+86-136-8113-8972 >> 邮件 e-mail ：wate...@gmail.com >> >> >> >> >> >> >> >> -- >>>
Re: Shall we start a regular community sync up?
Hey Ryan Could you please send me an invite list as well. We have been evaluating Iceberg strategically in our company and very interested in this. Best, Chen On Thu, Nov 19, 2020 at 7:51 AM Vivekanand Vellanki wrote: > Hi Ryan, > > I'd like to attend the regular community syncs. Can you send me an invite? > > Thanks > Vivek > > On Mon, Jun 15, 2020 at 11:16 PM Edgar Rodriguez > wrote: > >> Hi Ryan, >> >> I'd like to attend the regular community syncs, could you send me >> an invite? >> >> Thanks! >> >> - Edgar >> >> On Wed, Mar 25, 2020 at 6:38 PM Ryan Blue >> wrote: >> >>> Will do. >>> >>> On Wed, Mar 25, 2020 at 6:36 PM Jun Ma wrote: >>> Hi Ryan, Thanks for driving the sync up meeting. Could you please add Fan Diao( fan.dia...@gmail.com) and myself to the invitation list? Thanks, Jun Ma On Mon, Mar 23, 2020 at 9:57 PM OpenInx wrote: > Hi Ryan > > I received your invitation. Some guys from our Flink teams also want > to join the hangouts meeting. Do we need > also send an extra invitation to them ? Or could them just join the > meeting with entering the meeting address[1] ? > > If need so, please let the following guys in: > 1. ykt...@gmail.com > 2. imj...@gmail.com > 3. yuzhao@gmail.com > > BTW, I've written a draft to discuss in the meeting [2], anyone > could enrich the topics want to discuss. > > Thanks. > > [1]. https://meet.google.com/_meet/xdx-rknm-uvm > [2]. > https://docs.google.com/document/d/1wXTHGYhc7sDhP5DxlByba0S5YguNLWwY98FAp6Tx2mw/edit# > > On Mon, Mar 23, 2020 at 5:35 AM Ryan Blue > wrote: > >> I invited everyone that replied to this thread and the people that >> were on the last invite. >> >> If you have specific topics you'd like to put on the agenda, please >> send them to me! >> >> On Sun, Mar 22, 2020 at 2:28 PM Ryan Blue wrote: >> >>> Let's go with Wednesday. I'll send out an invite. >>> >>> On Sun, Mar 22, 2020 at 1:36 PM John Zhuge >>> wrote: >>> 5-5:30 pm work for me. Prefer Wednesdays. On Sun, Mar 22, 2020 at 1:33 PM Romin Parekh wrote: > Hi folks, > > Both times slots work for me next week. Can we confirm a day? > > Thanks, > Romin > > Sent from my iPhone > > > On Mar 20, 2020, at 11:38 PM, Jun H. wrote: > > > > The schedule works for me. > > > >> On Thu, Mar 19, 2020 at 6:55 PM Junjie Chen < > chenjunjied...@gmail.com> wrote: > >> > >> The same time works for me as well. > >> > >>> On Fri, Mar 20, 2020 at 9:43 AM Gautam < > gautamkows...@gmail.com> wrote: > >>> > >>> 5 / 5:30pm any day of next week works for me. > >>> > >>> On Thu, Mar 19, 2020 at 6:07 PM 李响 wrote: > > 5 or 5:30 PM (UTC-7, is it PDT now) in any day works for me. > Looking forward to it 8-) > > On Fri, Mar 20, 2020 at 8:17 AM RD wrote: > > > > Same time works for me too! > > > > On Thu, Mar 19, 2020 at 4:45 PM Xabriel Collazo Mojica > wrote: > >> > >> 5pm or 5:30pm PT any day next week would work for me. > >> > >> Thanks for restoring the community sync up! > >> > >> Xabriel J Collazo Mojica | Sr Computer Scientist II | > Adobe > >> > >> On 3/18/20, 6:45 PM, "justin_cof...@apple.com on behalf > of Justin Q Coffey" j...@apple.com.INVALID> wrote: > >> > >>Any chance we could actually do 5:30pm PST? I'm a bit > of a lurker, but this roadmap is important to mine and I have a daily > at > 5pm :(. > >> > >>-Justin > >> > >>> On Mar 18, 2020, at 6:43 PM, Saisai Shao < > sai.sai.s...@gmail.com> wrote: > >>> > >>> 5pm PST in any day works for me. > >>> > >>> Looking forward to it. > >>> > >>> Thanks > >>> Saisai > >> > >> > >> > > > -- > > 李响 Xiang Li > > 手机 cellphone ：+86-136-8113-8972 > 邮件 e-mail ：wate...@gmail.com > >> > >> > >> > >> -- > >> Best Regards > -- John Zhuge >>> >>> >>> -- >>> Ryan Blue >>> Software Engineer >>> Netflix >>> >> >> >> -- >> Ryan Blue >> Software Engineer >> Netflix >>
Re: Shall we start a regular community sync up?
Invite sent On Tue, Dec 1, 2020 at 1:20 PM Wing Yew Poon wrote: > I'd like to attend the community syncs as well. Can you please send me an > invite? > Thanks, > Wing Yew Poon > > On Thu, Nov 19, 2020 at 9:25 PM Chitresh Kakwani < > chitreshkakw...@gmail.com> wrote: > >> Hi Ryan, >> >> Could you please add me to the invitation list as well ? New entrant. >> Interested in Iceberg's roadmap. >> >> Regards, >> Chitresh Kakwani >> >> On Thu, Nov 19, 2020 at 6:21 PM Vivekanand Vellanki >> wrote: >> >>> Hi Ryan, >>> >>> I'd like to attend the regular community syncs. Can you send me an >>> invite? >>> >>> Thanks >>> Vivek >>> >>> On Mon, Jun 15, 2020 at 11:16 PM Edgar Rodriguez >>> wrote: >>> Hi Ryan, I'd like to attend the regular community syncs, could you send me an invite? Thanks! - Edgar On Wed, Mar 25, 2020 at 6:38 PM Ryan Blue wrote: > Will do. > > On Wed, Mar 25, 2020 at 6:36 PM Jun Ma wrote: > >> Hi Ryan, >> >> Thanks for driving the sync up meeting. Could you please add Fan Diao( >> fan.dia...@gmail.com) and myself to the invitation list? >> >> Thanks, >> Jun Ma >> >> On Mon, Mar 23, 2020 at 9:57 PM OpenInx wrote: >> >>> Hi Ryan >>> >>> I received your invitation. Some guys from our Flink teams also want >>> to join the hangouts meeting. Do we need >>> also send an extra invitation to them ? Or could them just join the >>> meeting with entering the meeting address[1] ? >>> >>> If need so, please let the following guys in: >>> 1. ykt...@gmail.com >>> 2. imj...@gmail.com >>> 3. yuzhao@gmail.com >>> >>> BTW, I've written a draft to discuss in the meeting [2], anyone >>> could enrich the topics want to discuss. >>> >>> Thanks. >>> >>> [1]. https://meet.google.com/_meet/xdx-rknm-uvm >>> [2]. >>> https://docs.google.com/document/d/1wXTHGYhc7sDhP5DxlByba0S5YguNLWwY98FAp6Tx2mw/edit# >>> >>> On Mon, Mar 23, 2020 at 5:35 AM Ryan Blue >>> wrote: >>> I invited everyone that replied to this thread and the people that were on the last invite. If you have specific topics you'd like to put on the agenda, please send them to me! On Sun, Mar 22, 2020 at 2:28 PM Ryan Blue wrote: > Let's go with Wednesday. I'll send out an invite. > > On Sun, Mar 22, 2020 at 1:36 PM John Zhuge > wrote: > >> 5-5:30 pm work for me. Prefer Wednesdays. >> >> On Sun, Mar 22, 2020 at 1:33 PM Romin Parekh < >> rominpar...@gmail.com> wrote: >> >>> Hi folks, >>> >>> Both times slots work for me next week. Can we confirm a day? >>> >>> Thanks, >>> Romin >>> >>> Sent from my iPhone >>> >>> > On Mar 20, 2020, at 11:38 PM, Jun H. >>> wrote: >>> > >>> > The schedule works for me. >>> > >>> >> On Thu, Mar 19, 2020 at 6:55 PM Junjie Chen < >>> chenjunjied...@gmail.com> wrote: >>> >> >>> >> The same time works for me as well. >>> >> >>> >>> On Fri, Mar 20, 2020 at 9:43 AM Gautam < >>> gautamkows...@gmail.com> wrote: >>> >>> >>> >>> 5 / 5:30pm any day of next week works for me. >>> >>> >>> >>> On Thu, Mar 19, 2020 at 6:07 PM 李响 >>> wrote: >>> >>> 5 or 5:30 PM (UTC-7, is it PDT now) in any day works for >>> me. Looking forward to it 8-) >>> >>> On Fri, Mar 20, 2020 at 8:17 AM RD >>> wrote: >>> > >>> > Same time works for me too! >>> > >>> > On Thu, Mar 19, 2020 at 4:45 PM Xabriel Collazo Mojica >>> wrote: >>> >> >>> >> 5pm or 5:30pm PT any day next week would work for me. >>> >> >>> >> Thanks for restoring the community sync up! >>> >> >>> >> Xabriel J Collazo Mojica | Sr Computer Scientist II | >>> Adobe >>> >> >>> >> On 3/18/20, 6:45 PM, "justin_cof...@apple.com on behalf >>> of Justin Q Coffey" >> j...@apple.com.INVALID> wrote: >>> >> >>> >>Any chance we could actually do 5:30pm PST? I'm a bit >>> of a lurker, but this roadmap is important to mine and I have a >>> daily at >>> 5pm :(. >>> >> >>> >>-Justin >>> >> >>> >>> On Mar 18, 2020, at 6:43 PM, Saisai Shao < >>> sai.sai.s...@gmail.com> wrote: >>> >>> >>> >>> 5pm PST in any day works for me. >>> >>> >>> >>> Looking forward to it. >>> >>> >>> >>> Thanks >>> >>> Saisai >>
Re: Shall we start a regular community sync up?
I'd like to attend the community syncs as well. Can you please send me an invite? Thanks, Wing Yew Poon On Thu, Nov 19, 2020 at 9:25 PM Chitresh Kakwani wrote: > Hi Ryan, > > Could you please add me to the invitation list as well ? New entrant. > Interested in Iceberg's roadmap. > > Regards, > Chitresh Kakwani > > On Thu, Nov 19, 2020 at 6:21 PM Vivekanand Vellanki > wrote: > >> Hi Ryan, >> >> I'd like to attend the regular community syncs. Can you send me an invite? >> >> Thanks >> Vivek >> >> On Mon, Jun 15, 2020 at 11:16 PM Edgar Rodriguez >> wrote: >> >>> Hi Ryan, >>> >>> I'd like to attend the regular community syncs, could you send me >>> an invite? >>> >>> Thanks! >>> >>> - Edgar >>> >>> On Wed, Mar 25, 2020 at 6:38 PM Ryan Blue >>> wrote: >>> Will do. On Wed, Mar 25, 2020 at 6:36 PM Jun Ma wrote: > Hi Ryan, > > Thanks for driving the sync up meeting. Could you please add Fan Diao( > fan.dia...@gmail.com) and myself to the invitation list? > > Thanks, > Jun Ma > > On Mon, Mar 23, 2020 at 9:57 PM OpenInx wrote: > >> Hi Ryan >> >> I received your invitation. Some guys from our Flink teams also want >> to join the hangouts meeting. Do we need >> also send an extra invitation to them ? Or could them just join the >> meeting with entering the meeting address[1] ? >> >> If need so, please let the following guys in: >> 1. ykt...@gmail.com >> 2. imj...@gmail.com >> 3. yuzhao@gmail.com >> >> BTW, I've written a draft to discuss in the meeting [2], anyone >> could enrich the topics want to discuss. >> >> Thanks. >> >> [1]. https://meet.google.com/_meet/xdx-rknm-uvm >> [2]. >> https://docs.google.com/document/d/1wXTHGYhc7sDhP5DxlByba0S5YguNLWwY98FAp6Tx2mw/edit# >> >> On Mon, Mar 23, 2020 at 5:35 AM Ryan Blue >> wrote: >> >>> I invited everyone that replied to this thread and the people that >>> were on the last invite. >>> >>> If you have specific topics you'd like to put on the agenda, please >>> send them to me! >>> >>> On Sun, Mar 22, 2020 at 2:28 PM Ryan Blue wrote: >>> Let's go with Wednesday. I'll send out an invite. On Sun, Mar 22, 2020 at 1:36 PM John Zhuge wrote: > 5-5:30 pm work for me. Prefer Wednesdays. > > On Sun, Mar 22, 2020 at 1:33 PM Romin Parekh < > rominpar...@gmail.com> wrote: > >> Hi folks, >> >> Both times slots work for me next week. Can we confirm a day? >> >> Thanks, >> Romin >> >> Sent from my iPhone >> >> > On Mar 20, 2020, at 11:38 PM, Jun H. >> wrote: >> > >> > The schedule works for me. >> > >> >> On Thu, Mar 19, 2020 at 6:55 PM Junjie Chen < >> chenjunjied...@gmail.com> wrote: >> >> >> >> The same time works for me as well. >> >> >> >>> On Fri, Mar 20, 2020 at 9:43 AM Gautam < >> gautamkows...@gmail.com> wrote: >> >>> >> >>> 5 / 5:30pm any day of next week works for me. >> >>> >> >>> On Thu, Mar 19, 2020 at 6:07 PM 李响 wrote: >> >> 5 or 5:30 PM (UTC-7, is it PDT now) in any day works for me. >> Looking forward to it 8-) >> >> On Fri, Mar 20, 2020 at 8:17 AM RD >> wrote: >> > >> > Same time works for me too! >> > >> > On Thu, Mar 19, 2020 at 4:45 PM Xabriel Collazo Mojica >> wrote: >> >> >> >> 5pm or 5:30pm PT any day next week would work for me. >> >> >> >> Thanks for restoring the community sync up! >> >> >> >> Xabriel J Collazo Mojica | Sr Computer Scientist II | >> Adobe >> >> >> >> On 3/18/20, 6:45 PM, "justin_cof...@apple.com on behalf >> of Justin Q Coffey" > j...@apple.com.INVALID> wrote: >> >> >> >>Any chance we could actually do 5:30pm PST? I'm a bit >> of a lurker, but this roadmap is important to mine and I have a >> daily at >> 5pm :(. >> >> >> >>-Justin >> >> >> >>> On Mar 18, 2020, at 6:43 PM, Saisai Shao < >> sai.sai.s...@gmail.com> wrote: >> >>> >> >>> 5pm PST in any day works for me. >> >>> >> >>> Looking forward to it. >> >>> >> >>> Thanks >> >>> Saisai >> >> >> >> >> >> >> >> >> -- >> >> 李响 Xiang Li >> >> 手机 cellphone ：+86-136-8113-8972 >> 邮件 e-mail
Re: Iceberg/Hive properties handling
I will ask Laszlo if he wants to update his doc. I see both pros and cons of catalog definition in config files. If there is an easy default then I do not mind any of the proposed solutions. OTOH I am in favor of the "use prefix for Iceberg table properties" solution, because in Hive it is common to add new keys to the property list - no restriction is in place (I am not even sure that the currently implemented blacklist for preventing to propagate properties to Iceberg tables is complete). Also I expect that we want to add Hive write specific configs to table level when the general engine independent configuration is not ideal for Hive, but every Hive query for a given table should use some specific config. Thanks, Peter Jacques Nadeau ezt írta (időpont: 2020. dec. 1., Ke 17:06): > Would someone be willing to create a document that states the current > proposal? > > It is becoming somewhat difficult to follow this thread. I also worry that > without a complete statement of the current shape that people may be > incorrectly thinking they are in alignment. > > > > -- > Jacques Nadeau > CTO and Co-Founder, Dremio > > > On Tue, Dec 1, 2020 at 5:32 AM Zoltán Borók-Nagy > wrote: > >> Thanks, Ryan. I answered inline. >> >> On Mon, Nov 30, 2020 at 8:26 PM Ryan Blue wrote: >> >>> This sounds like a good plan overall, but I have a couple of notes: >>> >>>1. We need to keep in mind that users plug in their own catalogs, so >>>iceberg.catalog could be a Glue or Nessie catalog, not just Hive or >>>Hadoop. I don’t think it makes much sense to use separate hadoop.catalog >>>and hive.catalog values. Those should just be names for catalogs >>> configured >>>in Configuration, i.e., via hive-site.xml. We then only need a >>>special value for loading Hadoop tables from paths. >>> >>> About extensibility, I think the usual Hive way is to use Java class >> names. So this way the value for 'iceberg.catalog' could be e.g. >> 'org.apache.iceberg.hive.HiveCatalog'. Then each catalog implementation >> would need to have a factory method that constructs the catalog object from >> a properties object (Map). E.g. >> 'org.apache.iceberg.hadoop.HadoopCatalog' would require >> 'iceberg.catalog_location' to be present in properties. >> >>> >>>1. I don’t think that catalog configuration should be kept in table >>>properties. A catalog should not be loaded for each table. So I don’t >>> think >>>we need iceberg.catalog_location. Instead, we should have a way to >>>define catalogs in the Configuration for tables in the metastore to >>>reference. >>> >>> I think it makes sense, on the other hand it would make adding new >> catalogs more heavy-weight, i.e. now you'd need to edit configuration files >> and restart/reinit services. Maybe it can be cumbersome in some >> environments. >> >>> >>>1. I’d rather use a prefix to exclude properties from being passed >>>to Iceberg than to include them. Otherwise, users don’t know what to do >>> to >>>pass table properties from Hive or Impala. If we exclude a prefix or >>>specific properties, then everything but the properties reserved for >>>locating the table are passed as the user would expect. >>> >>> I don't have a strong opinion about this, but yeah, maybe this behavior >> would cause the least surprises. >> >>> >>> >>> >>> On Mon, Nov 30, 2020 at 7:51 AM Zoltán Borók-Nagy >>> wrote: >>> Thanks, Peter. I answered inline. On Mon, Nov 30, 2020 at 3:13 PM Peter Vary wrote: > Hi Zoltan, > > Answers below: > > On Nov 30, 2020, at 14:19, Zoltán Borók-Nagy < > borokna...@cloudera.com.INVALID> wrote: > > Hi, > > Thanks for the replies. My take for the above questions are as follows > >- Should 'iceberg.catalog' be a required property? >- Yeah, I think it would be nice if this would be required to > avoid any implicit behavior > > Currently we have a Catalogs class to get/initialize/use the different > Catalogs. At that time the decision was to use HadoopTables as a default > catalog. > It might be worthwhile to use the same class in Impala as well, so the > behavior is consistent. > Yeah, I think it'd be beneficial for us to use the Iceberg classes whenever possible. The Catalogs class is very similar to what we have currently in Impala. > >- 'hadoop.catalog' LOCATION and catalog_location > - In Impala we don't allow setting LOCATION for tables stored > in 'hadoop.catalog'. But Impala internally sets LOCATION to the > Iceberg > table's actual location. We were also thinking about using only the > table > LOCATION, and set it to the catalog location, but we also found it > confusing. > > It could definitely work, but it is somewhat strange that we have an > external table location set to an arbitrary pat
Re: Could I participate into this email list
Of course! There is a virtual meetup tomorrow as well I will send you an invite On Tue, Dec 1, 2020, 10:23 AM 赵德儒 wrote: > Hi Guys, Could I participate into this email list, I would like to join > this discussion in the future or others activities, thank you so much. > > > -- > Best, > > *Axl Zhao* >
Could I participate into this email list
Hi Guys, Could I participate into this email list, I would like to join this discussion in the future or others activities, thank you so much. -- Best, Axl Zhao
Re: Iceberg/Hive properties handling
Would someone be willing to create a document that states the current
proposal?
It is becoming somewhat difficult to follow this thread. I also worry that
without a complete statement of the current shape that people may be
incorrectly thinking they are in alignment.
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Tue, Dec 1, 2020 at 5:32 AM Zoltán Borók-Nagy
wrote:
> Thanks, Ryan. I answered inline.
>
> On Mon, Nov 30, 2020 at 8:26 PM Ryan Blue wrote:
>
>> This sounds like a good plan overall, but I have a couple of notes:
>>
>>1. We need to keep in mind that users plug in their own catalogs, so
>>iceberg.catalog could be a Glue or Nessie catalog, not just Hive or
>>Hadoop. I don’t think it makes much sense to use separate hadoop.catalog
>>and hive.catalog values. Those should just be names for catalogs
>> configured
>>in Configuration, i.e., via hive-site.xml. We then only need a
>>special value for loading Hadoop tables from paths.
>>
>> About extensibility, I think the usual Hive way is to use Java class
> names. So this way the value for 'iceberg.catalog' could be e.g.
> 'org.apache.iceberg.hive.HiveCatalog'. Then each catalog implementation
> would need to have a factory method that constructs the catalog object from
> a properties object (Map). E.g.
> 'org.apache.iceberg.hadoop.HadoopCatalog' would require
> 'iceberg.catalog_location' to be present in properties.
>
>>
>>1. I don’t think that catalog configuration should be kept in table
>>properties. A catalog should not be loaded for each table. So I don’t
>> think
>>we need iceberg.catalog_location. Instead, we should have a way to
>>define catalogs in the Configuration for tables in the metastore to
>>reference.
>>
>> I think it makes sense, on the other hand it would make adding new
> catalogs more heavy-weight, i.e. now you'd need to edit configuration files
> and restart/reinit services. Maybe it can be cumbersome in some
> environments.
>
>>
>>1. I’d rather use a prefix to exclude properties from being passed to
>>Iceberg than to include them. Otherwise, users don’t know what to do to
>>pass table properties from Hive or Impala. If we exclude a prefix or
>>specific properties, then everything but the properties reserved for
>>locating the table are passed as the user would expect.
>>
>> I don't have a strong opinion about this, but yeah, maybe this behavior
> would cause the least surprises.
>
>>
>>
>>
>> On Mon, Nov 30, 2020 at 7:51 AM Zoltán Borók-Nagy
>> wrote:
>>
>>> Thanks, Peter. I answered inline.
>>>
>>> On Mon, Nov 30, 2020 at 3:13 PM Peter Vary
>>> wrote:
>>>
Hi Zoltan,
Answers below:
On Nov 30, 2020, at 14:19, Zoltán Borók-Nagy <
borokna...@cloudera.com.INVALID> wrote:
Hi,
Thanks for the replies. My take for the above questions are as follows
- Should 'iceberg.catalog' be a required property?
- Yeah, I think it would be nice if this would be required to avoid
any implicit behavior
Currently we have a Catalogs class to get/initialize/use the different
Catalogs. At that time the decision was to use HadoopTables as a default
catalog.
It might be worthwhile to use the same class in Impala as well, so the
behavior is consistent.
>>>
>>> Yeah, I think it'd be beneficial for us to use the Iceberg classes
>>> whenever possible. The Catalogs class is very similar to what we have
>>> currently in Impala.
>>>
- 'hadoop.catalog' LOCATION and catalog_location
- In Impala we don't allow setting LOCATION for tables stored in
'hadoop.catalog'. But Impala internally sets LOCATION to the Iceberg
table's actual location. We were also thinking about using only the
table
LOCATION, and set it to the catalog location, but we also found it
confusing.
It could definitely work, but it is somewhat strange that we have an
external table location set to an arbitrary path, and we have a different
location generated by other configs. It would be nice to have the real
location set in the external table location as well.
>>>
>>> Impala sets the real Iceberg table location for external tables. E.g. if
>>> the user issues
>>>
>>> CREATE EXTERNAL TABLE my_hive_db.iceberg_table_hadoop_catalog
>>> STORED AS ICEBERG
>>> TBLPROPERTIES('iceberg.catalog'='hadoop.catalog',
>>> 'iceberg.catalog_location'='/path/to/hadoop/catalog',
>>> 'iceberg.table_identifier'='namespace1.namespace2.ice_t');
>>>
>>> If the end user had specified LOCATION, then Impala would have raised an
>>> error. But the above DDL statement is correct, so Impala loads the iceberg
>>> table via Iceberg API, then creates the HMS table and sets LOCATION to the
>>> Iceberg table location (something like
>>> /path/to/hadoop/catalog/namespace1/namespace2/ice_t).
>>>
>>>
I like the flex
Re: Iceberg/Hive properties handling
Thanks, Ryan. I answered inline.
On Mon, Nov 30, 2020 at 8:26 PM Ryan Blue wrote:
> This sounds like a good plan overall, but I have a couple of notes:
>
>1. We need to keep in mind that users plug in their own catalogs, so
>iceberg.catalog could be a Glue or Nessie catalog, not just Hive or
>Hadoop. I don’t think it makes much sense to use separate hadoop.catalog
>and hive.catalog values. Those should just be names for catalogs configured
>in Configuration, i.e., via hive-site.xml. We then only need a special
>value for loading Hadoop tables from paths.
>
> About extensibility, I think the usual Hive way is to use Java class
names. So this way the value for 'iceberg.catalog' could be e.g.
'org.apache.iceberg.hive.HiveCatalog'. Then each catalog implementation
would need to have a factory method that constructs the catalog object from
a properties object (Map). E.g.
'org.apache.iceberg.hadoop.HadoopCatalog' would require
'iceberg.catalog_location' to be present in properties.
>
>1. I don’t think that catalog configuration should be kept in table
>properties. A catalog should not be loaded for each table. So I don’t think
>we need iceberg.catalog_location. Instead, we should have a way to
>define catalogs in the Configuration for tables in the metastore to
>reference.
>
> I think it makes sense, on the other hand it would make adding new
catalogs more heavy-weight, i.e. now you'd need to edit configuration files
and restart/reinit services. Maybe it can be cumbersome in some
environments.
>
>1. I’d rather use a prefix to exclude properties from being passed to
>Iceberg than to include them. Otherwise, users don’t know what to do to
>pass table properties from Hive or Impala. If we exclude a prefix or
>specific properties, then everything but the properties reserved for
>locating the table are passed as the user would expect.
>
> I don't have a strong opinion about this, but yeah, maybe this behavior
would cause the least surprises.
>
>
>
> On Mon, Nov 30, 2020 at 7:51 AM Zoltán Borók-Nagy
> wrote:
>
>> Thanks, Peter. I answered inline.
>>
>> On Mon, Nov 30, 2020 at 3:13 PM Peter Vary
>> wrote:
>>
>>> Hi Zoltan,
>>>
>>> Answers below:
>>>
>>> On Nov 30, 2020, at 14:19, Zoltán Borók-Nagy <
>>> borokna...@cloudera.com.INVALID> wrote:
>>>
>>> Hi,
>>>
>>> Thanks for the replies. My take for the above questions are as follows
>>>
>>>- Should 'iceberg.catalog' be a required property?
>>>- Yeah, I think it would be nice if this would be required to avoid
>>> any implicit behavior
>>>
>>> Currently we have a Catalogs class to get/initialize/use the different
>>> Catalogs. At that time the decision was to use HadoopTables as a default
>>> catalog.
>>> It might be worthwhile to use the same class in Impala as well, so the
>>> behavior is consistent.
>>>
>>
>> Yeah, I think it'd be beneficial for us to use the Iceberg classes
>> whenever possible. The Catalogs class is very similar to what we have
>> currently in Impala.
>>
>>>
>>>- 'hadoop.catalog' LOCATION and catalog_location
>>> - In Impala we don't allow setting LOCATION for tables stored in
>>> 'hadoop.catalog'. But Impala internally sets LOCATION to the Iceberg
>>> table's actual location. We were also thinking about using only the
>>> table
>>> LOCATION, and set it to the catalog location, but we also found it
>>> confusing.
>>>
>>> It could definitely work, but it is somewhat strange that we have an
>>> external table location set to an arbitrary path, and we have a different
>>> location generated by other configs. It would be nice to have the real
>>> location set in the external table location as well.
>>>
>>
>> Impala sets the real Iceberg table location for external tables. E.g. if
>> the user issues
>>
>> CREATE EXTERNAL TABLE my_hive_db.iceberg_table_hadoop_catalog
>> STORED AS ICEBERG
>> TBLPROPERTIES('iceberg.catalog'='hadoop.catalog',
>> 'iceberg.catalog_location'='/path/to/hadoop/catalog',
>> 'iceberg.table_identifier'='namespace1.namespace2.ice_t');
>>
>> If the end user had specified LOCATION, then Impala would have raised an
>> error. But the above DDL statement is correct, so Impala loads the iceberg
>> table via Iceberg API, then creates the HMS table and sets LOCATION to the
>> Iceberg table location (something like
>> /path/to/hadoop/catalog/namespace1/namespace2/ice_t).
>>
>>
>>> I like the flexibility of setting the table_identifier on table level,
>>> which could help removing naming conflicts. We might want to have this in
>>> the Iceberg Catalog implementation.
>>>
>>>
>>>- 'iceberg.table_identifier' for HiveCatalog
>>> - Yeah, it doesn't add much if we only allow using the current
>>> HMS. I think it can be only useful if we are allowing external HMSes.
>>>- Moving properties to SERDEPROPERTIES
>>> - I see that these properties are used by the SerDe classes in
>>>
