Re: [VOTE] Release Apache Uniffle (Incubating) 0.6.1-rc3

[Saisai Shao]

+1 (binding)

[x] incubating in name
[x] signature and hash fine
[x] DISCLAIMER is fine
[x] LICENSE and NOTICE are fine
[x] No unexpected binary files
[x] All source files have ASF headers
[x] Can be built from source

Best,
Jerry

Felix Cheung  于2022年12月8日周四 09:05写道：

> +1 (binding)
>
> - incubating in name
> - signature and hash fine
> - DISCLAIMER is fine
> - LICENSE and NOTICE are fine
> - No unexpected binary files
> - All source files have ASF headers
>
> What are the reason for these? btw, if this is from the Apache Spark
> project you might need to attribute it in LICENSE.
>
>
> apache-uniffle-0.6.1-incubating-src/spark-patches/spark-3.1.2_dynamic_allocation_support.patch
>
> apache-uniffle-0.6.1-incubating-src/spark-patches/spark-2.4.6_dynamic_allocation_support.patch
>
> apache-uniffle-0.6.1-incubating-src/spark-patches/spark-3.2.1_dynamic_allocation_support.patch
>
>
> On Wed, Dec 7, 2022 at 7:47 AM Xianjin YE  wrote:
>
> > +1 from my side.
> > - Download links are valid
> > - Building and unit tests are ok.
> >
> > > On Dec 7, 2022, at 18:53, Charles Zhang 
> wrote:
> > >
> > > +1(non-binding)  from me, and I checked the following items:
> > >
> > > - [X] Download links are valid.
> > > - [X] Checksums and PGP signatures are valid.
> > > - [X] All files have license headers if necessary.
> > > - [X] No compiled archives bundled in the source archive.
> > > - [X] Building is OK.
> > >
> > > Best wishes,
> > > Charles Zhang
> > > from Apache InLong
> > >
> > >
> > > Kent Yao  于2022年12月7日周三 17:26写道：
> > >
> > >> +1(non-binding)
> > >>
> > >> - Download links are valid
> > >> - Signatures are OK
> > >> - Checksums are OK
> > >> -  LICENSE, NOTICE, and DISCLAIMER  are OK
> > >> - Successfully passes UT
> > >>
> > >> Kent Yao
> > >>
> > >> On 2022/12/07 02:19:36 He Qi wrote:
> > >>> From my side
> > >>> +1
> > >>>
> > >>> I have checked
> > >>> * Download links, checksums and PGP signatures are valid
> > >>> * LICENSE, NOTICE and DISCLAIMER-WIP files are correct
> > >>> * No unlicensed compiled archives bundled in source archive.
> > >>> * UT passed. I use the command `mvn package -Pspark3
> > >>>
> > >>> On 2022/12/06 05:56:13 Kaijie Chen wrote:
> >  Hello Incubator Community,
> > 
> >  This is a call for a vote to release Apache Uniffle (Incubating)
> >  version 0.6.1-rc3.
> > 
> >  The Apache Uniffle community has voted on and approved a proposal to
> >  release Apache Uniffle (Incubating) version 0.6.1-rc3.
> > 
> >  We now kindly request the Incubator PMC members review and vote on
> > this
> >  incubator release.
> > 
> >  Uniffle community vote thread:
> >  https://lists.apache.org/thread/sorpwljxl2nssowhloswo662jng5qwr7
> > 
> >  Vote result thread:
> >  https://lists.apache.org/thread/4k9v3n6fvtmd3y37jhof2t6kb90qjg36
> > 
> >  The release candidate:
> >  https://dist.apache.org/repos/dist/dev/incubator/uniffle/0.6.1-rc3/
> > 
> >  Git tag for the release:
> >  https://github.com/apache/incubator-uniffle/tree/v0.6.1-rc3
> > 
> >  Release notes:
> >  https://uniffle.apache.org/download/release-notes-0.6.1/
> > 
> >  The artifacts signed with PGP key
> >  3B8167D035D33A4C2D725FAA39A39E985C5EFBA4
> >  corresponding to c...@apache.org, that can be found in keys file:
> >  https://dist.apache.org/repos/dist/dev/incubator/uniffle/KEYS
> > 
> >  The vote will be open for at least 72 hours or until the necessary
> >  number of votes are reached.
> > 
> >  Please vote accordingly:
> > 
> >  [ ] +1 approve
> >  [ ] +0 no opinion
> >  [ ] -1 disapprove with the reason
> > 
> >  Thanks,
> >  Kaijie Chen
> >  On behalf of Apache Uniffle (Incubating) community
> > 
> > 
> -
> >  To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> >  For additional commands, e-mail: general-h...@incubator.apache.org
> > 
> > 
> > >>>
> > >>> -
> > >>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > >>> For additional commands, e-mail: general-h...@incubator.apache.org
> > >>>
> > >>>
> > >>
> > >> -
> > >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > >> For additional commands, e-mail: general-h...@incubator.apache.org
> > >>
> > >>
> >
> >
> > -
> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > For additional commands, e-mail: general-h...@incubator.apache.org
> >
> >
>

[jira] [Updated] (HUDI-4193) Fail to compile in osx aarch_64 environment

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/HUDI-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated HUDI-4193:
--
Affects Version/s: 0.12.0

> Fail to compile in osx aarch_64 environment
> ---
>
> Key: HUDI-4193
> URL: https://issues.apache.org/jira/browse/HUDI-4193
> Project: Apache Hudi
>  Issue Type: Bug
>  Components: kafka-connect
>Affects Versions: 0.12.0
>    Reporter: Saisai Shao
>Priority: Minor
>
> hudi-kafka-connect module relies on protoc to generate code, current version 
> of protoc cannot support compiling under osx aarch_64 environment, which will 
> throw an error like below:
> {code:java}
> [ERROR] Failed to execute goal 
> com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project 
> hudi-kafka-connect: Error extracting protoc for version 3.11.4: Unsupported 
> platform: protoc-3.11.4-osx-aarch_64.exe -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the 
> command
> [ERROR]   mvn  -rf :hudi-kafka-connect
> {code}
> So here proposing to upgrade protoc version to fix this.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (HUDI-4193) Fail to compile in osx aarch_64 environment

[Saisai Shao (Jira)]

Saisai Shao created HUDI-4193:
-

 Summary: Fail to compile in osx aarch_64 environment
 Key: HUDI-4193
 URL: https://issues.apache.org/jira/browse/HUDI-4193
 Project: Apache Hudi
  Issue Type: Bug
  Components: kafka-connect
Reporter: Saisai Shao


hudi-kafka-connect module relies on protoc to generate code, current version of 
protoc cannot support compiling under osx aarch_64 environment, which will 
throw an error like below:


{code:java}
[ERROR] Failed to execute goal 
com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project 
hudi-kafka-connect: Error extracting protoc for version 3.11.4: Unsupported 
platform: protoc-3.11.4-osx-aarch_64.exe -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hudi-kafka-connect
{code}

So here proposing to upgrade protoc version to fix this.




--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSSION] Incubating Proposal of Firestorm

[Saisai Shao]

Thanks Justin for the explanation.

We discussed internally and think that a new name would be better to avoid
potential issue.

Will figure out a new name.

Best regards,
Jerry

Justin Mclean  于2022年5月23日周一 20:29写道：

> Hi,
>
> > There’s a typo in this paragraph that makes it impossible to
> > understand/changes the original meaning.
>
> Apologies I meant to say "I don’t think that is the case.”. From what I
> can see trademarks have not approved FireStorm as a name. If the project
> wants to enter the incubator with that name, and understands the risks that
> involves then that is OK. Just be aware there is a risk that this may stop
> the project from graduating from the Incubator under that name.
>
> Kind Regards,
> Justin

Re: [DISCUSSION] Incubating Proposal of Firestorm

[Saisai Shao]

I see, thanks Justin and Daniel for your inputs.

Best regards,
Jerry

Daniel Widdis  于2022年5月23日周一 11:32写道：

> Adding onto this, even if there may be no legal trademark issue, there is
> other software by this name which makes it less than desirable.  Quoted
> from [1]
>
> > Avoiding search-results confusion is important for another reason.
> Apache projects are often very quickly highly ranked. An Apache project
> with a similar name to another application in the same technical space may
> quickly come to dominate searches in that space. If someone else holds a
> related trademark, this may lead to a legal dispute. As a non-profit
> organization, the ASF and Apache projects have no business conflicting with
> an existing trademark for software products or related services.
>
> A current search for "firestorm software" reveals multiple sites related
> to [2]. Searching "firestorm download" links to [3].  Searching "apache
> firestorm" links to sites providing parts for remote control model cars, of
> which two models compatible with such parts are "Apache" and "Firestorm"
> [4].
>
> 1 - https://infra.apache.org/project-names.html
> 2 - https://www.zotac.com/us/page/firestorm
> 3 - https://www.firestormviewer.org/choose-your-platform/
> 4 - https://www.competitionx.com/hpi-racing-manuals/
>
> Not my call, I am not a lawyer and have no binding votes on the incubator,
> just adding my observations.
>
> On 5/22/22, 8:04 PM, "Justin Mclean"  wrote:
>
> HI,
>
> > According to the trademarks@ current cursory feedback, looks like
> project
> > name "Firestorm" is OK.
>
> I don’t this that is the case. Trademark checked “FlameStorm”, and
> said that if Firestorm was as issue then FlameStorm would also be an issue.
> I don’t believe they have checked FireStorm.
>
> Please don’t under estimate the cost of a project name change, I seen
> several project go through this and it it is always much more effort than
> you realise. Based on previous experience, Infra are not likely to make it
> a priority to help with a name change mid incubation or just before
> gradation.
>
> Kind Regards,
> Justin
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [DISCUSSION] Incubating Proposal of Firestorm

[Saisai Shao]

Thanks Willem for organizing, thanks Yu to join and have this open
discussion.

We have made a consensus that both projects developing separately, and seek
the opportunity to collaborate in future. So +1 from my side.

Back to the merging or separation discussion, I think both ways have pros
and cons, and currently we cannot clearly see which way is more good than
harm. So I think we could leave the user, community and time to decide. If
someday in the future, we think that the two projects should be merged for
any reason, and we can clearly see the advantages, we will definitely do
this.

Regard to the current challenges, I think it is existed and cannot be
neglected from many areas (legal, architecture, codebase, time efforts),
and will potentially make the cost too high to afford. So without seeing
the clear advantages, I would incline to be conservative.

Thanks all for your kind suggestions and open discussion.

Best regards,
Jerry


Willem Jiang  于2022年5月19日周四 11:28写道：

> Hi，
>
> We just had a quick meeting to talk about the possibility of the joint
> effort today.
>
> Jerry ,Yu and I expressed the pros and cons about merging the project
> together,  we also discussed the challenges (legal affairs, design and
> codebase decision making) of merging these two projects together. As
> there are a lot of things that need to be done before two teams can
> work together, both sides agree to enter the ASF incubator first then
> we can seek the possibility of joint efforts in the future.
>
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Wed, May 18, 2022 at 6:40 PM Yu Li  wrote:
> >
> > Thanks all for the considerate comments and thanks Jerry (and Firestorm
> > team) for the open mind.
> >
> > I totally agree that working as one project/community is a good wish
> > instead of a prerequisite (and I believe no one meant this actually),
> and I
> > don't have any intention to block the incubating process of Firestorm.
> >
> > I believe that even if we cannot make it a joint force, the two projects
> > could be good frenemy with competitive friendship (smile).
> >
> > Let's discuss more offline. :-)
> >
> > Best Regards,
> > Yu
> >
> >
> > On Wed, 18 May 2022 at 17:41, Jerry Shao  wrote:
> >
> > > Yes, I think we could discuss the possibilities offline.
> > >
> > > But I would like to clarify that "merging these projects together" is
> > > neither a "problem" (a "wish" indeed), nor a prerequisite for
> incubating.
> > > In the heyday of Hadoop, there were so many similar or even same
> purpose
> > > projects in ASF, the choice is made by user, not by ASF.
> > >
> > > Best
> > > Jerry
> > >
> > > Willem Jiang  于2022年5月18日周三 17:17写道：
> > >
> > > > On Wed, May 18, 2022 at 2:50 PM Sheng Wu 
> > > > wrote:
> > > > >
> > > > > Hi Firestorm community
> > > > >
> > > > > Considering what Yu Li is proposing, I would recommend you could do
> > > > > some discussions directly, maybe off the list if you want.
> > > > > Both of the projects are young and very new to the community, it
> may
> > > > > be not a concern from my perspective(may be concerned by some
> IPMC),
> > > > > as one/both could fail in the incubating. But with you are going to
> > > > > work on the similar things, the challenge would be for both of your
> > > > > projects to establish the incubating process. Also, when we talk
> about
> > > > > graduation in the future, it would be my concern of causing
> confusion
> > > > > to users.
> > > > >
> > > > > Two TLPs are nearly the same is really not a good practice from the
> > > > > foundation level.
> > > > >
> > > > > > We heartfully welcome you to join Firestorm project and
> community.
> > > > >
> > > > > So, I suggest not just saying joining Firestorm, this may be a
> simple
> > > > > reaction, I hope you could do more.
> > > > > This is an opportunity to do a bigger and more powerful project
> > > > > supported by Tencent and Alibaba's initial teams.
> > > > >
> > > >
> > > > After talking with LiYu and Jerry Shao,  I realized there could be
> > > > some challenges we may face.
> > > > But it would be great if we can work together to tackle the same
> problem.
> > > > I'd like to help two teams to start a meeting to discuss the
> > > > possibility of merging these projects together.
> > > >
> > > > Willem Jiang
> > > >
> > > > Twitter: willemjiang
> > > > Weibo: 姜宁willem
> > > >
> > > > > I know it is not easy to do a joint project, but it is actually
> much
> > > > > better to see you two competing with each other in the next 1-2
> years.
> > > > > In the ASF, we hope to create a better project for the whole
> industry.
> > > > >
> > > > >
> > > > > Sheng Wu 吴晟
> > > > > Twitter, wusheng1108
> > > > >
> > > > > Jerry Shao  于2022年5月18日周三 14:38写道：
> > > > > >
> > > > > > Hi Yu，
> > > > > >
> > > > > > IMO, I think we're open and welcome all the contributions. We
> > > > heartfully
> > > > > > welcome you to join Firestorm project and community.
> > > > > >
> > > > > > Best regards,
> 

Re: [DISCUSSION] Incubating Proposal of Firestorm

[Saisai Shao]

Thanks Weiwei, let me add you to the mentor list.

Best regards,
Jerry

Weiwei Yang  于2022年5月17日周二 12:18写道：

> +1
> This is an interesting project, I'd be happy to help the project's
> incubation process.
> Let me know if you need my help, Thanks
>
> On Mon, May 16, 2022 at 8:09 PM Saisai Shao 
> wrote:
>
> > Got it, thanks a lot Justin.
> >
> > Best regards,
> > Jerry
> >
> > Justin Mclean  于2022年5月17日周二 10:59写道：
> >
> > > Hi,
> > >
> > > The new project name doesn’t need to be a registered trademark, but you
> > > would still need to pick a name that is now likely to have any
> trademark
> > > issues.
> > >
> > > The project may want to get the ASF to register the mark at a later
> date,
> > > but it wold be best to talk abut that on the trademarks@ list.
> > >
> > > Kind Regards,
> > > Justin
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > > For additional commands, e-mail: general-h...@incubator.apache.org
> > >
> > >
> >
>

Re: [DISCUSSION] Incubating Proposal of Firestorm

[Saisai Shao]

Got it, thanks a lot Justin.

Best regards,
Jerry

Justin Mclean  于2022年5月17日周二 10:59写道：

> Hi,
>
> The new project name doesn’t need to be a registered trademark, but you
> would still need to pick a name that is now likely to have any trademark
> issues.
>
> The project may want to get the ASF to register the mark at a later date,
> but it wold be best to talk abut that on the trademarks@ list.
>
> Kind Regards,
> Justin
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [DISCUSSION] Incubating Proposal of Firestorm

[Saisai Shao]

Hi Daniel and Justin，

Sorry about one more questions. Seems like the name "firestorm" has some
conflicts in USPTO searching and we only submitted China trademark
registration.

So about a new name, do we need to register this new "name" before donation
(and transfer to ASF before graduation),  or just a new suitable name
(without registration) is enough for incubation?

Best regards,
Jerry

Saisai Shao  于2022年5月17日周二 10:01写道：

> Thanks Daniel and Justin, let me resolve the naming issue first.
>
> Best regards,
> Jerry
>
> Justin Mclean  于2022年5月17日周二 09:52写道：
>
>> Hi,
>>
>> > Thanks for your reply. The trademark "firestorm" as a software has
>> already
>> > been submitted the registration by Tencent in China, I'm not sure is
>> that
>> > enough or not.
>>
>> If that trademark is approved it would need to be transferred to the ASF
>> before graduation. Even f it was approved there may still be an issue with
>> the name in the USA or other countries.
>>
>> > Is there any guidance in ASF about naming or trademark things? I will
>> > consult this to our company lawyer.
>>
>> Please see [1].
>>
>> Kind Regards,
>> Justin
>>
>> 1. https://incubator.apache.org/guides/names.html
>> -
>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
>> For additional commands, e-mail: general-h...@incubator.apache.org
>>
>>

Re: [DISCUSSION] Incubating Proposal of Firestorm

[Saisai Shao]

Thanks Daniel and Justin, let me resolve the naming issue first.

Best regards,
Jerry

Justin Mclean  于2022年5月17日周二 09:52写道：

> Hi,
>
> > Thanks for your reply. The trademark "firestorm" as a software has
> already
> > been submitted the registration by Tencent in China, I'm not sure is that
> > enough or not.
>
> If that trademark is approved it would need to be transferred to the ASF
> before graduation. Even f it was approved there may still be an issue with
> the name in the USA or other countries.
>
> > Is there any guidance in ASF about naming or trademark things? I will
> > consult this to our company lawyer.
>
> Please see [1].
>
> Kind Regards,
> Justin
>
> 1. https://incubator.apache.org/guides/names.html
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [DISCUSSION] Incubating Proposal of Firestorm

[Saisai Shao]

Hi Daniel and Justin,

Thanks for your reply. The trademark "firestorm" as a software has already
been submitted the registration by Tencent in China, I'm not sure is that
enough or not.

Is there any guidance in ASF about naming or trademark things? I will
consult this to our company lawyer.

Thanks
Jerry

Justin Mclean  于2022年5月17日周二 05:59写道：

> Hi,
>
> Sounds like an interesting project. I also think the name may be
> problematic from a trademark point of view. It might be best to do a
> trademark search as see if there are any issues. Changing the name is
> difficult, but is much much harder if it needs to be done later.
>
> Kind Regards,
> Justin
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [DISCUSS] Spark version support strategy

[Saisai Shao]

>From Dev's point, it has less burden to always support the latest version
of Spark (for example). But from user's point, especially for us who
maintain Spark internally, it is not easy to upgrade the Spark version for
the first time (since we have many customizations internally), and we're
still promoting to upgrade to 3.1.2. If the community ditches the support
of old version of Spark3, users have to maintain it themselves unavoidably.

So I'm inclined to make this support in community, not by users themselves,
as for Option 2 or 3, I'm fine with either. And to relieve the burden, we
could support limited versions of Spark (for example 2 versions).

Just my two cents.

-Saisai


Jack Ye  于2021年9月15日周三 下午1:35写道：

> Hi Wing Yew,
>
> I think 2.4 is a different story, we will continue to support Spark 2.4,
> but as you can see it will continue to have very limited functionalities
> comparing to Spark 3. I believe we discussed about option 3 when we were
> doing Spark 3.0 to 3.1 upgrade. Recently we are seeing the same issue for
> Flink 1.11, 1.12 and 1.13 as well. I feel we need a consistent strategy
> around this, let's take this chance to make a good community guideline for
> all future engine versions, especially for Spark, Flink and Hive that are
> in the same repository.
>
> I can totally understand your point of view Wing, in fact, speaking from
> the perspective of AWS EMR, we have to support over 40 versions of the
> software because there are people who are still using Spark 1.4, believe it
> or not. After all, keep backporting changes will become a liability not
> only on the user side, but also on the service provider side, so I believe
> it's not a bad practice to push for user upgrade, as it will make the life
> of both parties easier in the end. New feature is definitely one of the
> best incentives to promote an upgrade on user side.
>
> I think the biggest issue of option 3 is about its scalability, because we
> will have an unbounded list of packages to add and compile in the future,
> and we probably cannot drop support of that package once created. If we go
> with option 1, I think we can still publish a few patch versions for old
> Iceberg releases, and committers can control the amount of patch versions
> to guard people from abusing the power of patching. I see this as a
> consistent strategy also for Flink and Hive. With this strategy, we can
> truly have a compatibility matrix for engine versions against Iceberg
> versions.
>
> -Jack
>
>
>
> On Tue, Sep 14, 2021 at 10:00 PM Wing Yew Poon 
> wrote:
>
>> I understand and sympathize with the desire to use new DSv2 features in
>> Spark 3.2. I agree that Option 1 is the easiest for developers, but I don't
>> think it considers the interests of users. I do not think that most users
>> will upgrade to Spark 3.2 as soon as it is released. It is a "minor
>> version" upgrade in name from 3.1 (or from 3.0), but I think we all know
>> that it is not a minor upgrade. There are a lot of changes from 3.0 to 3.1
>> and from 3.1 to 3.2. I think there are even a lot of users running Spark
>> 2.4 and not even on Spark 3 yet. Do we also plan to stop supporting Spark
>> 2.4?
>>
>> Please correct me if I'm mistaken, but the folks who have spoken out in
>> favor of Option 1 all work for the same organization, don't they? And they
>> don't have a problem with making their users, all internal, simply upgrade
>> to Spark 3.2, do they? (Or they are already running an internal fork that
>> is close to 3.2.)
>>
>> I work for an organization with customers running different versions of
>> Spark. It is true that we can backport new features to older versions if we
>> wanted to. I suppose the people contributing to Iceberg work for some
>> organization or other that either use Iceberg in-house, or provide software
>> (possibly in the form of a service) to customers, and either way, the
>> organizations have the ability to backport features and fixes to internal
>> versions. Are there any users out there who simply use Apache Iceberg and
>> depend on the community version?
>>
>> There may be features that are broadly useful that do not depend on Spark
>> 3.2. Is it worth supporting them on Spark 3.0/3.1 (and even 2.4)?
>>
>> I am not in favor of Option 2. I do not oppose Option 1, but I would
>> consider Option 3 too. Anton, you said 5 modules are required; what are the
>> modules you're thinking of?
>>
>> - Wing Yew
>>
>>
>>
>>
>>
>> On Tue, Sep 14, 2021 at 5:38 PM Yufei Gu  wrote:
>>
>>> Option 1 sounds good to me. Here are my reasons:
>>>
>>> 1. Both 2 and 3 will slow down the development. Considering the limited
>>> resources in the open source community, the upsides of option 2 and 3 are
>>> probably not worthy.
>>> 2. Both 2 and 3 assume the use cases may not exist. It's hard to predict
>>> anything, but even if these use cases are legit, users can still get the
>>> new feature by backporting it to an older version in case of upgrading to a
>>> newer version 

Re: [VOTE] Accept Linkis into the Apache Incubator

[Saisai Shao]

+1 (binding)

Best,
Saisai

Willem Jiang  于2021年7月29日周四 上午7:59写道：

> +1 (binding)
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Mon, Jul 26, 2021 at 11:12 PM 俊平堵  wrote:
> >
> > Dear all,
> >
> >
> > Following up the [DISCUSS] thread on Linkis[1], I would like to call a
> > VOTE to accept Linkis into the Apache Incubator. Please cast your
> > vote:
> >
> >   [ ] +1, bring Linkis into the Incubator
> >   [ ] +0, I don't care either way
> >   [ ] -1, do not bring Linkis into the Incubator, because...
> >
> > The vote will open for at least 72 hours. Only votes from the
> > Incubator PMC are binding, but welcome for the votes from everyone.
> >
> > Please check out the Linkis Proposal from the incubator wiki[2].
> > [1]
> https://lists.apache.org/thread.html/r5d3af665795ebd3028ca9978e92af5d650a43b6680470d16e43f3847%40%3Cgeneral.incubator.apache.org%3E
> > [2] https://cwiki.apache.org/confluence/display/INCUBATOR/LinkisProposal
> >
> > Regards,
> > Junping Du
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [VOTE] Release Apache InLong 0.9.0-incubating RC1

[Saisai Shao]

+1 (binding)

Checked: signatures，LICENSE and NOTICE files，DISCLAIMER，ASF header. LGTM.

Thanks!
Saisai

Jean-Baptiste Onofre  于2021年7月21日周三 下午1:22写道：

> +1 (binding)
>
> Checked:
> - incubating in the name
> - signatures OK
> - LICENSE and NOTICE files look good
> - DISCLAIMER file is there
> - ASF header is present in files
> - no binary file detected in source archive
>
> Thanks !
> Regards
> JB
>
> > Le 19 juil. 2021 à 04:49, Goson zhang  a écrit :
> >
> > Hello Incubator Community,
> >
> >This is a call for a vote to release Apache InLong (Incubating)
> version
> >0.9.0-incubating RC1
> >
> >The Apache InLong community has voted on and approved a proposal to
> > release
> >Apache InLong (Incubating) version 0.9.0-incubating RC1
> >
> >We now kindly request the Incubator PMC members review and vote on
> this
> > incubator release.
> >
> >InLong community vote thread:
> >•
> >
> https://lists.apache.org/thread.html/r87faccf8c1ebe22c433bfcc43c17541fa81b7b7b6df9c393ed65460b%40%3Cdev.inlong.apache.org%3E
> >
> >Vote result thread:
> >•
> >
> https://lists.apache.org/thread.html/rb64923310d6d1a7ae0bbd4c89a4426ef032e936cc9330154526e72ad%40%3Cdev.inlong.apache.org%3E
> >
> >The release candidate:
> >•
> >
> https://dist.apache.org/repos/dist/dev/incubator/inlong/0.9.0-incubating-RC1
> >
> >Git tag for the release:
> >•
> https://github.com/apache/incubator-inlong/tree/0.9.0-incubating-RC1
> >
> >Release notes:
> >•
> >
> https://github.com/apache/incubator-inlong/releases/tag/0.9.0-incubating-RC1
> >
> >The artifacts signed with PGP key
> > 9B12C2228BDFF4F4CFE849445EF3A66D57EC647A, corresponding to
> > gxch...@apache.org, that can be found in keys file:
> >• https://dist.apache.org/repos/dist/dev/incubator/inlong/KEYS
> >
> >The vote will be open for at least 72 hours or until a necessary
> number
> > of votes are reached.
> >
> >Please vote accordingly:
> >
> >[ ] +1 approve
> >[ ] +0 no opinion
> >[ ] -1 disapprove with the reason
> >
> > Thanks,
> > On behalf of Apache InLong (Incubating) community
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [EXTERNAL] Re: Spark 3.0 with Scala 2.12 support in Livy

[Saisai Shao]

Yes, Livy supports Spark 3.0 + Scala 2.12 in the master branch. But it
doesn't have a lot of tests, I expect there're some issues in it.

Ashish Agarwal  于2021年4月19日周一 下午2:01写道：

> Thanks Renat I saw that "https://issues.apache.org/jira/browse/LIVY-756;
> is fixed for version 0.8.0.
>
> Any idea of the date for the next release of livy ?
>
> On Mon, Apr 5, 2021 at 9:31 PM Renat Bekbolatov 
> wrote:
>
>> Hi Ashish!
>>
>> I found this:
>> https://github.com/apache/incubator-livy/pull/300#issuecomment-664909277
>>
>>
>>
>> Kind regards,
>>
>> Renat
>>
>>
>>
>> *From:* Ashish Agarwal 
>> *Sent:* Saturday, April 3, 2021 10:18 AM
>> *To:* u...@livy.incubator.apache.org
>> *Subject:* [EXTERNAL] Re: Spark 3.0 with Scala 2.12 support in Livy
>>
>>
>>
>> A gentle reminder
>>
>>
>>
>> On Thu, Apr 1, 2021 at 11:34 PM Ashish Agarwal <
>> ashishagarwal1...@gmail.com> wrote:
>>
>> Does Livy has a support for Spark 3.0 with scala 2.12. ? From the docs it
>> seems no Please confirm
>>
>>
>>
>> I see a FeatureRequest
>> https://issues.apache.org/jira/projects/LIVY/issues/LIVY-756
>> ?
>> resolved and will be available in 0.8 version  what is the expected release
>> date for 0.8 ?
>>
>>
>>
>>
>>
>> --
>>
>> Regards
>> Ashish Agarwal
>> Ph - +91- 9711163631
>>
>>
>>
>> --
>>
>> Regards
>> Ashish Agarwal
>> Ph - +91- 9711163631
>>
>
>
> --
> Regards
> Ashish Agarwal
> Ph - +91- 9711163631
>

Re: [VOTE] Release Apache TubeMQ (Incubating) 0.8.0-incubating RC4

[Saisai Shao]

+1 binding.

Checked license file, ASC and SHA512, LGTM.

Best regards,
Sasisai

Goson zhang  于2021年2月26日周五 上午9:40写道：

> Hi all:
>
> We need your help, can anyone help us to review this release?
>
> The release of this version has lasted for more than a month, and many
> changes have been delayed due to version release issues and cannot be
> merged into the master.
>
> We have restored the "-WIP" label until the bdb relies [1] is resolved :
> currently relies on BDB's code implementation needs to be adjusted, and it
> will take a long time.
>
>
> 1.
>
> https://lists.apache.org/thread.html/r6858f56cb88ea8a279fb8bbce7be06b2230a701af46d4ac79fc08fee%40%3Cgeneral.incubator.apache.org%3E
>
> Thanks!
>
> Yuanbo Liu  于2021年2月22日周一 下午4:19写道：
>
> > Hello Incubator Community,
> >
> > This is a call for a vote to release Apache TubeMQ (Incubating)
> version
> > 0.8.0-incubating RC4
> >
> > The Apache TubeMQ community has voted on and approved a proposal to
> > release
> > Apache TubeMQ (Incubating) version 0.8.0-incubating RC4
> >
> > We now kindly request the Incubator PMC members review and vote on
> this
> > incubator release.
> >
> > TubeMQ community vote thread:
> > •
> >
> >
> https://lists.apache.org/thread.html/rd8ff5969f969f91334cf576b648aa8dbb55194dd55508dc399806053%40%3Cdev.tubemq.apache.org%3E
> >
> > Vote result thread:
> > •
> >
> >
> https://lists.apache.org/thread.html/reb37cce379a2c4d8048266b7899c99f5b988cd3225ad3fb685e4fbf1%40%3Cdev.tubemq.apache.org%3E
> >
> > The release candidate:
> > •
> >
> >
> https://dist.apache.org/repos/dist/dev/incubator/tubemq/0.8.0-incubating-RC4/
> >
> > Git tag for the release:
> > •
> https://github.com/apache/incubator-tubemq/tree/0.8.0-incubating-RC4
> >
> > Release notes:
> > •
> >
> >
> https://github.com/apache/incubator-tubemq/releases/tag/0.8.0-incubating-RC4
> >
> > The artifacts signed with PGP key E8754243E7B3E044, corresponding to
> > yua...@apache.org, that can be found in keys file:
> > • https://dist.apache.org/repos/dist/dev/incubator/tubemq/KEYS
> >
> > The vote will be open for at least 72 hours or until a necessary
> number
> > of votes are reached.
> >
> > Please vote accordingly:
> >
> > [ ] +1 approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove with the reason
> >
> > Thanks,
> > On behalf of Apache TubeMQ (Incubating) community
> >
>

[jira] [Closed] (LIVY-833) Livy allows users to see password in config files (spark.ssl.keyPassword,spark.ssl.keyStorePassword,spark.ssl.trustStorePassword, etc)

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao closed LIVY-833.

Resolution: Won't Fix

> Livy allows users to see password in config files 
> (spark.ssl.keyPassword,spark.ssl.keyStorePassword,spark.ssl.trustStorePassword,
>  etc)
> --
>
> Key: LIVY-833
> URL: https://issues.apache.org/jira/browse/LIVY-833
> Project: Livy
>  Issue Type: Bug
>  Components: Server
>Affects Versions: 0.7.0
>Reporter: Kaidi Zhao
>Priority: Major
>  Labels: security
>
> It looks like a regular user (client) of Livy, can use commands like: 
> spark.sparkContext.getConf().getAll()
> The command will retry all spark configurations including those passwords 
> (such as spark.ssl.trustStorePassword, spark.ssl.keyPassword). 
> I would suggest to block / mask these password. 
> PS, Spark's UI fixed this issue in this 
> https://issues.apache.org/jira/browse/SPARK-16796



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (LIVY-833) Livy allows users to see password in config files (spark.ssl.keyPassword,spark.ssl.keyStorePassword,spark.ssl.trustStorePassword, etc)

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289457#comment-17289457
 ] 

Saisai Shao commented on LIVY-833:
--

This is the problem of Spark, not Livy. Spark uses the configuration to store 
everything including passwords, and user could get configurations within 
application through many ways. Besides Livy, user still could get password by 
using spark-shell, spark-submit and others.

If user could submit code through Livy to spark when Livy security is enabled, 
it means user permission to execute code, it is acceptable to see the passwords.

> Livy allows users to see password in config files 
> (spark.ssl.keyPassword,spark.ssl.keyStorePassword,spark.ssl.trustStorePassword,
>  etc)
> --
>
> Key: LIVY-833
> URL: https://issues.apache.org/jira/browse/LIVY-833
> Project: Livy
>  Issue Type: Bug
>  Components: Server
>Affects Versions: 0.7.0
>Reporter: Kaidi Zhao
>Priority: Major
>  Labels: security
>
> It looks like a regular user (client) of Livy, can use commands like: 
> spark.sparkContext.getConf().getAll()
> The command will retry all spark configurations including those passwords 
> (such as spark.ssl.trustStorePassword, spark.ssl.keyPassword). 
> I would suggest to block / mask these password. 
> PS, Spark's UI fixed this issue in this 
> https://issues.apache.org/jira/browse/SPARK-16796



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release Apache Livy 0.7.1 (incubating) based on RC1

[Saisai Shao]

Thanks all for your votes, we already 3 binding +1 and 1 non-binding +1,
I'm going to close the vote and prepare for the release.

Best regards,
Saisai

Jean-Baptiste Onofre  于2021年2月10日周三 下午12:48写道：

> +1 (binding)
>
> Quickly check name, signature, DISCLAIMER, NOTICE and LICENSE.
>
> Regards
> JB
>
> > Le 7 févr. 2021 à 02:50, Saisai Shao  a écrit :
> >
> > The Livy PPMC has voted to release Livy 0.7.1 RC1 as the next Livy
> release.
> >
> > Livy enables programmatic, fault-tolerant, multi-tenant submission of
> > Spark jobs from web/mobile apps (no Spark client needed). So, multiple
> > users can interact with your Spark cluster concurrently and reliably.
> >
> > Vote thread:
> >
> https://lists.apache.org/thread.html/r9ac9717f267828028484a7b4aff63e9adbf0cdeca9ce6d674096bf71%40%3Cdev.livy.apache.org%3E
> >
> > Result thread:
> >
> https://lists.apache.org/thread.html/r083a6f1a5926df7441842a280f60c532c25a3eddc0e24416493814ae%40%3Cdev.livy.apache.org%3E
> >
> > The RC is based on tag v0.7.1-incubating-rc1:
> > https://github.com/apache/incubator-livy/commit/
> > 7c3d341926db69fb57a4978b15d4e96f06312267
> >
> > The release files can be found here:
> >
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.1-incubating-rc1/
> >
> > The staged maven artifacts can be found here:
> > https://repository.apache.org/content/repositories/orgapachelivy-1014
> >
> > This release is a hot-fix release, only added one commit compared to
> 0.7.0
> > -incubating
> > https://github.com/apache/incubator-livy
> > /commit/9f1ba47a2f0d8accc435b133b42c3a76aa9ac846
> >
> > Please help to vote. Thanks!
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [VOTE] Release Apache Livy 0.7.1 (incubating) based on RC1

[Saisai Shao]

My +1 as well. Please help to vote.

Thanks
Saisai

Justin Mclean  于2021年2月8日周一 上午9:36写道：

> Hi,
>
> +1 (binding)
>
> I checked in the source release:
> - incubating in name
> - signatures and hashes correct
> - NOTICE  file is incorrect, please don’t say "Copyright 2018 and onwards”
> but include current year.
> - LICENSE is missing license for [1][2] (and other font files)
> - No unexpected binary files
> - source files have correct headers
> - can compile from source except for livy-python-api but it looks like I’m
> missing a dependancy
>
> This is the error:
> kg_resources.DistributionNotFound: The 'flake8' distribution was not found
> and is required by the application
>
> Thanks,
> Justin
>
> 1.
> ./docs/assets/themes/apache/bootstrap/fonts/glyphicons-halflings-regular.svg
> 2
> ./server/src/main/resources/org/apache/livy/server/ui/static/fonts/glyphicons-halflings-regular.svg
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

[VOTE] Release Apache Livy 0.7.1 (incubating) based on RC1

[Saisai Shao]

The Livy PPMC has voted to release Livy 0.7.1 RC1 as the next Livy release.

Livy enables programmatic, fault-tolerant, multi-tenant submission of
Spark jobs from web/mobile apps (no Spark client needed). So, multiple
users can interact with your Spark cluster concurrently and reliably.

Vote thread:
https://lists.apache.org/thread.html/r9ac9717f267828028484a7b4aff63e9adbf0cdeca9ce6d674096bf71%40%3Cdev.livy.apache.org%3E

Result thread:
https://lists.apache.org/thread.html/r083a6f1a5926df7441842a280f60c532c25a3eddc0e24416493814ae%40%3Cdev.livy.apache.org%3E

The RC is based on tag v0.7.1-incubating-rc1:
https://github.com/apache/incubator-livy/commit/
7c3d341926db69fb57a4978b15d4e96f06312267

The release files can be found here:
https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.1-incubating-rc1/

The staged maven artifacts can be found here:
https://repository.apache.org/content/repositories/orgapachelivy-1014

This release is a hot-fix release, only added one commit compared to 0.7.0
-incubating
https://github.com/apache/incubator-livy
/commit/9f1ba47a2f0d8accc435b133b42c3a76aa9ac846

Please help to vote. Thanks!

Re: [VOTE] Release Livy 0.7.1 based on RC1

[Saisai Shao]

Thanks guys for your votes, we have 3 binding +1, no 0s or -1s. Vote
passes, I'll follow up on the general@ list.

Thanks
Saisai

Jeff Zhang  于2021年2月6日周六 上午8:03写道：

> +1
>
>
> Alex Bozarth  于2021年2月6日周六 上午2:46写道：
>
> > +1
> >
> > thanks for the release work
> >
> >
> > *Alex Bozarth*
> > Software Engineer
> > Center for Open-Source Data & AI Technologies
> > --
> > *E-mail:* *ajboz...@us.ibm.com* 
> > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
> >
> >
> > 505 Howard Street
> > San Francisco, CA 94105
> > United States
> >
> >
> >
> > [image: Inactive hide details for Saisai Shao ---02/03/2021 09:44:31
> > PM---This vote is for releasing Livy 0.7.1 based on RC1. The vote]Saisai
> > Shao ---02/03/2021 09:44:31 PM---This vote is for releasing Livy 0.7.1
> > based on RC1. The vote will be open until Feb 10 23:59 UTC and
> >
> > From: Saisai Shao 
> > To: d...@livy.incubator.apache.org
> > Date: 02/03/2021 09:44 PM
> > Subject: [EXTERNAL] [VOTE] Release Livy 0.7.1 based on RC1
> >
> > --
> >
> >
> >
> > This vote is for releasing Livy 0.7.1 based on RC1.
> >
> > The vote will be open until Feb 10 23:59 UTC and
> > will pass with a minimum of 3 +1 binding votes and a majority of positive
> > votes.
> >
> > [+1] This release is ready to send to the Incubator PMC for approval
> >
> > [-1] This release is not ready because...
> >
> > This vote is held according to Apache Incubator release policy:
> > https://incubator.apache.org/policy/incubation.html#releases
> >
> > The RC is based on tag v0.7.1-incubating-rc1:
> >
> >
> https://github.com/apache/incubator-livy/commit/7c3d341926db69fb57a4978b15d4e96f06312267
> >
> >
> > The release files can be found here:
> >
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.1-incubating-rc1/
> >
> >
> > The staged maven artifacts can be found here:
> > https://repository.apache.org/content/repositories/orgapachelivy-1014
> >
> > This release is a hot-fix release, only added one commit compared to
> > 0.7.0-incubating
> >
> >
> https://github.com/apache/incubator-livy/commit/9f1ba47a2f0d8accc435b133b42c3a76aa9ac846
> >
> >
> > Thanks
> > Saisai
> >
> >
> >
> >
>
> --
> Best Regards
>
> Jeff Zhang
>

[VOTE] Release Livy 0.7.1 based on RC1

[Saisai Shao]

This vote is for releasing Livy 0.7.1 based on RC1.

The vote will be open until Feb 10 23:59 UTC and
will pass with a minimum of 3 +1 binding votes and a majority of positive
votes.

[+1] This release is ready to send to the Incubator PMC for approval

[-1] This release is not ready because...

This vote is held according to Apache Incubator release policy:
https://incubator.apache.org/policy/incubation.html#releases

The RC is based on tag v0.7.1-incubating-rc1:
https://github.com/apache/incubator-livy/commit/7c3d341926db69fb57a4978b15d4e96f06312267

The release files can be found here:
https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.1-incubating-rc1/

The staged maven artifacts can be found here:
https://repository.apache.org/content/repositories/orgapachelivy-1014

This release is a hot-fix release, only added one commit compared to
0.7.0-incubating
https://github.com/apache/incubator-livy/commit/9f1ba47a2f0d8accc435b133b42c3a76aa9ac846

Thanks
Saisai

Re: Process to rename the project

[Saisai Shao]

Thanks all for the reply.

Yes we will put into discussion in TubeMQ mail list. But just want know
beforehand if it is possible to rename the project (I haven't seen this
before)? if so, is there any documentation things to follow, if not, we
will figure out other ways to do so.

Thanks
Jerry

zhangli...@apache.org  于2020年11月23日周一 下午8:09写道：

> Hi Jerry,
>
> It is better to discuss it in TubeMQ dev mail list first.
>
> --
>
> Sincerely,
> Liang Zhang (John)
> Apache ShardingSphere
>
>
> Saisai Shao  于2020年11月23日周一 上午10:38写道：
>
> > The podling project is Apache TubeMQ. Currently the project is planning
> to
> > change the name, but haven't yet figured out a new name, neither did a
> > podling name search. Just want to understand what is the process.
> >
> > CC @general@incubator.apache.org 
> >
> > Thanks
> > Jerry
> >
> > Dave Fisher  于2020年11月23日周一 上午10:26写道：
> >
> > > Has the podling done a podling name search?
> > >
> > > What podling?
> > >
> > > What is the new name?
> > >
> > > Changing names is work for infrastructure so we want to be careful not
> to
> > > be spurious!
> > >
> > > Regards,
> > > Dave
> > >
> > > Sent from my iPhone
> > >
> > > > On Nov 22, 2020, at 6:19 PM, Jerry Shao  wrote:
> > > >
> > > > 
> > > > Hi Team,
> > > >
> > > > We have a podling project which wants to change its current name, the
> > > main reason of changing is to expand the scope of the project. I would
> > like
> > > to know if it is possible to change the project name? If so, what is
> the
> > > process, do we have any document about this?
> > > >
> > > > Thanks
> > > > Jerry
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: private-unsubscr...@incubator.apache.org
> > > For additional commands, e-mail: private-h...@incubator.apache.org
> > >
> > >
> >
>

Re: Process to rename the project

[Saisai Shao]

The podling project is Apache TubeMQ. Currently the project is planning to
change the name, but haven't yet figured out a new name, neither did a
podling name search. Just want to understand what is the process.

CC @general@incubator.apache.org 

Thanks
Jerry

Dave Fisher  于2020年11月23日周一 上午10:26写道：

> Has the podling done a podling name search?
>
> What podling?
>
> What is the new name?
>
> Changing names is work for infrastructure so we want to be careful not to
> be spurious!
>
> Regards,
> Dave
>
> Sent from my iPhone
>
> > On Nov 22, 2020, at 6:19 PM, Jerry Shao  wrote:
> >
> > 
> > Hi Team,
> >
> > We have a podling project which wants to change its current name, the
> main reason of changing is to expand the scope of the project. I would like
> to know if it is possible to change the project name? If so, what is the
> process, do we have any document about this?
> >
> > Thanks
> > Jerry
>
>
> -
> To unsubscribe, e-mail: private-unsubscr...@incubator.apache.org
> For additional commands, e-mail: private-h...@incubator.apache.org
>
>

Re: Suggested S3 FileIO/Getting Started

[Saisai Shao]

Thanks a lot Ryan for your explanation, greatly helpful.

Best regards,
Saisai

Ryan Blue  于2020年11月14日周六 上午2:03写道：

> Saisai,
>
> Iceberg's FileIO interface doesn't require guarantees as strict as a
> Hadoop-compatible FileSystem. For S3, that allows us to avoid negative
> caching that can cause problems when reading a table that has just been
> updated. (Specifically, S3A performs a HEAD request to check whether a path
> is a directory even for overwrites.)
>
> It's up to users whether they want to use S3FileIO or S3A with
> HadoopFileIO, or even HadoopFileIO with a custom FileSystem. We are working
> to make it easy to switch between them, but there are notable problems like
> ORC not using the FileIO interface yet. I would recommend using S3FileIO to
> avoid negative caching, but S3A with S3Guard may be another way to avoid
> the problem. I think the default should be to use S3FileIO if it is in the
> classpath, and fall back to a Hadoop FileSystem if it isn't.
>
> For cloud providers, I think the question is whether there is a need for
> the relaxed guarantees. If there is no need, then using HadoopFileIO and a
> FileSystem works great and is probably the easiest way to maintain an
> implementation for the object store.
>
> On Thu, Nov 12, 2020 at 7:31 PM Saisai Shao 
> wrote:
>
>> Hi all,
>>
>> Sorry to chime in, I also have a same concern about using Iceberg with
>> Object storage.
>>
>> One of my concerns with S3FileIO is getting tied too much to a single
>>> cloud provider. I'm wondering if an ObjectStoreFileIO would be helpful
>>> so that S3FileIO and (a future) GCSFileIO could share logic? I haven't
>>> looked deep enough into the S3FileIO to know how much logic is not s3
>>> specific. Maybe the FileIO interface is enough.
>>>
>>
>> Now we have a S3 specific FileIO implementation, is it recommended to use
>> this one instead of s3a like HCFS implementation? Also each public cloud
>> provider has its own HCFS implementation for its own object storage. Are we
>> going to suggest to create a specific FileIO implementation or use the
>> existing HCFS implementation?
>>
>> Thanks
>> Saisai
>>
>>
>> Daniel Weeks  于2020年11月13日周五 上午1:09写道：
>>
>>> Hey John, about the concerns around cloud provider dependency, I feel
>>> like the FileIO interface is actually the right level of abstraction
>>> already.
>>>
>>> That interface basically requires "open for read" and "open for write",
>>> where the implementation will diverge across different platforms.
>>>
>>> I guess you could think of it as S3FileIO is to FileIO what
>>> S3AFileSystem is to FileSystem (in Hadoop).  You can have many different
>>> implementations that coexist.
>>>
>>> In fact, recent changes to the Catalog allow for very flexible
>>> management of FIleIO and you could even have files within a table split
>>> across multiple cloud vendors.
>>>
>>> As to the consistency questions, the list operation can be inconsistent
>>> (e.g. if a new file is created and the implementation relies on list then
>>> read, it may not see newly created objects.  Iceberg does not list, so that
>>> should not be an issue).
>>>
>>> The stated read-after-write consistency is limited and does not include:
>>>  - Read after overwrite
>>>  - Read after delete
>>>  - Read after negative cache (e.g. a GET or HEAD that occurred before
>>> the object was created).
>>>
>>> Some of those inconsistencies have caused problems in certain cases when
>>> it comes to committing data (the negative cache being the main culprit).
>>>
>>> -Dan
>>>
>>>
>>> On Wed, Nov 11, 2020 at 6:49 PM John Clara 
>>> wrote:
>>>
>>>> Update: I think I'm wrong about the listing part. I think it will only
>>>> do the HEAD request. Also it seems like the consistency issue is
>>>> probably not something my team would encounter with our current jobs.
>>>>
>>>> On 2020/11/12 02:17:10, John Clara  wrote:
>>>>  > (Not sure if this is actually replying or just starting a new
>>>> thread)>
>>>>  >
>>>>  > Hi Daniel,>
>>>>  >
>>>>  > Thanks for the response! It's very helpful and answers a lot my
>>>> questions.>
>>>>  >
>>>>  > A couple follow ups:>
>>>>  >
>>>>  > One of my concerns with S3FileIO is getting tied too mu

Re: Suggested S3 FileIO/Getting Started

[Saisai Shao]

Hi all,

Sorry to chime in, I also have a same concern about using Iceberg with
Object storage.

One of my concerns with S3FileIO is getting tied too much to a single
> cloud provider. I'm wondering if an ObjectStoreFileIO would be helpful
> so that S3FileIO and (a future) GCSFileIO could share logic? I haven't
> looked deep enough into the S3FileIO to know how much logic is not s3
> specific. Maybe the FileIO interface is enough.
>

Now we have a S3 specific FileIO implementation, is it recommended to use
this one instead of s3a like HCFS implementation? Also each public cloud
provider has its own HCFS implementation for its own object storage. Are we
going to suggest to create a specific FileIO implementation or use the
existing HCFS implementation?

Thanks
Saisai


Daniel Weeks  于2020年11月13日周五 上午1:09写道：

> Hey John, about the concerns around cloud provider dependency, I feel like
> the FileIO interface is actually the right level of abstraction already.
>
> That interface basically requires "open for read" and "open for write",
> where the implementation will diverge across different platforms.
>
> I guess you could think of it as S3FileIO is to FileIO what S3AFileSystem
> is to FileSystem (in Hadoop).  You can have many different implementations
> that coexist.
>
> In fact, recent changes to the Catalog allow for very flexible management
> of FIleIO and you could even have files within a table split across
> multiple cloud vendors.
>
> As to the consistency questions, the list operation can be inconsistent
> (e.g. if a new file is created and the implementation relies on list then
> read, it may not see newly created objects.  Iceberg does not list, so that
> should not be an issue).
>
> The stated read-after-write consistency is limited and does not include:
>  - Read after overwrite
>  - Read after delete
>  - Read after negative cache (e.g. a GET or HEAD that occurred before the
> object was created).
>
> Some of those inconsistencies have caused problems in certain cases when
> it comes to committing data (the negative cache being the main culprit).
>
> -Dan
>
>
> On Wed, Nov 11, 2020 at 6:49 PM John Clara 
> wrote:
>
>> Update: I think I'm wrong about the listing part. I think it will only
>> do the HEAD request. Also it seems like the consistency issue is
>> probably not something my team would encounter with our current jobs.
>>
>> On 2020/11/12 02:17:10, John Clara  wrote:
>>  > (Not sure if this is actually replying or just starting a new thread)>
>>  >
>>  > Hi Daniel,>
>>  >
>>  > Thanks for the response! It's very helpful and answers a lot my
>> questions.>
>>  >
>>  > A couple follow ups:>
>>  >
>>  > One of my concerns with S3FileIO is getting tied too much to a single >
>>  > cloud provider. I'm wondering if an ObjectStoreFileIO would be helpful
>> >
>>  > so that S3FileIO and (a future) GCSFileIO could share logic? I haven't
>> >
>>  > looked deep enough into the S3FileIO to know how much logic is not s3 >
>>  > specific. Maybe the FileIO interface is enough.>
>>  >
>>  > About consistency (no need to respond here):>
>>  > I'm seeing that during "getFileStatus" my version of s3a does some
>> list >
>>  > requests (but I'm not sure if that could fail from consistency
>> issues).>
>>  > I'm also confused about the read-after-(initial) write part:>
>>  > "Amazon S3 provides read-after-write consistency for PUTS of new
>> objects >
>>  > in your S3 bucket in all Regions with one caveat. The caveat is that
>> if >
>>  > you make a HEAD or GET request to a key name before the object is >
>>  > created, then create the object shortly after that, a subsequent GET >
>>  > might not return the object due to eventual consistency. - >
>>  > https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html;>
>>  >
>>  > When my version of s3a does a create, it first does a
>> getMetadataRequest >
>>  > (HEAD) to check if the object exists before creating the object. I
>> think >
>>  > this is talked about in this issue: >
>>  > https://github.com/apache/iceberg/issues/1398 and talked about in the
>> >
>>  > S3FileIO PR: https://github.com/apache/iceberg/pull/1573. I'll follow
>> up >
>>  > in that issue for more info.>
>>  >
>>  > John>
>>  >
>>  >
>>  > On 2020/11/12 00:36:10, Daniel Weeks 
>> wrote:>
>>  > > Hey John, I might be able to help answer some of your questions and >
>>  > provide>>
>>  > > some context around how you might want to go forward.>>
>>  > >>
>>  > > So, one fundamental aspect of Iceberg is that it only relies on a
>> few>>
>>  > > operations (as defined by the FileIO interface). This makes much of
>> the>>
>>  > > functionality and complexity of full file system implementations>>
>>  > > unnecessary. You should not need features like S3Guard or
>> additional S3>>
>>  > > operations these implementations rely on in order to achieve file >
>>  > system>>
>>  > > contract behavior. Consistency issues should also not be a problem >
>>  > since>>
>>  > > Iceberg does not 

[jira] [Updated] (SPARK-13704) TaskSchedulerImpl.createTaskSetManager can be expensive, and result in lost executors due to blocked heartbeats

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated SPARK-13704:

Description: 
In some cases, TaskSchedulerImpl.createTaskSetManager can be expensive. For 
example, in a Yarn cluster, it may call the topology script for rack awareness. 
When submit a very large job in a very large Yarn cluster, the topology script 
may take signifiant time to run. And this blocks receiving executors' 
heartbeats, which may result in lost executors

Stacktraces we observed which is related to this issue:
{code}https://issues.apache.org/jira/browse/SPARK-13704#
"dag-scheduler-event-loop" daemon prio=10 tid=0x7f8392875800 nid=0x26e8 
runnable [0x7f83576f4000]
   java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:272)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0xf551f460> (a 
java.lang.UNIXProcess$ProcessPipeInputStream)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
- locked <0xf5529740> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:154)
at java.io.BufferedReader.read1(BufferedReader.java:205)
at java.io.BufferedReader.read(BufferedReader.java:279)
- locked <0xf5529740> (a java.io.InputStreamReader)
at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.parseExecResult(Shell.java:728)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:524)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at 
org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:251)
at 
org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:188)
at 
org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
at 
org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:101)
at 
org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:81)
at 
org.apache.spark.scheduler.cluster.YarnScheduler.getRackForHost(YarnScheduler.scala:38)
at 
org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:210)
at 
org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:189)
at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at 
org.apache.spark.scheduler.TaskSetManager.org$apache$spark$scheduler$TaskSetManager$$addPendingTask(TaskSetManager.scala:189)
at 
org.apache.spark.scheduler.TaskSetManager$$anonfun$1.apply$mcVI$sp(TaskSetManager.scala:158)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at 
org.apache.spark.scheduler.TaskSetManager.(TaskSetManager.scala:157)
at 
org.apache.spark.scheduler.TaskSchedulerImpl.createTaskSetManager(TaskSchedulerImpl.scala:187)
at 
org.apache.spark.scheduler.TaskSchedulerImpl.submitTasks(TaskSchedulerImpl.scala:161)
- locked <0xea3b8a88> (a 
org.apache.spark.scheduler.cluster.YarnScheduler)
at 
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:872)
at 
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:778)
at 
org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:762)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1362)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

"sparkDriver-akka.actor.default-dispatcher-15" daemon prio=10 
tid=0x7f829c02 nid=0x2737 waiting for monitor entry [0x7f8355ebd000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at 
org.apache.spark.scheduler.TaskSchedulerImpl.executorHeartbeatReceived(TaskSchedulerImpl.scala:362)
- waiting to lock <0xea3b8a88> (a 
org.apache.spark.scheduler.cluster.YarnScheduler)
 

Re: Question about Iceberg release cadence

[Saisai Shao]

Would like to get structured streaming reader in in the next release :).
Will spend time on addressing new feedbacks.

Thanks
Saisai

Mass Dosage  于2020年8月27日周四 下午10:36写道：

> I'm all for a release. The only thing still required for basic Hive read
> support (other than documentation of course!) is producing a *single* jar
> that can be added to Hive's classpath, the PR for that is at
> https://github.com/apache/iceberg/pull/1267.
>
> Thanks,
>
> Adrian
>
> On Thu, 27 Aug 2020 at 01:26, Anton Okolnychyi
>  wrote:
>
>> +1 on releasing structured streaming source. I should be able to do one
>> more review round tomorrow.
>>
>> - Anton
>>
>> On 26 Aug 2020, at 17:12, Jungtaek Lim 
>> wrote:
>>
>> I hope we include Spark structured streaming read as well in the next
>> release; that was proposed in Feb this year and still around. Quoting my
>> comment on benefit of the streaming read on Spark;
>>
>> This would be the major feature to cover the gap on use case for
>>> structured streaming between Delta Lake and Iceberg. There's a technical
>>> limitation on Spark structured streaming itself (global watermark), which
>>> requires workaround via splitting query into multiple queries &
>>> intermediate storage supporting end-to-end exactly once. Delta Lake covers
>>> the case, and I really would like to see the case also covered by Iceberg.
>>> I see there're lots of works in progress on the milestone (and these are
>>> great features which should be done), but after this we cover both batch
>>> and streaming workloads being done with Spark, which is a huge step forward
>>> on Spark users.
>>
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>> On Thu, Aug 27, 2020 at 1:13 AM Ryan Blue 
>> wrote:
>>
>>> Hi Marton,
>>>
>>> 0.9.0 was released about 6 weeks ago, so I don't think we've planned
>>> when the next release will be yet. I think it's a good idea to release
>>> soon, though. The Flink sink is close to being ready as well and I'd like
>>> to get both of those released so that the contributors can start using them.
>>>
>>> Seems like a good question for the broader community: how about a
>>> release in the next month or so for Hive reads and the Flink sink?
>>>
>>> rb
>>>
>>> On Wed, Aug 26, 2020 at 8:58 AM Marton Bod  wrote:
>>>
 Hi Team,

 I was wondering whether there is a release cadence already in place for
 Iceberg, e.g. how often releases will take place approximately? Which
 commits/features as release candidates in the near term?

 We're looking to integrate Iceberg into Hive, however, the current
 0.9.1 release does not yet contain the StorageHandler code in iceberg-mr.
 Knowing the approximate release timelines would help greatly with our
 integration planning.

 Of course, happy to get involved with ongoing dev/stability efforts to
 help achieve a new release of this module.

 Thanks a lot,
 Marton

>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
>>

Re: [DISCUSS] Rename iceberg-hive module?

[Saisai Shao]

+1 for the changes.

Mass Dosage  于2020年8月20日周四 下午5:46写道：

> +1 for `iceberg-hive-metastore` as I found this confusing when I first
> started working with the code.
>
> On Thu, 20 Aug 2020 at 03:27, Jungtaek Lim 
> wrote:
>
>> +1 for `iceberg-hive-metastore` and also +1 for RD's proposal.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>>
>>
>> On Thu, Aug 20, 2020 at 11:20 AM Jingsong Li 
>> wrote:
>>
>>> +1 for `iceberg-hive-metastore`
>>>
>>> I'm confused about `iceberg-hive` and `iceberg-mr`.
>>>
>>> Best,
>>> Jingsong
>>>
>>> On Thu, Aug 20, 2020 at 9:48 AM Dongjoon Hyun 
>>> wrote:
>>>
 +1 for `iceberg-hive-metastore`.

 Maybe, is `Apache Iceberg 1.0.0` a good candidate to have that breaking
 change?

 Bests,
 Dongjoon.

 On Wed, Aug 19, 2020 at 6:35 PM RD  wrote:

> I'm +1 for this rename.  I think we should keep the iceberg-mr module
> as is and maybe add a new module iceberg-hive-exec [not sure if it is a
> good idea to salvage iceberg-hive for this purpose] which contains hive
> specific StorageHandler, Serde and IcebergHivInputFormat classes.
>
> -R
>
> On Wed, Aug 19, 2020 at 5:06 PM Ryan Blue  wrote:
>
>> In the discussion this morning, we talked about what to name the
>> runtime module we want to add for Hive, iceberg-hive-runtime.
>> Unfortunately, iceberg-hive is the Hive _metastore_ module, so it is a 
>> bit
>> misleading to name the Hive runtime module iceberg-hive-runtime. It was
>> also pointed out that the iceberg-hive module is confusing for other
>> reasons: someone unfamiliar with it would expect to use it to work with
>> Hive, but it has no InputFormat or StorageHandler classes.
>>
>> Both problems are a result of a poor name for iceberg-hive. Maybe we
>> should rename iceberg-hive to iceberg-hive-metastore.
>>
>> The drawback is that a module people could use will disappear (I'm
>> assuming we won't rename iceberg-mr to iceberg-hive right away). But most
>> people probably use a runtime Jar, so it might be a good time to make 
>> this
>> change before there are more people depending on it.
>>
>> What does everyone think? Should we do the rename?
>>
>> rb
>>
>> --
>> Ryan Blue
>>
>
>>>
>>> --
>>> Best, Jingsong Lee
>>>
>>

Re: New committer: Shardul Mahadik

[Saisai Shao]

Congrats!

Thanks
Saisai

OpenInx  于2020年7月23日周四 上午10:06写道：

> Congratulations !
>
> On Thu, Jul 23, 2020 at 9:31 AM Jingsong Li 
> wrote:
>
>> Congratulations Shardul! Well deserved!
>>
>> Best,
>> Jingsong
>>
>> On Thu, Jul 23, 2020 at 7:27 AM Anton Okolnychyi
>>  wrote:
>>
>>> Congrats and welcome! Keep up the good work!
>>>
>>> - Anton
>>>
>>> On 22 Jul 2020, at 16:02, RD  wrote:
>>>
>>> Congratulations Shardul! Well deserved!
>>>
>>> -Best,
>>> R.
>>>
>>> On Wed, Jul 22, 2020 at 2:24 PM Ryan Blue  wrote:
>>>
 Hi everyone,

 I'd like to congratulate Shardul Mahadik, who was just invited to join
 the Iceberg committers!

 Thanks for all your contributions, Shardul!

 rb


 --
 Ryan Blue

>>>
>>>
>>
>> --
>> Best, Jingsong Lee
>>
>

Re: Next release

[Saisai Shao]

Hi,

I just merged the Spark 3 support yesterday, I would wait for most tests to
stabilize this feature. In the meanwhile, you can build the master branch
yourself to try this latest feature.

Thanks
Saisai

Aliaksandr Sasnouskikh  于2020年7月2日周四 下午10:50写道：

> Hi, thanks for notifying about the release, will check it today, shouldn’t
> be too complex. Will update you on the plans till the end of the week.
>
> чт, 2 июля 2020 г. в 14:41, murat migdisoglu :
>
> > Hello,
> >
> > When can we expect the next release for livy?
> >  Spark3.0 support (which has been merged today) is kind of critical for
> us.
> >
> >
> > Thank you
> >
> > --
> > "Talkers aren’t good doers. Rest assured that we’re going there to use
> our
> > hands, not our tongues."
> > W. Shakespeare
> >
> --
> Best regards,
> Aliaksandr Sasnouskikh
> https://github.com/jahstreet/
> https://www.linkedin.com/in/jahstreet/
> +31 (0) 6 51 29 58 39
>

[jira] [Closed] (LIVY-593) Adding Scala 2.12 support

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao closed LIVY-593.

Fix Version/s: 0.8.0
   Resolution: Duplicate

> Adding Scala 2.12 support
> -
>
> Key: LIVY-593
> URL: https://issues.apache.org/jira/browse/LIVY-593
> Project: Livy
>  Issue Type: Improvement
>Affects Versions: 0.6.0
>Reporter: Pateyron
>Priority: Major
> Fix For: 0.8.0
>
>
> Spark 2.4.2 is build with Scala 2.12 however Apache Livy 0.6.0 do not support 
> Scala 2.12.
> I am writing to let you know that Livy 0.6.0 do not work with Spark 2.4.2 
> Scala 2.12.
> Could you tell me when Livy will support Scala 2.12 ?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (LIVY-562) Add support of scala 2.12

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao closed LIVY-562.

Fix Version/s: 0.8.0
   Resolution: Duplicate

> Add support of scala 2.12
> -
>
> Key: LIVY-562
> URL: https://issues.apache.org/jira/browse/LIVY-562
> Project: Livy
>  Issue Type: Improvement
>Reporter: Viacheslav Tradunsky
>Priority: Major
> Fix For: 0.8.0
>
>
> I know that adding support of a new scala version is something that can be 
> time consuming and would require some testing phase, but it would be great at 
> least to have this support in integration-test module (no scala suffix 
> supported there, yet).
> We can use livy with spark 2.4.0 with scala 2.12 and livy java client, but we 
> cannot use integration-test module for testing because it fails on scala 
> compatibility issues, such as 
> {code:java}
> java.lang.NoSuchMethodError: 
> scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
>  at org.apache.livy.test.framework.MiniCluster.(MiniCluster.scala:211)
>  at org.apache.livy.test.framework.Cluster$.liftedTree1$1(Cluster.scala:102)
>  at 
> org.apache.livy.test.framework.Cluster$.cluster$lzycompute(Cluster.scala:100)
>  at org.apache.livy.test.framework.Cluster$.cluster(Cluster.scala:98)
>  at org.apache.livy.test.framework.Cluster$.get(Cluster.scala:125)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (LIVY-423) Adding Scala 2.12 support

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-423.
--
Fix Version/s: 0.8.0
 Assignee: Thomas Prelle
   Resolution: Fixed

Issue resolved by pull request 300
https://github.com/apache/incubator-livy/pull/300

> Adding Scala 2.12 support
> -
>
> Key: LIVY-423
> URL: https://issues.apache.org/jira/browse/LIVY-423
> Project: Livy
>  Issue Type: New Feature
>  Components: Build, Core, REPL
>    Reporter: Saisai Shao
>Assignee: Thomas Prelle
>Priority: Major
> Fix For: 0.8.0
>
>
> Spark 2.3 already integrates with Scala 2.12 support, it will possibly 
> release 2.12 artifacts. So in the Livy side we should support Scala 2.12 
> build and interpreter. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (LIVY-756) Add spark 3 support

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-756.
--
Fix Version/s: 0.8.0
 Assignee: Thomas Prelle
   Resolution: Fixed

> Add spark 3 support
> ---
>
> Key: LIVY-756
> URL: https://issues.apache.org/jira/browse/LIVY-756
> Project: Livy
>  Issue Type: Improvement
>Reporter: Thomas Prelle
>Assignee: Thomas Prelle
>Priority: Major
> Fix For: 0.8.0
>
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> Spark 3 will be release soon.
> A support of spark 3 will be nice.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (LIVY-756) Add spark 3 support

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17150037#comment-17150037
 ] 

Saisai Shao commented on LIVY-756:
--

Issue resolved by pull request 300
https://github.com/apache/incubator-livy/pull/300

> Add spark 3 support
> ---
>
> Key: LIVY-756
> URL: https://issues.apache.org/jira/browse/LIVY-756
> Project: Livy
>  Issue Type: Improvement
>Reporter: Thomas Prelle
>Priority: Major
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> Spark 3 will be release soon.
> A support of spark 3 will be nice.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Iceberg sync notes - 17 & 19 June

[Saisai Shao]

Hi team,

Any plan to get this Structured Streaming Read support (
https://github.com/apache/iceberg/pull/796) in 0.9.0 release? Would be
appreciated anyone can take a  review. Thanks!

Best regards,
Saisai

Ryan Blue  于2020年6月23日周二 上午6:42写道：

> Hi everyone,
>
> I just posted my notes from the community sync on the 17th. And, I saw
> that Adrian had already added notes for the one on the 19th! So if you're
> interested in either one, please have a look at the doc:
>
>
> https://docs.google.com/document/d/1YuGhUdukLP5gGiqCbk0A5_Wifqe2CZWgOd3TbhY3UQg/edit#heading=h.bb281cffsdf1
>
> Thanks to everyone that came for the discussion!
>
> rb
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: [VOTE] Graduate to a top-level project

[Saisai Shao]

+1 for graduation.

Junjie Chen  于2020年5月13日周三 上午9:33写道：

> +1
>
> On Wed, May 13, 2020 at 8:07 AM RD  wrote:
>
>> +1 for graduation!
>>
>> On Tue, May 12, 2020 at 3:50 PM John Zhuge  wrote:
>>
>>> +1
>>>
>>> On Tue, May 12, 2020 at 3:33 PM parth brahmbhatt <
>>> brahmbhatt.pa...@gmail.com> wrote:
>>>
 +1

 On Tue, May 12, 2020 at 3:31 PM Anton Okolnychyi
  wrote:

> +1 for graduation
>
> On 12 May 2020, at 15:30, Ryan Blue  wrote:
>
> +1
>
> Jacques, I agree. I'll make sure to let you know about the IPMC vote
> because we'd love to have your support there as well.
>
> On Tue, May 12, 2020 at 3:02 PM Jacques Nadeau 
> wrote:
>
>> I'm +1.
>>
>> (I think that is non-binding here but binding at the incubator level)
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>>
>> On Tue, May 12, 2020 at 2:35 PM Romin Parekh 
>> wrote:
>>
>>> +1
>>>
>>> On Tue, May 12, 2020 at 2:32 PM Owen O'Malley <
>>> owen.omal...@gmail.com> wrote:
>>>
 +1

 On Tue, May 12, 2020 at 2:16 PM Ryan Blue  wrote:

> Hi everyone,
>
> I propose that the Iceberg community should petition to graduate
> from the Apache Incubator to a top-level project.
>
> Here is the draft board resolution:
>
> Establish the Apache Iceberg Project
>
> WHEREAS, the Board of Directors deems it to be in the best interests 
> of
> the Foundation and consistent with the Foundation's purpose to 
> establish
> a Project Management Committee charged with the creation and 
> maintenance
> of open-source software, for distribution at no charge to the public,
> related to managing huge analytic datasets using a standard at-rest
> table format that is designed for high performance and ease of use..
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
> (PMC), to be known as the "Apache Iceberg Project", be and hereby is
> established pursuant to Bylaws of the Foundation; and be it further
>
> RESOLVED, that the Apache Iceberg Project be and hereby is responsible
> for the creation and maintenance of software related to managing huge
> analytic datasets using a standard at-rest table format that is 
> designed
> for high performance and ease of use; and be it further
>
> RESOLVED, that the office of "Vice President, Apache Iceberg" be and
> hereby is created, the person holding such office to serve at the
> direction of the Board of Directors as the chair of the Apache Iceberg
> Project, and to have primary responsibility for management of the
> projects within the scope of responsibility of the Apache Iceberg
> Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and hereby are
> appointed to serve as the initial members of the Apache Iceberg 
> Project:
>
>  * Anton Okolnychyi 
>  * Carl Steinbach   
>  * Daniel C. Weeks  
>  * James R. Taylor  
>  * Julien Le Dem
>  * Owen O'Malley
>  * Parth Brahmbhatt 
>  * Ratandeep Ratti  
>  * Ryan Blue
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Ryan Blue be appointed to
> the office of Vice President, Apache Iceberg, to serve in accordance
> with and subject to the direction of the Board of Directors and the
> Bylaws of the Foundation until death, resignation, retirement, removal
> or disqualification, or until a successor is appointed; and be it
> further
>
> RESOLVED, that the Apache Iceberg Project be and hereby is tasked with
> the migration and rationalization of the Apache Incubator Iceberg
> podling; and be it further
>
> RESOLVED, that all responsibilities pertaining to the Apache Incubator
> Iceberg podling encumbered upon the Apache Incubator PMC are hereafter
> discharged.
>
> Please vote in the next 72 hours.
>
> [ ] +1 Petition the IPMC to graduate to top-level project
> [ ] +0
> [ ] -1 Wait to graduate because . . .
> --
> Ryan Blue
>

>>>
>>> --
>>> Thanks,
>>> Romin
>>>
>>>
>>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
>
>
>>>
>>> --
>>> John Zhuge
>>>
>>
>
> --
> Best Regards
>

Re: [Discuss] Merge spark-3 branch into master

[Saisai Shao]

We're still facing the version constraint problem by gradle plugins :(


jiantao yu  于2020年4月22日周三 下午12:08写道：

> Hi saisai,
> Would you please share your progress on merging spark-3 branch into
> master?
> We  are trying iceberg with spark sql, which is only supported in spark 3.
>
> On 2020/03/27 01:53:09, Saisai Shao  wrote:
> > Thanks Ryan, let me take a try.>
> >
> > Best regards,>
> > Saisai>
> >
> > Ryan Blue  于2020年3月27日周五 上午12:15写道：>
> >
> > > Here’s how it was done before:>
> > >
> https://github.com/apache/incubator-iceberg/blob/867ec79a5c2f7619cb10546b5cc7f7bbc7d61621/build.gradle#L225-L244>
>
> > >>
> > > That defines a set of projects called baselineProjects and applies>
> > > baseline like this:>
> > >>
> > > configure(baselineProjects) {>
> > >   apply plugin: 'com.palantir.baseline-checkstyle'>
> > >   ...>
> > > }>
> > >>
> > > The baseline config has since been moved into baseline.gradle>
> > > <
> https://github.com/apache/incubator-iceberg/blob/master/baseline.gradle>>
> > > so changes should probably go into that file. Thanks for looking into
> this!>
> > >>
> > > On Thu, Mar 26, 2020 at 6:23 AM Mass Dosage  wrote:>
> > >>
> > >> We'd like to know how to do this too. We're working on the Hive>
> > >> integration and Hive requires older versions of many of the libraries
> that>
> > >> Iceberg uses (Guava, Calcite and Avro are being the most
> problematic).>
> > >> We're going to need to shade some of these in the iceberg modules we
> depend>
> > >> on but it would also be very useful to be able to override the
> versions in>
> > >> the iceberg-hive and iceberg-mr modules so that they aren't locked to
> the>
> > >> same versions as the rest of the projects.>
> > >>>
> > >> On Thu, 26 Mar 2020 at 01:53, Saisai Shao  wrote:>
> > >>>
> > >>> Hi Ryan,>
> > >>>>
> > >>> As mentioned in the meeting, would you please point me out the way
> to>
> > >>> make some submodules excluded from consistent-versions plugin.>
> > >>>>
> > >>> Thanks>
> > >>> Saisai>
> > >>>>
> > >>> Anton Okolnychyi  于2020年3月18日周三 上午4:14写道：>
> > >>>>
> > >>>> I am +1 on having spark-2 and spark-3 modules as well.>
> > >>>>>
> > >>>> On 7 Mar 2020, at 15:03, RD  wrote:>
> > >>>>>
> > >>>> I'm +1 to separate modules for spark-2 and spark-3, after the 0.8>
> > >>>> release.>
> > >>>> I think it would be a big change in organizations to adopt Spark-3>
> > >>>> since that brings in Scala-2.12 which is binary incompatible to
> previous>
> > >>>> Scala versions. Hence this adoption could take a lot of time. I
> know in our>
> > >>>> company we have no near term plans to move to Spark 3.>
> > >>>>>
> > >>>> -Best,>
> > >>>> R.>
> > >>>>>
> > >>>> On Thu, Mar 5, 2020 at 6:33 PM Saisai Shao >
> > >>>> wrote:>
> > >>>>>
> > >>>>> I was thinking that if it is possible to limit version lock plugin
> to>
> > >>>>> only iceberg core related subprojects., seems like current>
> > >>>>> consistent-versions plugin doesn't allow to do so. So not sure if
> there're>
> > >>>>> some other plugins which could provide similar functionality with
> more>
> > >>>>> flexibility?>
> > >>>>>>
> > >>>>>  Any suggestions on this?>
> > >>>>>>
> > >>>>> Best regards,>
> > >>>>> Saisai>
> > >>>>>>
> > >>>>> Saisai Shao  于2020年3月5日周四 下午3:12写道：>
> > >>>>>>
> > >>>>>> I think the requirement of supporting different version should
> be>
> > >>>>>> quite common. As Iceberg is a table format which should be
> adapted to>
> > >>>>>> different engines like Hive, Flink, Spark. To support different
> versions is>
> > >>>>>> a real problem, Spark i

Re: [Discuss] Merge spark-3 branch into master

[Saisai Shao]

Thanks Ryan, let me take a try.

Best regards,
Saisai

Ryan Blue  于2020年3月27日周五 上午12:15写道：

> Here’s how it was done before:
> https://github.com/apache/incubator-iceberg/blob/867ec79a5c2f7619cb10546b5cc7f7bbc7d61621/build.gradle#L225-L244
>
> That defines a set of projects called baselineProjects and applies
> baseline like this:
>
> configure(baselineProjects) {
>   apply plugin: 'com.palantir.baseline-checkstyle'
>   ...
> }
>
> The baseline config has since been moved into baseline.gradle
> <https://github.com/apache/incubator-iceberg/blob/master/baseline.gradle>
> so changes should probably go into that file. Thanks for looking into this!
>
> On Thu, Mar 26, 2020 at 6:23 AM Mass Dosage  wrote:
>
>> We'd like to know how to do this too. We're working on the Hive
>> integration and Hive requires older versions of many of the libraries that
>> Iceberg uses (Guava, Calcite and Avro are being the most problematic).
>> We're going to need to shade some of these in the iceberg modules we depend
>> on but it would also be very useful to be able to override the versions in
>> the iceberg-hive and iceberg-mr modules so that they aren't locked to the
>> same versions as the rest of the projects.
>>
>> On Thu, 26 Mar 2020 at 01:53, Saisai Shao  wrote:
>>
>>> Hi Ryan,
>>>
>>> As mentioned in the meeting, would you please point me out the way to
>>> make some submodules excluded from consistent-versions plugin.
>>>
>>> Thanks
>>> Saisai
>>>
>>> Anton Okolnychyi  于2020年3月18日周三 上午4:14写道：
>>>
>>>> I am +1 on having spark-2 and spark-3 modules as well.
>>>>
>>>> On 7 Mar 2020, at 15:03, RD  wrote:
>>>>
>>>> I'm +1 to separate modules for spark-2 and spark-3, after the 0.8
>>>> release.
>>>> I think it would be a big change in organizations to adopt Spark-3
>>>> since that brings in Scala-2.12 which is binary incompatible to previous
>>>> Scala versions. Hence this adoption could take a lot of time. I know in our
>>>> company we have no near term plans to move to Spark 3.
>>>>
>>>> -Best,
>>>> R.
>>>>
>>>> On Thu, Mar 5, 2020 at 6:33 PM Saisai Shao 
>>>> wrote:
>>>>
>>>>> I was thinking that if it is possible to limit version lock plugin to
>>>>> only iceberg core related subprojects., seems like current
>>>>> consistent-versions plugin doesn't allow to do so. So not sure if there're
>>>>> some other plugins which could provide similar functionality with more
>>>>> flexibility?
>>>>>
>>>>>  Any suggestions on this?
>>>>>
>>>>> Best regards,
>>>>> Saisai
>>>>>
>>>>> Saisai Shao  于2020年3月5日周四 下午3:12写道：
>>>>>
>>>>>> I think the requirement of supporting different version should be
>>>>>> quite common. As Iceberg is a table format which should be adapted to
>>>>>> different engines like Hive, Flink, Spark. To support different versions 
>>>>>> is
>>>>>> a real problem, Spark is just one case, Hive, Flink could also be the 
>>>>>> case
>>>>>> if the interface is changed across major versions. Also version lock may
>>>>>> have problems when several engines coexisted in the same build, as they
>>>>>> will transiently introduce lots of dependencies which may be conflicted, 
>>>>>> it
>>>>>> may be hard to figure out one version which could satisfy all, and 
>>>>>> usually
>>>>>> they only confined to a single module.
>>>>>>
>>>>>>  So I think we should figure out a way to support such scenario, not
>>>>>> just maintaining branches one by one.
>>>>>>
>>>>>> Ryan Blue  于2020年3月5日周四 上午2:53写道：
>>>>>>
>>>>>>> I think the key is that this wouldn't be using the same published
>>>>>>> artifacts. This work would create a spark-2.4 artifact and a spark-3.0
>>>>>>> artifact. (And possibly a spark-common artifact.)
>>>>>>>
>>>>>>> It seems reasonable to me to have those in the same build instead of
>>>>>>> in separate branches, as long as the Spark dependencies are not leaked
>>>>>>> outside of the modules. That said, I'd rather have the additional checks
>>

[jira] [Assigned] (LIVY-751) Livy server should allow to customize LIVY_CLASSPATH

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao reassigned LIVY-751:


Assignee: Shingo Furuyama

> Livy server should allow to customize LIVY_CLASSPATH
> 
>
> Key: LIVY-751
> URL: https://issues.apache.org/jira/browse/LIVY-751
> Project: Livy
>  Issue Type: Improvement
>  Components: Server
>Affects Versions: 0.7.0
>Reporter: Shingo Furuyama
>Assignee: Shingo Furuyama
>Priority: Minor
> Fix For: 0.8.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> What we want to do is - specify LIVY_CLASSPATH at the time of booting livy 
> server to use the other version of hadoop, which is not included in livy 
> artifact.
> The background is - we are trying to use livy 0.7.0-incubating and spark 
> 2.4.5 with YARN HDP2.6.4, but we encountered an error due to the 
> incompatibility of livy included hadoop and HDP2.6.4 hadoop. We came up with 
> a workaround that 1. remove hadoop from livy artifact by building with `mvn 
> -Dhadoop.scope=provided` and 2. add HDP2.6.4 hadoop jars to the classpath for 
> livy server by `hadoop classpath` command at the time of booting livy server. 
> However, bin/livy-server is not allowing change LIVY_CLASSPATH.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (LIVY-751) Livy server should allow to customize LIVY_CLASSPATH

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067383#comment-17067383
 ] 

Saisai Shao commented on LIVY-751:
--

Issue resolved by pull request 282
https://github.com/apache/incubator-livy/pull/282

> Livy server should allow to customize LIVY_CLASSPATH
> 
>
> Key: LIVY-751
> URL: https://issues.apache.org/jira/browse/LIVY-751
> Project: Livy
>  Issue Type: Improvement
>  Components: Server
>Affects Versions: 0.7.0
>Reporter: Shingo Furuyama
>Priority: Minor
> Fix For: 0.8.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> What we want to do is - specify LIVY_CLASSPATH at the time of booting livy 
> server to use the other version of hadoop, which is not included in livy 
> artifact.
> The background is - we are trying to use livy 0.7.0-incubating and spark 
> 2.4.5 with YARN HDP2.6.4, but we encountered an error due to the 
> incompatibility of livy included hadoop and HDP2.6.4 hadoop. We came up with 
> a workaround that 1. remove hadoop from livy artifact by building with `mvn 
> -Dhadoop.scope=provided` and 2. add HDP2.6.4 hadoop jars to the classpath for 
> livy server by `hadoop classpath` command at the time of booting livy server. 
> However, bin/livy-server is not allowing change LIVY_CLASSPATH.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (LIVY-751) Livy server should allow to customize LIVY_CLASSPATH

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-751.
--
Resolution: Fixed

> Livy server should allow to customize LIVY_CLASSPATH
> 
>
> Key: LIVY-751
> URL: https://issues.apache.org/jira/browse/LIVY-751
> Project: Livy
>  Issue Type: Improvement
>  Components: Server
>Affects Versions: 0.7.0
>Reporter: Shingo Furuyama
>Priority: Minor
> Fix For: 0.8.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> What we want to do is - specify LIVY_CLASSPATH at the time of booting livy 
> server to use the other version of hadoop, which is not included in livy 
> artifact.
> The background is - we are trying to use livy 0.7.0-incubating and spark 
> 2.4.5 with YARN HDP2.6.4, but we encountered an error due to the 
> incompatibility of livy included hadoop and HDP2.6.4 hadoop. We came up with 
> a workaround that 1. remove hadoop from livy artifact by building with `mvn 
> -Dhadoop.scope=provided` and 2. add HDP2.6.4 hadoop jars to the classpath for 
> livy server by `hadoop classpath` command at the time of booting livy server. 
> However, bin/livy-server is not allowing change LIVY_CLASSPATH.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [Discuss] Merge spark-3 branch into master

[Saisai Shao]

Hi Ryan,

As mentioned in the meeting, would you please point me out the way to make
some submodules excluded from consistent-versions plugin.

Thanks
Saisai

Anton Okolnychyi  于2020年3月18日周三 上午4:14写道：

> I am +1 on having spark-2 and spark-3 modules as well.
>
> On 7 Mar 2020, at 15:03, RD  wrote:
>
> I'm +1 to separate modules for spark-2 and spark-3, after the 0.8 release.
> I think it would be a big change in organizations to adopt Spark-3 since
> that brings in Scala-2.12 which is binary incompatible to previous Scala
> versions. Hence this adoption could take a lot of time. I know in our
> company we have no near term plans to move to Spark 3.
>
> -Best,
> R.
>
> On Thu, Mar 5, 2020 at 6:33 PM Saisai Shao  wrote:
>
>> I was thinking that if it is possible to limit version lock plugin to
>> only iceberg core related subprojects., seems like current
>> consistent-versions plugin doesn't allow to do so. So not sure if there're
>> some other plugins which could provide similar functionality with more
>> flexibility?
>>
>>  Any suggestions on this?
>>
>> Best regards,
>> Saisai
>>
>> Saisai Shao  于2020年3月5日周四 下午3:12写道：
>>
>>> I think the requirement of supporting different version should be quite
>>> common. As Iceberg is a table format which should be adapted to different
>>> engines like Hive, Flink, Spark. To support different versions is a real
>>> problem, Spark is just one case, Hive, Flink could also be the case if the
>>> interface is changed across major versions. Also version lock may have
>>> problems when several engines coexisted in the same build, as they will
>>> transiently introduce lots of dependencies which may be conflicted, it may
>>> be hard to figure out one version which could satisfy all, and usually they
>>> only confined to a single module.
>>>
>>>  So I think we should figure out a way to support such scenario, not
>>> just maintaining branches one by one.
>>>
>>> Ryan Blue  于2020年3月5日周四 上午2:53写道：
>>>
>>>> I think the key is that this wouldn't be using the same published
>>>> artifacts. This work would create a spark-2.4 artifact and a spark-3.0
>>>> artifact. (And possibly a spark-common artifact.)
>>>>
>>>> It seems reasonable to me to have those in the same build instead of in
>>>> separate branches, as long as the Spark dependencies are not leaked outside
>>>> of the modules. That said, I'd rather have the additional checks that
>>>> baseline provides in general since this is a short-term problem. It would
>>>> just be nice if we could have versions that are confined to a single
>>>> module. The Nebula plugin that baseline uses claims to support that, but I
>>>> couldn't get it to work.
>>>>
>>>> On Wed, Mar 4, 2020 at 6:38 AM Saisai Shao 
>>>> wrote:
>>>>
>>>>> Just think a bit on this. I agree that generally introducing different
>>>>> versions of same dependencies could be error prone. But I think the case
>>>>> here should not lead to  issue:
>>>>>
>>>>> 1.  These two sub-modules spark-2 and spark-3 are isolated, they're
>>>>> not dependent on either.
>>>>> 2. They can be differentiated by names when generating jars, also they
>>>>> will not be relied by other modules in Iceberg.
>>>>>
>>>>> So this dependency issue should not be the case here. And in Maven it
>>>>> could be achieved easily. Please correct me if wrong.
>>>>>
>>>>> Best regards,
>>>>> Saisai
>>>>>
>>>>> Saisai Shao  于2020年3月4日周三 上午10:01写道：
>>>>>
>>>>>> Thanks Matt,
>>>>>>
>>>>>> If branching is the only choice, then we would potentially have two
>>>>>> *master* branches until spark-3 is vastly adopted. That will somehow
>>>>>> increase the maintenance burden and lead to inconsistency. IMO I'm OK 
>>>>>> with
>>>>>> the branching way, just think that we should have a clear way to keep
>>>>>> tracking of two branches.
>>>>>>
>>>>>> Best,
>>>>>> Saisai
>>>>>>
>>>>>> Matt Cheah  于2020年3月4日周三 上午9:50写道：
>>>>>>
>>>>>>> I think it’s generally dangerous and error-prone to try to support
>>>>>>> two versions of the same library in the same build,

Re: Review request

[Saisai Shao]

Sorry for the delay, I will take a review.

Thanks
Saisai

Shingo Furuyama  于2020年3月24日周二 下午12:45写道：

> Hello,
>
> I am Shingo Furuyama, working at Yahoo! JAPAN.
>
> We are trying to use livy 0.7.0-incubating and spark 2.4.5 with YARN
> HDP2.6.4 and have sent a few tiny patches related to the work.
> Can anyone review the patches?
>
> [LIVY-751]Livy server should allow to customize LIVY_CLASSPATH
> https://github.com/apache/incubator-livy/pull/282
>
> Add description of POST /batches
> https://github.com/apache/incubator-livy/pull/283
>
> modified the description of POST /sessions/{sessionId}/completion
> https://github.com/apache/incubator-livy/pull/285
>
> Regards,
> Shingo
>
>

Re: Shall we start a regular community sync up?

[Saisai Shao]

5pm PST in any day works for me.

Looking forward to it.

Thanks
Saisai

Re: [DISCUSSION] plan to release TubeMQ 0.3.0 version

[Saisai Shao]

Let's cut and stabilize the branch-0.3 for now. And do the 0.3.0 release
when it is ready.

Thanks
Saisai

俊平堵  于2020年3月9日周一 下午5:16写道：

> Thanks Goson for proposing this. It sounds like a reasonable plan. +1.
>
> PS: In most Apache projects, there is no need to vote for release plan.
> Just make sure no -1 (from PMC as binding vote) for your proposal (branch,
> version, timeline, etc.), and go ahead to do the release work (but we vote
> for your release bits).
>
> Thanks,
>
> Junping
>
> Goson zhang  于2020年3月9日周一 下午5:12写道：
>
> >  I plan to release version 0.3.0. Let ’s see if it ’s OK and then start
> > voting.
> >
> > The current version is stable, and the 0.3.0 version is planned to be
> > released at the end of March. I will first establish a 0.3.0 branch,
> which
> > will no longer accept new requirements, and the bugs will be merged into
> > the master and  the branch, until the 0.3.0 version is released.
> >
>

Re: Can a livy client upload and run different jar packages multiple times?

[Saisai Shao]

Though we haven't tried such use case before, from my understanding I think
Livy cannot support it, and such usage scenario seems not a typical usage
pattern.

adley  于2020年3月8日周日 下午9:40写道：

> Hi：
> I followed the official example-PiApp.java, and tried to submit and
> run successfully.
> Then I modified the code and called client.uploadJar (newjar) again
> after client.submit, and finally called the class implementation in newjar,
> client.submit ( newjar-newclass), the upload is successful, no error is
> reported, but it has no effect. The execution result shows that the class
> implementation in oldjar is still called.
> Can Livy support the repeated submission of jar packages under one
> client?
>

Re: [Discuss] Merge spark-3 branch into master

[Saisai Shao]

I was thinking that if it is possible to limit version lock plugin to only
iceberg core related subprojects., seems like current consistent-versions
plugin doesn't allow to do so. So not sure if there're some other plugins
which could provide similar functionality with more flexibility?

 Any suggestions on this?

Best regards,
Saisai

Saisai Shao  于2020年3月5日周四 下午3:12写道：

> I think the requirement of supporting different version should be quite
> common. As Iceberg is a table format which should be adapted to different
> engines like Hive, Flink, Spark. To support different versions is a real
> problem, Spark is just one case, Hive, Flink could also be the case if the
> interface is changed across major versions. Also version lock may have
> problems when several engines coexisted in the same build, as they will
> transiently introduce lots of dependencies which may be conflicted, it may
> be hard to figure out one version which could satisfy all, and usually they
> only confined to a single module.
>
>  So I think we should figure out a way to support such scenario, not just
> maintaining branches one by one.
>
> Ryan Blue  于2020年3月5日周四 上午2:53写道：
>
>> I think the key is that this wouldn't be using the same published
>> artifacts. This work would create a spark-2.4 artifact and a spark-3.0
>> artifact. (And possibly a spark-common artifact.)
>>
>> It seems reasonable to me to have those in the same build instead of in
>> separate branches, as long as the Spark dependencies are not leaked outside
>> of the modules. That said, I'd rather have the additional checks that
>> baseline provides in general since this is a short-term problem. It would
>> just be nice if we could have versions that are confined to a single
>> module. The Nebula plugin that baseline uses claims to support that, but I
>> couldn't get it to work.
>>
>> On Wed, Mar 4, 2020 at 6:38 AM Saisai Shao 
>> wrote:
>>
>>> Just think a bit on this. I agree that generally introducing different
>>> versions of same dependencies could be error prone. But I think the case
>>> here should not lead to  issue:
>>>
>>> 1.  These two sub-modules spark-2 and spark-3 are isolated, they're not
>>> dependent on either.
>>> 2. They can be differentiated by names when generating jars, also they
>>> will not be relied by other modules in Iceberg.
>>>
>>> So this dependency issue should not be the case here. And in Maven it
>>> could be achieved easily. Please correct me if wrong.
>>>
>>> Best regards,
>>> Saisai
>>>
>>> Saisai Shao  于2020年3月4日周三 上午10:01写道：
>>>
>>>> Thanks Matt,
>>>>
>>>> If branching is the only choice, then we would potentially have two
>>>> *master* branches until spark-3 is vastly adopted. That will somehow
>>>> increase the maintenance burden and lead to inconsistency. IMO I'm OK with
>>>> the branching way, just think that we should have a clear way to keep
>>>> tracking of two branches.
>>>>
>>>> Best,
>>>> Saisai
>>>>
>>>> Matt Cheah  于2020年3月4日周三 上午9:50写道：
>>>>
>>>>> I think it’s generally dangerous and error-prone to try to support two
>>>>> versions of the same library in the same build, in the same published
>>>>> artifacts. This is the stance that Baseline
>>>>> <https://github.com/palantir/gradle-baseline> + Gradle Consistent
>>>>> Versions <https://github.com/palantir/gradle-consistent-versions>
>>>>> takes. Gradle Consistent Versions is specifically opinionated towards
>>>>> building against one version of a library across all modules in the build.
>>>>>
>>>>>
>>>>>
>>>>> I would think that branching would be the best way to build and
>>>>> publish against multiple versions of a dependency.
>>>>>
>>>>>
>>>>>
>>>>> -Matt Cheah
>>>>>
>>>>>
>>>>>
>>>>> *From: *Saisai Shao 
>>>>> *Reply-To: *"dev@iceberg.apache.org" 
>>>>> *Date: *Tuesday, March 3, 2020 at 5:45 PM
>>>>> *To: *Iceberg Dev List 
>>>>> *Cc: *Ryan Blue 
>>>>> *Subject: *Re: [Discuss] Merge spark-3 branch into master
>>>>>
>>>>>
>>>>>
>>>>> I didn't realized that Gradle cannot support two different versions in
>>>>> one build. I think I did such things for Livy to build

Re: [Discuss] Merge spark-3 branch into master

[Saisai Shao]

I think the requirement of supporting different version should be quite
common. As Iceberg is a table format which should be adapted to different
engines like Hive, Flink, Spark. To support different versions is a real
problem, Spark is just one case, Hive, Flink could also be the case if the
interface is changed across major versions. Also version lock may have
problems when several engines coexisted in the same build, as they will
transiently introduce lots of dependencies which may be conflicted, it may
be hard to figure out one version which could satisfy all, and usually they
only confined to a single module.

 So I think we should figure out a way to support such scenario, not just
maintaining branches one by one.

Ryan Blue  于2020年3月5日周四 上午2:53写道：

> I think the key is that this wouldn't be using the same published
> artifacts. This work would create a spark-2.4 artifact and a spark-3.0
> artifact. (And possibly a spark-common artifact.)
>
> It seems reasonable to me to have those in the same build instead of in
> separate branches, as long as the Spark dependencies are not leaked outside
> of the modules. That said, I'd rather have the additional checks that
> baseline provides in general since this is a short-term problem. It would
> just be nice if we could have versions that are confined to a single
> module. The Nebula plugin that baseline uses claims to support that, but I
> couldn't get it to work.
>
> On Wed, Mar 4, 2020 at 6:38 AM Saisai Shao  wrote:
>
>> Just think a bit on this. I agree that generally introducing different
>> versions of same dependencies could be error prone. But I think the case
>> here should not lead to  issue:
>>
>> 1.  These two sub-modules spark-2 and spark-3 are isolated, they're not
>> dependent on either.
>> 2. They can be differentiated by names when generating jars, also they
>> will not be relied by other modules in Iceberg.
>>
>> So this dependency issue should not be the case here. And in Maven it
>> could be achieved easily. Please correct me if wrong.
>>
>> Best regards,
>> Saisai
>>
>> Saisai Shao  于2020年3月4日周三 上午10:01写道：
>>
>>> Thanks Matt,
>>>
>>> If branching is the only choice, then we would potentially have two
>>> *master* branches until spark-3 is vastly adopted. That will somehow
>>> increase the maintenance burden and lead to inconsistency. IMO I'm OK with
>>> the branching way, just think that we should have a clear way to keep
>>> tracking of two branches.
>>>
>>> Best,
>>> Saisai
>>>
>>> Matt Cheah  于2020年3月4日周三 上午9:50写道：
>>>
>>>> I think it’s generally dangerous and error-prone to try to support two
>>>> versions of the same library in the same build, in the same published
>>>> artifacts. This is the stance that Baseline
>>>> <https://github.com/palantir/gradle-baseline> + Gradle Consistent
>>>> Versions <https://github.com/palantir/gradle-consistent-versions>
>>>> takes. Gradle Consistent Versions is specifically opinionated towards
>>>> building against one version of a library across all modules in the build.
>>>>
>>>>
>>>>
>>>> I would think that branching would be the best way to build and publish
>>>> against multiple versions of a dependency.
>>>>
>>>>
>>>>
>>>> -Matt Cheah
>>>>
>>>>
>>>>
>>>> *From: *Saisai Shao 
>>>> *Reply-To: *"dev@iceberg.apache.org" 
>>>> *Date: *Tuesday, March 3, 2020 at 5:45 PM
>>>> *To: *Iceberg Dev List 
>>>> *Cc: *Ryan Blue 
>>>> *Subject: *Re: [Discuss] Merge spark-3 branch into master
>>>>
>>>>
>>>>
>>>> I didn't realized that Gradle cannot support two different versions in
>>>> one build. I think I did such things for Livy to build scala 2.10 and 2.11
>>>> jars simultaneously with Maven. I'm not so familiar with Gradle thing, I
>>>> can take a shot to see if there's some hacky ways to make it work.
>>>>
>>>>
>>>>
>>>> Besides, are we saying that we will move to spark-3 support after 0.8
>>>> release in the master branch to replace Spark-2, or we maintain two
>>>> branches for both spark-2 and spark-3 and make two releases? From
>>>> my understanding, the adoption of spark-3 may not be so fast, and there
>>>> still has lots users who stick on spark-2. Ideally, it might be better to
>>>> support two versions in a near future.
>>>>
&

Re: [Discuss] Merge spark-3 branch into master

[Saisai Shao]

Just think a bit on this. I agree that generally introducing different
versions of same dependencies could be error prone. But I think the case
here should not lead to  issue:

1.  These two sub-modules spark-2 and spark-3 are isolated, they're not
dependent on either.
2. They can be differentiated by names when generating jars, also they will
not be relied by other modules in Iceberg.

So this dependency issue should not be the case here. And in Maven it could
be achieved easily. Please correct me if wrong.

Best regards,
Saisai

Saisai Shao  于2020年3月4日周三 上午10:01写道：

> Thanks Matt,
>
> If branching is the only choice, then we would potentially have two
> *master* branches until spark-3 is vastly adopted. That will somehow
> increase the maintenance burden and lead to inconsistency. IMO I'm OK with
> the branching way, just think that we should have a clear way to keep
> tracking of two branches.
>
> Best,
> Saisai
>
> Matt Cheah  于2020年3月4日周三 上午9:50写道：
>
>> I think it’s generally dangerous and error-prone to try to support two
>> versions of the same library in the same build, in the same published
>> artifacts. This is the stance that Baseline
>> <https://github.com/palantir/gradle-baseline> + Gradle Consistent
>> Versions <https://github.com/palantir/gradle-consistent-versions> takes.
>> Gradle Consistent Versions is specifically opinionated towards building
>> against one version of a library across all modules in the build.
>>
>>
>>
>> I would think that branching would be the best way to build and publish
>> against multiple versions of a dependency.
>>
>>
>>
>> -Matt Cheah
>>
>>
>>
>> *From: *Saisai Shao 
>> *Reply-To: *"dev@iceberg.apache.org" 
>> *Date: *Tuesday, March 3, 2020 at 5:45 PM
>> *To: *Iceberg Dev List 
>> *Cc: *Ryan Blue 
>> *Subject: *Re: [Discuss] Merge spark-3 branch into master
>>
>>
>>
>> I didn't realized that Gradle cannot support two different versions in
>> one build. I think I did such things for Livy to build scala 2.10 and 2.11
>> jars simultaneously with Maven. I'm not so familiar with Gradle thing, I
>> can take a shot to see if there's some hacky ways to make it work.
>>
>>
>>
>> Besides, are we saying that we will move to spark-3 support after 0.8
>> release in the master branch to replace Spark-2, or we maintain two
>> branches for both spark-2 and spark-3 and make two releases? From
>> my understanding, the adoption of spark-3 may not be so fast, and there
>> still has lots users who stick on spark-2. Ideally, it might be better to
>> support two versions in a near future.
>>
>>
>>
>> Thanks
>>
>> Saisai
>>
>>
>>
>>
>>
>>
>>
>> Mass Dosage  于2020年3月4日周三 上午1:33写道：
>>
>> +1 for a 0.8.0 release with Spark 2.4 and then move on for Spark 3.0 when
>> it's ready.
>>
>>
>>
>> On Tue, 3 Mar 2020 at 16:32, Ryan Blue  wrote:
>>
>> Thanks for bringing this up, Saisai. I tried to do this a couple of
>> months ago, but ran into a problem with dependency locks. I couldn't get
>> two different versions of Spark packages in the build with baseline, but
>> maybe I was missing something. If you can get it working, I think it's a
>> great idea to get this into master.
>>
>>
>>
>> Otherwise, I was thinking about proposing an 0.8.0 release in the next
>> month or so based on Spark 2.4. Then we could merge the branch into master
>> and do another release for Spark 3.0 when it's ready.
>>
>>
>>
>> rb
>>
>>
>>
>> On Tue, Mar 3, 2020 at 6:07 AM Saisai Shao 
>> wrote:
>>
>> Hi team,
>>
>>
>>
>> I was thinking of merging spark-3 branch into master, also per the
>> discussion before we could make spark-2 and spark-3 coexisted into 2
>> different sub-modules. With this, one build could generate both spark-2 and
>> spark-3 runtime jars, user could pick either at preference.
>>
>>
>>
>> One concern is that they share lots of common code in read/write path,
>> this will increase the maintenance overhead to keep consistency of two
>> copies.
>>
>>
>>
>> So I'd like to hear your thoughts, any suggestions on it?
>>
>>
>>
>> Thanks
>>
>> Saisai
>>
>>
>>
>>
>> --
>>
>> Ryan Blue
>>
>> Software Engineer
>>
>> Netflix
>>
>>

Re: [Discuss] Merge spark-3 branch into master

[Saisai Shao]

Thanks Matt,

If branching is the only choice, then we would potentially have two
*master* branches until spark-3 is vastly adopted. That will somehow
increase the maintenance burden and lead to inconsistency. IMO I'm OK with
the branching way, just think that we should have a clear way to keep
tracking of two branches.

Best,
Saisai

Matt Cheah  于2020年3月4日周三 上午9:50写道：

> I think it’s generally dangerous and error-prone to try to support two
> versions of the same library in the same build, in the same published
> artifacts. This is the stance that Baseline
> <https://github.com/palantir/gradle-baseline> + Gradle Consistent Versions
> <https://github.com/palantir/gradle-consistent-versions> takes. Gradle
> Consistent Versions is specifically opinionated towards building against
> one version of a library across all modules in the build.
>
>
>
> I would think that branching would be the best way to build and publish
> against multiple versions of a dependency.
>
>
>
> -Matt Cheah
>
>
>
> *From: *Saisai Shao 
> *Reply-To: *"dev@iceberg.apache.org" 
> *Date: *Tuesday, March 3, 2020 at 5:45 PM
> *To: *Iceberg Dev List 
> *Cc: *Ryan Blue 
> *Subject: *Re: [Discuss] Merge spark-3 branch into master
>
>
>
> I didn't realized that Gradle cannot support two different versions in one
> build. I think I did such things for Livy to build scala 2.10 and 2.11 jars
> simultaneously with Maven. I'm not so familiar with Gradle thing, I can
> take a shot to see if there's some hacky ways to make it work.
>
>
>
> Besides, are we saying that we will move to spark-3 support after 0.8
> release in the master branch to replace Spark-2, or we maintain two
> branches for both spark-2 and spark-3 and make two releases? From
> my understanding, the adoption of spark-3 may not be so fast, and there
> still has lots users who stick on spark-2. Ideally, it might be better to
> support two versions in a near future.
>
>
>
> Thanks
>
> Saisai
>
>
>
>
>
>
>
> Mass Dosage  于2020年3月4日周三 上午1:33写道：
>
> +1 for a 0.8.0 release with Spark 2.4 and then move on for Spark 3.0 when
> it's ready.
>
>
>
> On Tue, 3 Mar 2020 at 16:32, Ryan Blue  wrote:
>
> Thanks for bringing this up, Saisai. I tried to do this a couple of months
> ago, but ran into a problem with dependency locks. I couldn't get two
> different versions of Spark packages in the build with baseline, but maybe
> I was missing something. If you can get it working, I think it's a great
> idea to get this into master.
>
>
>
> Otherwise, I was thinking about proposing an 0.8.0 release in the next
> month or so based on Spark 2.4. Then we could merge the branch into master
> and do another release for Spark 3.0 when it's ready.
>
>
>
> rb
>
>
>
> On Tue, Mar 3, 2020 at 6:07 AM Saisai Shao  wrote:
>
> Hi team,
>
>
>
> I was thinking of merging spark-3 branch into master, also per the
> discussion before we could make spark-2 and spark-3 coexisted into 2
> different sub-modules. With this, one build could generate both spark-2 and
> spark-3 runtime jars, user could pick either at preference.
>
>
>
> One concern is that they share lots of common code in read/write path,
> this will increase the maintenance overhead to keep consistency of two
> copies.
>
>
>
> So I'd like to hear your thoughts, any suggestions on it?
>
>
>
> Thanks
>
> Saisai
>
>
>
>
> --
>
> Ryan Blue
>
> Software Engineer
>
> Netflix
>
>

Re: [Discuss] Merge spark-3 branch into master

[Saisai Shao]

I didn't realized that Gradle cannot support two different versions in one
build. I think I did such things for Livy to build scala 2.10 and 2.11 jars
simultaneously with Maven. I'm not so familiar with Gradle thing, I can
take a shot to see if there's some hacky ways to make it work.

Besides, are we saying that we will move to spark-3 support after 0.8
release in the master branch to replace Spark-2, or we maintain two
branches for both spark-2 and spark-3 and make two releases? From
my understanding, the adoption of spark-3 may not be so fast, and there
still has lots users who stick on spark-2. Ideally, it might be better to
support two versions in a near future.

Thanks
Saisai



Mass Dosage  于2020年3月4日周三 上午1:33写道：

> +1 for a 0.8.0 release with Spark 2.4 and then move on for Spark 3.0 when
> it's ready.
>
> On Tue, 3 Mar 2020 at 16:32, Ryan Blue  wrote:
>
>> Thanks for bringing this up, Saisai. I tried to do this a couple of
>> months ago, but ran into a problem with dependency locks. I couldn't get
>> two different versions of Spark packages in the build with baseline, but
>> maybe I was missing something. If you can get it working, I think it's a
>> great idea to get this into master.
>>
>> Otherwise, I was thinking about proposing an 0.8.0 release in the next
>> month or so based on Spark 2.4. Then we could merge the branch into master
>> and do another release for Spark 3.0 when it's ready.
>>
>> rb
>>
>> On Tue, Mar 3, 2020 at 6:07 AM Saisai Shao 
>> wrote:
>>
>>> Hi team,
>>>
>>> I was thinking of merging spark-3 branch into master, also per the
>>> discussion before we could make spark-2 and spark-3 coexisted into 2
>>> different sub-modules. With this, one build could generate both spark-2 and
>>> spark-3 runtime jars, user could pick either at preference.
>>>
>>> One concern is that they share lots of common code in read/write path,
>>> this will increase the maintenance overhead to keep consistency of two
>>> copies.
>>>
>>> So I'd like to hear your thoughts, any suggestions on it?
>>>
>>> Thanks
>>> Saisai
>>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>

[Discuss] Merge spark-3 branch into master

[Saisai Shao]

Hi team,

I was thinking of merging spark-3 branch into master, also per the
discussion before we could make spark-2 and spark-3 coexisted into 2
different sub-modules. With this, one build could generate both spark-2 and
spark-3 runtime jars, user could pick either at preference.

One concern is that they share lots of common code in read/write path, this
will increase the maintenance overhead to keep consistency of two copies.

So I'd like to hear your thoughts, any suggestions on it?

Thanks
Saisai

[jira] [Resolved] (LIVY-748) Add support for running Livy Integration tests against secure external clusters

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-748.
--
Fix Version/s: 0.8.0
 Assignee: Roger Liu
   Resolution: Fixed

Issue resolved by pull request 278
https://github.com/apache/incubator-livy/pull/278

> Add support for running Livy Integration tests against secure external 
> clusters
> ---
>
> Key: LIVY-748
> URL: https://issues.apache.org/jira/browse/LIVY-748
> Project: Livy
>  Issue Type: Improvement
>Reporter: Roger Liu
>Assignee: Roger Liu
>Priority: Major
> Fix For: 0.8.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Add support so Livy integration tests can be run against secure external 
> clusters. Currently Livy integration tests only test Livy functionality 
> against a minicluster and does not support running them against an external 
> livy endpoint



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (LIVY-423) Adding Scala 2.12 support

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17045049#comment-17045049
 ] 

Saisai Shao commented on LIVY-423:
--

We don't typically assign JIRA to someone before it merges. I think you could 
leave message here in this JIRA that you're working on this, once the feature 
is merged, we will assign this to you.

> Adding Scala 2.12 support
> -
>
> Key: LIVY-423
> URL: https://issues.apache.org/jira/browse/LIVY-423
> Project: Livy
>  Issue Type: New Feature
>  Components: Build, Core, REPL
>    Reporter: Saisai Shao
>Priority: Major
>
> Spark 2.3 already integrates with Scala 2.12 support, it will possibly 
> release 2.12 artifacts. So in the Livy side we should support Scala 2.12 
> build and interpreter. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (LIVY-423) Adding Scala 2.12 support

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044012#comment-17044012
 ] 

Saisai Shao commented on LIVY-423:
--

No update on it. There's no one working on this feature, you can take it if 
you're interested.

> Adding Scala 2.12 support
> -
>
> Key: LIVY-423
> URL: https://issues.apache.org/jira/browse/LIVY-423
> Project: Livy
>  Issue Type: New Feature
>  Components: Build, Core, REPL
>    Reporter: Saisai Shao
>Priority: Major
>
> Spark 2.3 already integrates with Scala 2.12 support, it will possibly 
> release 2.12 artifacts. So in the Livy side we should support Scala 2.12 
> build and interpreter. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (SPARK-30586) NPE in LiveRDDDistribution (AppStatusListener)

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-30586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036734#comment-17036734
 ] 

Saisai Shao commented on SPARK-30586:
-

We also met the same issue. Seems like the code doesn't check the nullable of 
string and directly called String intern, which throws NPE from guava. My first 
thinking is to add nullable check in {{weakIntern}}. Still investigating how 
this could be happened, might be due to the lost or out-of-order spark listener 
event.

> NPE in LiveRDDDistribution (AppStatusListener)
> --
>
> Key: SPARK-30586
> URL: https://issues.apache.org/jira/browse/SPARK-30586
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.4.4
> Environment: A Hadoop cluster consisting of Centos 7.4 machines.
>Reporter: Jan Van den bosch
>Priority: Major
>
> We've been noticing a great amount of NullPointerExceptions in our 
> long-running Spark job driver logs:
> {noformat}
> 20/01/17 23:40:12 ERROR AsyncEventQueue: Listener AppStatusListener threw an 
> exception
> java.lang.NullPointerException
> at 
> org.spark_project.guava.base.Preconditions.checkNotNull(Preconditions.java:191)
> at 
> org.spark_project.guava.collect.MapMakerInternalMap.putIfAbsent(MapMakerInternalMap.java:3507)
> at 
> org.spark_project.guava.collect.Interners$WeakInterner.intern(Interners.java:85)
> at 
> org.apache.spark.status.LiveEntityHelpers$.weakIntern(LiveEntity.scala:603)
> at 
> org.apache.spark.status.LiveRDDDistribution.toApi(LiveEntity.scala:486)
> at 
> org.apache.spark.status.LiveRDD$$anonfun$2.apply(LiveEntity.scala:548)
> at 
> org.apache.spark.status.LiveRDD$$anonfun$2.apply(LiveEntity.scala:548)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at 
> scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:139)
> at 
> scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:139)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap$$anon$2.foreach(HashMap.scala:139)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> at org.apache.spark.status.LiveRDD.doUpdate(LiveEntity.scala:548)
> at org.apache.spark.status.LiveEntity.write(LiveEntity.scala:49)
> at 
> org.apache.spark.status.AppStatusListener.org$apache$spark$status$AppStatusListener$$update(AppStatusListener.scala:991)
> at 
> org.apache.spark.status.AppStatusListener.org$apache$spark$status$AppStatusListener$$maybeUpdate(AppStatusListener.scala:997)
> at 
> org.apache.spark.status.AppStatusListener$$anonfun$onExecutorMetricsUpdate$2.apply(AppStatusListener.scala:764)
> at 
> org.apache.spark.status.AppStatusListener$$anonfun$onExecutorMetricsUpdate$2.apply(AppStatusListener.scala:764)
> at 
> scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:139)
> at 
> scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:139)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap$$anon$2.foreach(HashMap.scala:139)
> at 
> org.apache.spark.status.AppStatusListener.org$apache$spark$status$AppStatusListener$$flush(AppStatusListener.scala:788)
> at 
> org.apache.spark.status.AppStatusListener.onExecutorMetricsUpdate(AppStatusListener.scala:764)
> at 
> org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:59)
> at 
> org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
> at 
> org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
> at 
> org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:91)
> at 
> org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$super$postToAll(AsyncEventQueue.scala:92)
> at 
> org.apache.spark.scheduler.AsyncEventQueue$$anonfun$o

[jira] [Comment Edited] (SPARK-30586) NPE in LiveRDDDistribution (AppStatusListener)

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-30586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036734#comment-17036734
 ] 

Saisai Shao edited comment on SPARK-30586 at 2/14/20 7:13 AM:
--

We also met the same issue. Seems like the code doesn't check the nullable of 
string and directly called String intern, which throws NPE from guava. My first 
thinking is to add nullable check in {{weakIntern}}. Still investigating how 
this could be happened, might be due to the lost or out-of-order spark listener 
event.

CC [~vanzin]


was (Author: jerryshao):
We also met the same issue. Seems like the code doesn't check the nullable of 
string and directly called String intern, which throws NPE from guava. My first 
thinking is to add nullable check in {{weakIntern}}. Still investigating how 
this could be happened, might be due to the lost or out-of-order spark listener 
event.

> NPE in LiveRDDDistribution (AppStatusListener)
> --
>
> Key: SPARK-30586
> URL: https://issues.apache.org/jira/browse/SPARK-30586
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.4.4
> Environment: A Hadoop cluster consisting of Centos 7.4 machines.
>Reporter: Jan Van den bosch
>Priority: Major
>
> We've been noticing a great amount of NullPointerExceptions in our 
> long-running Spark job driver logs:
> {noformat}
> 20/01/17 23:40:12 ERROR AsyncEventQueue: Listener AppStatusListener threw an 
> exception
> java.lang.NullPointerException
> at 
> org.spark_project.guava.base.Preconditions.checkNotNull(Preconditions.java:191)
> at 
> org.spark_project.guava.collect.MapMakerInternalMap.putIfAbsent(MapMakerInternalMap.java:3507)
> at 
> org.spark_project.guava.collect.Interners$WeakInterner.intern(Interners.java:85)
> at 
> org.apache.spark.status.LiveEntityHelpers$.weakIntern(LiveEntity.scala:603)
> at 
> org.apache.spark.status.LiveRDDDistribution.toApi(LiveEntity.scala:486)
> at 
> org.apache.spark.status.LiveRDD$$anonfun$2.apply(LiveEntity.scala:548)
> at 
> org.apache.spark.status.LiveRDD$$anonfun$2.apply(LiveEntity.scala:548)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at 
> scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:139)
> at 
> scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:139)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap$$anon$2.foreach(HashMap.scala:139)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> at org.apache.spark.status.LiveRDD.doUpdate(LiveEntity.scala:548)
> at org.apache.spark.status.LiveEntity.write(LiveEntity.scala:49)
> at 
> org.apache.spark.status.AppStatusListener.org$apache$spark$status$AppStatusListener$$update(AppStatusListener.scala:991)
> at 
> org.apache.spark.status.AppStatusListener.org$apache$spark$status$AppStatusListener$$maybeUpdate(AppStatusListener.scala:997)
> at 
> org.apache.spark.status.AppStatusListener$$anonfun$onExecutorMetricsUpdate$2.apply(AppStatusListener.scala:764)
> at 
> org.apache.spark.status.AppStatusListener$$anonfun$onExecutorMetricsUpdate$2.apply(AppStatusListener.scala:764)
> at 
> scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:139)
> at 
> scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:139)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap$$anon$2.foreach(HashMap.scala:139)
> at 
> org.apache.spark.status.AppStatusListener.org$apache$spark$status$AppStatusListener$$flush(AppStatusListener.scala:788)
> at 
> org.apache.spark.status.AppStatusListener.onExecutorMetricsUpdate(AppStatusListener.scala:764)
> at 
> org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:59)
> at 
> org.apache.spark.scheduler.AsyncEventQueue

Re: About building and releasing livy java/scala doc

[Saisai Shao]

Just figured out there's a doc about it. Thanks!

Best regards,
Saisai

Saisai Shao  于2020年2月2日周日 下午7:15写道：

> Hi Team,
>
> Do you know how to build and release java/scala doc for new Livy (0.7.0)
> release. I think document thing is the only remaining thing for 0.7.0
> release. Please help to guide, would be appreciated if there's doc about it.
>
> Thanks
> Saisai
>

[ANNOUNCE] Apache Livy 0.7.0-incubating released

[Saisai Shao]

The Apache Livy team is proud to announce the release of Apache Livy
0.7.0-incubating.

Livy is web service that exposes a REST interface for managing long
running Apache Spark contexts in your cluster. Livy enables
programmatic, fault-tolerant, multi-tenant submission of Spark jobs
from web/mobile apps (no Spark client needed). So, multiple users can
interact with your Spark cluster concurrently and reliably.

Download Apache Livy 0.7.0-incubating:
http://livy.incubator.apache.org/download/

Release Notes:
http://livy.incubator.apache.org/history/

For more about Livy check our website:
http://livy.incubator.apache.org/

We would like to thank the contributors that made the release possible!

=
*Disclaimer*

Apache Livy (incubating) is an effort undergoing incubation at The
Apache Software Foundation (ASF), sponsored by the name of Apache
Incubator PMC. Incubation is required of all newly accepted projects
until a further review indicates that the infrastructure,
communications, and decision making process have
stabilized in a manner consistent with other successful ASF projects.
While incubation status is not necessarily a reflection of the
completeness or stability of the code, it does indicate that the
project has yet to be fully endorsed by the ASF.

[ANNOUNCE] Apache Livy 0.7.0-incubating released

[Saisai Shao]

The Apache Livy team is proud to announce the release of Apache Livy
0.7.0-incubating.

Livy is web service that exposes a REST interface for managing long
running Apache Spark contexts in your cluster. Livy enables
programmatic, fault-tolerant, multi-tenant submission of Spark jobs
from web/mobile apps (no Spark client needed). So, multiple users can
interact with your Spark cluster concurrently and reliably.

Download Apache Livy 0.7.0-incubating:
http://livy.incubator.apache.org/download/

Release Notes:
http://livy.incubator.apache.org/history/

For more about Livy check our website:
http://livy.incubator.apache.org/

We would like to thank the contributors that made the release possible!

=
*Disclaimer*

Apache Livy (incubating) is an effort undergoing incubation at The
Apache Software Foundation (ASF), sponsored by the name of Apache
Incubator PMC. Incubation is required of all newly accepted projects
until a further review indicates that the infrastructure,
communications, and decision making process have
stabilized in a manner consistent with other successful ASF projects.
While incubation status is not necessarily a reflection of the
completeness or stability of the code, it does indicate that the
project has yet to be fully endorsed by the ASF.

About building and releasing livy java/scala doc

[Saisai Shao]

Hi Team,

Do you know how to build and release java/scala doc for new Livy (0.7.0)
release. I think document thing is the only remaining thing for 0.7.0
release. Please help to guide, would be appreciated if there's doc about it.

Thanks
Saisai

Re: [RESULT][VOTE] Apache Livy (incubating) 0.7.0 RC4

[Saisai Shao]

Hi all,

(amending the typo in previous mail).

The voting for releasing Apache Livy (incubating) 0.7.0 RC4 has passed with
3 +1 binding votes and no -1 votes.

+1 Vote from:

   -  Junping Du (binding)
   - Justin Mclean (binding)
   - Bikas Saha (binding)

Voting Thread:

https://lists.apache.org/thread.html/r02710b9b34f5dfbdadeaeb73305a2da345b74f3623b597420ff45ae7%40%3Cgeneral.incubator.apache.org%3E

We will continue the release process and soon follow with the announcement
email for the release to general@incubator.

Thanks,
Saisai

Saisai Shao  于2020年2月2日周日 上午10:02写道：

> Hi all,
>
> The voting for releasing Apache Hudi (incubating) 0.7.0 RC4 has passed with
> 3 +1 binding votes and no -1 votes.
>
> +1 Vote from:
>
>-  Junping Du (binding)
>- Justin Mclean (binding)
>- Bikas Saha (binding)
>
> Voting Thread:
>
>
> https://lists.apache.org/thread.html/r02710b9b34f5dfbdadeaeb73305a2da345b74f3623b597420ff45ae7%40%3Cgeneral.incubator.apache.org%3E
>
> We will continue the release process and soon follow with the announcement
> email for the release to general@incubator.
>
> Thanks,
> Saisai
>

[RESULT][VOTE] Apache Livy (incubating) 0.7.0 RC4

[Saisai Shao]

Hi all,

The voting for releasing Apache Hudi (incubating) 0.7.0 RC4 has passed with
3 +1 binding votes and no -1 votes.

+1 Vote from:

   -  Junping Du (binding)
   - Justin Mclean (binding)
   - Bikas Saha (binding)

Voting Thread:

https://lists.apache.org/thread.html/r02710b9b34f5dfbdadeaeb73305a2da345b74f3623b597420ff45ae7%40%3Cgeneral.incubator.apache.org%3E

We will continue the release process and soon follow with the announcement
email for the release to general@incubator.

Thanks,
Saisai

Re: Running local and yarn using the same livy server

[Saisai Shao]

I don't think current Livy support such behavior, the cluster manager
specified in the conf file is a global configuration which affects all the
created sessions.

Thanks
Saisai

Ravi Shankar  于2020年1月28日周二 上午4:02写道：

> Hey guys,
> Is there a way to start different kind of spark sessions using the same
> livy server ? For instance, i want my application to tell livy whether the
> new session being requested should be started as a "local" session or a
> "yarn" session on the cluster.
>
> Ravi
>

Re: python-api CI failure

[Saisai Shao]

Thanks for the fix. Would you please create a separate PR about this issue?
It should be enough to create a "minor" GH PR directly without JIRA issue.

Best regards,
Saisai

Wing Yew Poon  于2020年1月31日周五 上午6:33写道：

> I updated python-api/setup.py to include
>
> 'mock~=3.0.5',
>
> in requirements.
> That got the CI passing on my PR,
> https://github.com/apache/incubator-livy/pull/275.
>
> On Thu, Jan 30, 2020 at 12:02 PM Wing Yew Poon 
> wrote:
> >
> > Resending ...
> >
> > On Wed, Jan 29, 2020 at 9:26 PM Wing Yew Poon 
> wrote:
> > >
> > > Hi,
> > > I ran into a CI failure in the python-api unit tests. (See
> > > https://travis-ci.org/apache/incubator-livy/builds/643685980.)
> > > From what I can tell, something is pulling in mock 4.0.0b1 (in
> > > pre-release according to https://pypi.org/project/mock/#history)
> > > instead of 3.0.5, and a method signature has changed in mock.py.
> > > Any guidance on how I can get around this?
> > > Thanks,
> > > Wing Yew
> >
> > From https://pypi.org/project/mock/#history it appears that mock
> > 4.0.0b1 showed up Jan 29, 2020, which is why this failure only just
> > started happening.
>

Re: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC4

[Saisai Shao]

Anyone please help to vote on this thread, greatly appreciated!

Thanks
Saisai



Justin Mclean 于2020年1月22日 周三09:59写道：

> Hi
>
> > About the missing license for the font, we've already investigated a bit
> on
> > this, below is the conclusion. In this release I would like to leave as
> it
> > is.
>
> As these fronts are binary files and don’t have an license header it could
> confuse people who use your software to what their license is. I would
> suggest mentioning the license in LICENSE.
>
> > Based on https://www.glyphicons.com/license/ we do not need to directly
> >> mention the licensing for those fonts.
>
> IMO ASF policy says otherwise. Also the terms of that license are probably
> not compatible with the Apache license. Notice for instance "You cannot use
> icons in any product intended for further processing or where your customer
> is not a final owner.” Also that is actually not the license they are under
> as Glyphicons use to have a other options and you’re using an older
> version. I suggest you read this [1] and this [2].
>
> Thanks,
> Justin
>
> 1. https://github.com/twbs/bootstrap/issues/3942
> 2. https://glyphicons.com/old/license.html#old-halflings-bootstrap
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC4

[Saisai Shao]

Thanks Justin for the reply.

About the missing license for the font, we've already investigated a bit on
this, below is the conclusion. In this release I would like to leave as it
is.

Based on https://www.glyphicons.com/license/ we do not need to directly
> mention the licensing for those fonts. This is again backed up by the
> comment at the top of the bootstrap page
> https://getbootstrap.com/docs/3.3/components/ which is where our usage of
> the fonts/icons comes from. Based on these links we are fine as is but
> could also add a mention somewhere in our docs/licenses to the glyphicons
> website. Another reference would be this PR for the retired Apache project
> Falcon which also used these fonts
> https://issues.apache.org/jira/browse/FALCON-2110


Thanks
Saisai

Justin Mclean  于2020年1月22日周三 上午6:59写道：

> Hi,
>
> +1 binding
>
> I checked:
> - incubating in name
> - signatures and hashes are fine
> - DISCLAIMER exists
> - LICENSE is missing a license
> - NOTICE incorrect states "2018 and onwards” please fix
> - No unexpected binary files
> - All source files have ASF headers
> - unable to compile from source bu assume it my setup (on OSX)
>
> LICENSE is missing license for this font [1][2]
>
> Thanks,
> Justin
>
> 1.
> ./docs/assets/themes/apache/bootstrap/fonts/glyphicons-halflings-regular.*
> 2.
> ./server/src/main/resources/org/apache/livy/server/ui/static/fonts/glyphicons-halflings-regular.*
>
>
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC4

[Saisai Shao]

Can any IPMC/mentor help to vote on this? Thanks a ton.

Best regards,
Saisai

Saisai Shao  于2020年1月15日周三 下午5:48写道：

> The Livy PPMC has voted to release Livy 0.7.0 RC4 as the next Livy release.
>
> Livy enables programmatic, fault-tolerant, multi-tenant submission of
> Spark jobs from web/mobile apps (no Spark client needed). So, multiple
> users can interact with your Spark cluster concurrently and reliably.
>
> Vote thread:
>
> https://lists.apache.org/thread.html/r171f4a079f2c27c9039aaee1083ab6adb3ba516234a69bfb2d9c60fd%40%3Cdev.livy.apache.org%3E
>
> Result thread:
>
> https://lists.apache.org/thread.html/rc5524835a62dec8b94b2a827dcb32ba8bdfa5a8576bd438d1835026e%40%3Cdev.livy.apache.org%3E
>
> The RC is based on tag v0.7.0-incubating-rc4:
>
> https://github.com/apache/incubator-livy/commit/664503355d8bae763989ebac087122e306239d54
>
> The release files can be found here:
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc4/
>
> The staged maven artifacts can be found here:
> https://repository.apache.org/content/repositories/orgapachelivy-1013
>
> The list of resolved JIRAs in this release can be found here:
> https://issues.apache.org/jira/projects/LIVY/versions/12345179
>
> Please help to vote. Thanks!
>

[VOTE] Release Apache Livy 0.7.0 (incubating) based on RC4

[Saisai Shao]

The Livy PPMC has voted to release Livy 0.7.0 RC4 as the next Livy release.

Livy enables programmatic, fault-tolerant, multi-tenant submission of
Spark jobs from web/mobile apps (no Spark client needed). So, multiple
users can interact with your Spark cluster concurrently and reliably.

Vote thread:
https://lists.apache.org/thread.html/r171f4a079f2c27c9039aaee1083ab6adb3ba516234a69bfb2d9c60fd%40%3Cdev.livy.apache.org%3E

Result thread:
https://lists.apache.org/thread.html/rc5524835a62dec8b94b2a827dcb32ba8bdfa5a8576bd438d1835026e%40%3Cdev.livy.apache.org%3E

The RC is based on tag v0.7.0-incubating-rc4:
https://github.com/apache/incubator-livy/commit/664503355d8bae763989ebac087122e306239d54

The release files can be found here:
https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc4/

The staged maven artifacts can be found here:
https://repository.apache.org/content/repositories/orgapachelivy-1013

The list of resolved JIRAs in this release can be found here:
https://issues.apache.org/jira/projects/LIVY/versions/12345179

Please help to vote. Thanks!

Re: Bootstrapping process

[Saisai Shao]

IIUC, the current block issue is SGA thing. And I think from code/doc
level, we also have several other things:


1. Change to use Apache License V2 and enable RAT check
2. Refactor README to remove any proprietary word

3. Fix the version number and add versioning track in JIRA.
4. Bootstrap apache tubemq website ASAP.
5. Add document about how to contribute to formalize the way of
contributing .
6. Add document about how to commit.


Some of them may not be blocking issues, but I think it is necessary to fix.


Thanks

Saisai

Justin Mclean  于2020年1月15日周三 上午9:08写道：

> Hi,
>
> A couple of weeks back I posted an email asking about progress of this
> podling.
>
> I can see a lot of work has been done but there still some more to do:
>
> - Submitting the software grant?
> This doesn’t seem to have been submitted yet, is that correct?
>
> - Moving the repos over to ASF infrastructure.
> Work was done on this before the SGA was submitted. This is not ideal.
>
> - Moving the website over to ASF infrastructure
> I’ve got conflicting information on what happening here. There is
> development off list and a POC being shown onlist. I’ve received no answer
> about the incubator requirements for this website or where this work is
> being done.
>
> - Having all PPMC members sign ICLAs
> I believe this has been completed.
>
> - Adding all PPMC members to the roster
> I believe this has been completed.
>
> - Having all PPMC members signed up to the private list
> A couple of PPMC members are still not signed up.
>
> Anyone add anything to the above?
>
> Thanks,
> Justin

Re: [VOTE] Release Livy 0.7.0 based on RC4

[Saisai Shao]

Thanks all,

We have 3 +1 (binding), 3 +1 (non-binding), vote is passed. I will start
another vote on incubating general.

Thanks
Saisai

Jeff Zhang  于2020年1月14日周二 下午12:07写道：

> +1
>
> Bikas Saha  于2020年1月11日周六 下午9:40写道：
>
> > +1
> >
> > Bikas
> > 
> > From: mingchao zhao 
> > Sent: Wednesday, January 8, 2020 6:16 PM
> > To: dev@livy.incubator.apache.org 
> > Subject: Re: [VOTE] Release Livy 0.7.0 based on RC4
> >
> > Tested the general functionality of apache-livy-0.7.0-incubat-bin.zip
> and
> > looks good.
> >
> > +1
> >
> > On Wed, Jan 8, 2020 at 8:07 PM Yiheng Wang  wrote:
> >
> > > Check the release files. Looks good.
> > >
> > > +1
> > >
> > > On Tue, Jan 7, 2020 at 9:25 PM Saisai Shao 
> > wrote:
> > >
> > > > This vote is for releasing Livy 0.7.0 based on RC4.
> > > >
> > > > This RC mainly fixes the license issue.
> > > >
> > > > The vote will be open until Sunday  Jan 12, 23:59 UTC and
> > > > will pass with a minimum of 3 +1 binding votes and a majority of
> > positive
> > > > votes.
> > > >
> > > > [+1] This release is ready to send to the Incubator PMC for approval
> > > >
> > > > [-1] This release is not ready because...
> > > >
> > > > This vote is held according to Apache Incubator release policy:
> > > > https://incubator.apache.org/policy/incubation.html#releases
> > > >
> > > > The RC is based on tag v0.7.0-incubating-rc4:
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-livy/commit/664503355d8bae763989ebac087122e306239d54
> > > >
> > > > The release files can be found here:
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc4/
> > > >
> > > > The staged maven artifacts can be found here:
> > > >
> https://repository.apache.org/content/repositories/orgapachelivy-1013
> > > >
> > > > The list of resolved JIRAs in this release can be found here:
> > > > https://issues.apache.org/jira/projects/LIVY/versions/12345179
> > > >
> > > > Thanks
> > > > Saisai
> > > >
> > >
> >
>
>
> --
> Best Regards
>
> Jeff Zhang
>

[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014846#comment-17014846
 ] 

Saisai Shao commented on LIVY-718:
--

Active-active HA doesn't only address scalability issue, but also high 
availability. 

Personally I don't feel super useful for active-standby HA about Livy. Usually 
it is because master node has large amount of state to maintain, so it is hard 
to implement active-active HA with consistency. If this is not the case, then 
active-active HA is better both for HA and scalability. 

> Support multi-active high availability in Livy
> --
>
> Key: LIVY-718
> URL: https://issues.apache.org/jira/browse/LIVY-718
> Project: Livy
>  Issue Type: Epic
>  Components: RSC, Server
>Reporter: Yiheng Wang
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In this JIRA we want to discuss how to implement multi-active high 
> availability in Livy.
> Currently, Livy only supports single node recovery. This is not sufficient in 
> some production environments. In our scenario, the Livy server serves many 
> notebook and JDBC services. We want to make Livy service more fault-tolerant 
> and scalable.
> There're already some proposals in the community for high availability. But 
> they're not so complete or just for active-standby high availability. So we 
> propose a multi-active high availability design to achieve the following 
> goals:
> # One or more servers will serve the client requests at the same time.
> # Sessions are allocated among different servers.
> # When one node crashes, the affected sessions will be moved to other active 
> services.
> Here's our design document, please review and comment:
> https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: .NET for Apache Spark interactive interpreter support

[Saisai Shao]

I see your point. Personally I don't have a strong opinion on this, I'm not
sure what others think about it. Why don't you start a design doc and call
for a vote about this feature.

Thanks
Saisai

Steve Suh  于2020年1月14日周二 上午10:12写道：

> @Saisai
>
> This will not be a wrapper around the REST API.  The plan is to support a
> Livy POST /sessions request where kind == "sparkdotnet".  Livy will
> eventually push this request down to the ReplDriver, and in turn the
> ReplDriver will create a new SparkDotnet Interpreter class.  This will be
> similar to how the  PythonInterpreter, SparkRInterpreter, and
> SQLInterpreter classes get instantiated and used.
>
>
> Regards,
> -Steve
>
> On Sun, Jan 12, 2020 at 5:45 PM Saisai Shao 
> wrote:
>
> > Is this just a wrapper of Livy REST API, or it could support Livy Job
> API?
> >
> > From my opinion, if it is just a wrapper of REST API, then it would be
> > better to maintain out of Livy, since REST API is language independent,
> if
> > we're going to have all the languages support in Livy, then it is hard to
> > maintain.
> >
> > Just my two cents.
> >
> > Thanks
> > Saisai
> >
> > Steve Suh  于2020年1月12日周日 下午3:49写道：
> >
> > > Hi,
> > >
> > > I contribute to the .NET for Apache Spark <
> > https://github.com/dotnet/spark
> > > >
> > > project and we have seen *a lot* of *user interest* in providing first
> > > class notebook support for *.NET for Apache Spark*.
> > >
> > > Livy currently supports *Scala*, *Python *and *R **interactive
> sessions*.
> > > We have a working prototype
> > > <
> > >
> >
> https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks
> > > >
> > > available
> > > that adds support for a Spark *Dotnet **interactive session* and I
> would
> > > like to know if the Livy community would be interested in adding this
> > > feature to the main code base.  If so, please let me know and I can
> work
> > on
> > > creating this PR.
> > >
> > > For now, I've created a Jira item
> > > <https://issues.apache.org/jira/browse/LIVY-742> to track this.
> > >
> > >
> > > Regards,
> > > -Steve
> > >
> >
>

[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013965#comment-17013965
 ] 

Saisai Shao commented on LIVY-718:
--

[~bikassaha] The merged two sub-tasks, they're actually required by both 
solutions (the one you proposed and another [~yihengw] proposed). That's why I 
merged it beforehand, they're not the key differences for two solutions.

Actually the proposal [~yihengw] made is just the mid-term solution compared to 
stateless Livy Server, the key difference is to:

1. change the time when RSCDriver and Livy Server get reconnection.
2. Refactor the most of the current code to make Livy Server stateless.

I'm more concerned about the 2nd point, because it has lots of works to do, and 
could easily introduce regressions. So IMHO, I think we could move on with the 
current mid-term proposal. If someone else want to pursue a stateless solution, 
they could simply continue based on our current solution, that would take less 
efforts compared to start from scratch.

Just my two cents.


> Support multi-active high availability in Livy
> --
>
> Key: LIVY-718
> URL: https://issues.apache.org/jira/browse/LIVY-718
> Project: Livy
>  Issue Type: Epic
>  Components: RSC, Server
>Reporter: Yiheng Wang
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In this JIRA we want to discuss how to implement multi-active high 
> availability in Livy.
> Currently, Livy only supports single node recovery. This is not sufficient in 
> some production environments. In our scenario, the Livy server serves many 
> notebook and JDBC services. We want to make Livy service more fault-tolerant 
> and scalable.
> There're already some proposals in the community for high availability. But 
> they're not so complete or just for active-standby high availability. So we 
> propose a multi-active high availability design to achieve the following 
> goals:
> # One or more servers will serve the client requests at the same time.
> # Sessions are allocated among different servers.
> # When one node crashes, the affected sessions will be moved to other active 
> services.
> Here's our design document, please review and comment:
> https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: .NET for Apache Spark interactive interpreter support

[Saisai Shao]

Is this just a wrapper of Livy REST API, or it could support Livy Job API?

>From my opinion, if it is just a wrapper of REST API, then it would be
better to maintain out of Livy, since REST API is language independent, if
we're going to have all the languages support in Livy, then it is hard to
maintain.

Just my two cents.

Thanks
Saisai

Steve Suh  于2020年1月12日周日 下午3:49写道：

> Hi,
>
> I contribute to the .NET for Apache Spark  >
> project and we have seen *a lot* of *user interest* in providing first
> class notebook support for *.NET for Apache Spark*.
>
> Livy currently supports *Scala*, *Python *and *R **interactive sessions*.
> We have a working prototype
> <
> https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks
> >
> available
> that adds support for a Spark *Dotnet **interactive session* and I would
> like to know if the Livy community would be interested in adding this
> feature to the main code base.  If so, please let me know and I can work on
> creating this PR.
>
> For now, I've created a Jira item
>  to track this.
>
>
> Regards,
> -Steve
>

Re: Livy metrics

[Saisai Shao]

Currently we don't have concrete plan on this, it would be great if you
could contribute on this.

Thanks
Saisai

Shanyu Zhao  于2020年1月10日周五 上午8:44写道：

> Hi,
>
> Are there ongoing effort to add Livy metrics? I saw that dropwizard is
> added as dependency in pom.xml but didn't see it get used anywhere.
>
> Thanks,
> Shanyu
>

[DISCUSS] About the version number of TubeMQ

[Saisai Shao]

I see there're three open issues about about TubeMQ version number:

1. The version number is x.y-SNASHOT, typically it would be x.y.z-SNAPHOST
2. For incubating project, it should have "incubating" suffix,
x.y.z-incubating-SNAPSHOT.
3. There's no version number tracking for each JIRA.

So I think we should properly choose an initial version and fix the above
issues. Can anyone volunteer to fix them.

Thanks
Saisai

[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011485#comment-17011485
 ] 

Saisai Shao commented on LIVY-718:
--

Hi [~shanyu] what is the main reason that we should have "active-standby" HA? 
From my understanding, looks like compared to active-active HA, active-standby 
HA seems not so useful, and current fail recovery could cover most of the 
scenarios.

> Support multi-active high availability in Livy
> --
>
> Key: LIVY-718
> URL: https://issues.apache.org/jira/browse/LIVY-718
> Project: Livy
>  Issue Type: Epic
>  Components: RSC, Server
>Reporter: Yiheng Wang
>Priority: Major
>
> In this JIRA we want to discuss how to implement multi-active high 
> availability in Livy.
> Currently, Livy only supports single node recovery. This is not sufficient in 
> some production environments. In our scenario, the Livy server serves many 
> notebook and JDBC services. We want to make Livy service more fault-tolerant 
> and scalable.
> There're already some proposals in the community for high availability. But 
> they're not so complete or just for active-standby high availability. So we 
> propose a multi-active high availability design to achieve the following 
> goals:
> # One or more servers will serve the client requests at the same time.
> # Sessions are allocated among different servers.
> # When one node crashes, the affected sessions will be moved to other active 
> services.
> Here's our design document, please review and comment:
> https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Change TubeMQ's License to use Apache License V2 and enable RAT check

[Saisai Shao]

Hi Goson, why don't you create a JIRA to track this issue?

俊平堵  于2020年1月8日周三 下午5:17写道：

> +1. I think we *must* use standard Apache license given this is an apache
> project.
> Would like to hear other mentors' opinion here as well.
>
> Thanks,
>
> Junping
>
> Goson zhang  于2020年1月8日周三 下午5:14写道：
>
> > Hi community
> >
> > I've noticed that the License in code files need modify, for example :
> >
> > /*
> >
> >
> > * * Tencent is pleased to support the open source community by making
> > TubeMQ available. * * Copyright (C) 2012-2019 Tencent. All Rights
> > Reserved.*
> >  *
> >  * Licensed under the Apache License, Version 2.0 (the "License"); you
> may
> > not use
> >  * this file except in compliance with the License. You may obtain a copy
> > of the
> >  * License at
> >  *
> >
> > Should we use the standard Apache License for declaration?
> >
>

[jira] [Resolved] (LIVY-735) Fix RPC Channel Closed When Multi Clients Connect to One Driver

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-735.
--
Fix Version/s: 0.8.0
 Assignee: runzhiwang
   Resolution: Fixed

Issue resolved by pull request 268
https://github.com/apache/incubator-livy/pull/268

> Fix RPC Channel Closed When Multi Clients Connect to One Driver 
> 
>
> Key: LIVY-735
> URL: https://issues.apache.org/jira/browse/LIVY-735
> Project: Livy
>  Issue Type: Sub-task
>Reporter: Yiheng Wang
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.8.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Currently, the driver tries to support communicating with multi-clients, by 
> registering each client at 
> https://github.com/apache/incubator-livy/blob/master/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java#L220.
> But actually, if multi-clients connect to one driver, the rpc channel will 
> close, the reason are as follows.
> 1. In every communication, client sends two packages to driver: header\{type, 
> id}, and payload at 
> https://github.com/apache/incubator-livy/blob/master/rsc/src/main/java/org/apache/livy/rsc/rpc/RpcDispatcher.java#L144.
> 2. If client1 sends header1, payload1, and client2 sends header2, payload2 at 
> the same time. 
>  The driver receives the package in the order: header1, header2, payload1, 
> payload2.
> 3. When driver receives header1, driver assigns lastHeader at 
> https://github.com/apache/incubator-livy/blob/master/rsc/src/main/java/org/apache/livy/rsc/rpc/RpcDispatcher.java#L73.
> 4. Then driver receives header2, driver process it as a payload at 
> https://github.com/apache/incubator-livy/blob/master/rsc/src/main/java/org/apache/livy/rsc/rpc/RpcDispatcher.java#L78
>  which cause exception and rpc channel closed.
> In the muti-active HA mode, the design doc is at: 
> https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing,
>  the session is allocated among servers by consistent hashing. If a new livy 
> joins, some session will be migrated from old livy to new livy. If the 
> session client in new livy connect to driver before stoping session client in 
> old livy, then two session clients will both connect to driver, and rpc 
> channel close. In this case, it's hard to ensure only one client connect to 
> one driver at any time. So it's better to support multi-clients connect to 
> one driver, which has no side effects.
> How to fix:
> 1. Move the code of processing client message from `RpcDispatcher` to each 
> `Rpc`.
> 2. Each `Rpc` registers itself to `channelRpc` in RpcDispatcher.
> 3. `RpcDispatcher` dispatches each message to `Rpc` according to 
> `ctx.channel()`.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: PR title format

[Saisai Shao]

I agreed, I think there should a wiki about "how to contribute".

Also about the commit log, I think we should use the script to formalize
the commit log and use this to merge, not directly using GH button, one
consideration is to formalize the commit log, another is to avoid
additional merge commit. Here is what Spark community does (
https://github.com/apache/spark/blob/master/dev/merge_spark_pr.py) .

Thanks
Saisai

Yiheng Wang  于2020年1月8日周三 上午11:38写道：

> Hi community
>
> I've noticed that PR titles are little causal. For example:
>
>- optimization for delete unnecessary code
>- TUBEMQ-7: Using StringBuilder instead of StringBuffer in BaseResult
>
> Should we consider follow common Apache project pattern, like:
> [TUBEMQ-XXX] PR description
> or
> [TUBEMQ-XXX][Component] PR description
>
> Thanks
> Yiheng
>

Re: [VOTE] Release Livy 0.7.0 based on RC4

[Saisai Shao]

My +1 of course.

Saisai Shao  于2020年1月7日周二 下午9:25写道：

> This vote is for releasing Livy 0.7.0 based on RC4.
>
> This RC mainly fixes the license issue.
>
> The vote will be open until Sunday  Jan 12, 23:59 UTC and
> will pass with a minimum of 3 +1 binding votes and a majority of positive
> votes.
>
> [+1] This release is ready to send to the Incubator PMC for approval
>
> [-1] This release is not ready because...
>
> This vote is held according to Apache Incubator release policy:
> https://incubator.apache.org/policy/incubation.html#releases
>
> The RC is based on tag v0.7.0-incubating-rc4:
>
> https://github.com/apache/incubator-livy/commit/664503355d8bae763989ebac087122e306239d54
>
> The release files can be found here:
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc4/
>
> The staged maven artifacts can be found here:
> https://repository.apache.org/content/repositories/orgapachelivy-1013
>
> The list of resolved JIRAs in this release can be found here:
> https://issues.apache.org/jira/projects/LIVY/versions/12345179
>
> Thanks
> Saisai
>

[VOTE] Release Livy 0.7.0 based on RC4

[Saisai Shao]

This vote is for releasing Livy 0.7.0 based on RC4.

This RC mainly fixes the license issue.

The vote will be open until Sunday  Jan 12, 23:59 UTC and
will pass with a minimum of 3 +1 binding votes and a majority of positive
votes.

[+1] This release is ready to send to the Incubator PMC for approval

[-1] This release is not ready because...

This vote is held according to Apache Incubator release policy:
https://incubator.apache.org/policy/incubation.html#releases

The RC is based on tag v0.7.0-incubating-rc4:
https://github.com/apache/incubator-livy/commit/664503355d8bae763989ebac087122e306239d54

The release files can be found here:
https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc4/

The staged maven artifacts can be found here:
https://repository.apache.org/content/repositories/orgapachelivy-1013

The list of resolved JIRAs in this release can be found here:
https://issues.apache.org/jira/projects/LIVY/versions/12345179

Thanks
Saisai

[jira] [Resolved] (LIVY-732) A Common Zookeeper Wrapper Utility

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-732.
--
Fix Version/s: 0.8.0
 Assignee: runzhiwang
   Resolution: Fixed

Issue resolved by pull request 267
https://github.com/apache/incubator-livy/pull/267

> A Common Zookeeper Wrapper Utility 
> ---
>
> Key: LIVY-732
> URL: https://issues.apache.org/jira/browse/LIVY-732
> Project: Livy
>  Issue Type: Sub-task
>Reporter: Yiheng Wang
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.8.0
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Currently, the utilities of zookeeper mixed with ZooKeeperStateStore. To use 
> the utility of zookeeper, the instance of ZooKeeperStateStore has to be 
> created , which looks weird.
> This Jira aims to achieve two targets:
> 1.  Extract the utilities of zookeeper from ZooKeeperStateStore to support 
> such as distributed lock, service discovery and so on.
> 2.  ZooKeeperManager which contains the utilities of zookeeper should be a 
> single instance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (LIVY-738) Fix Livy third-party library license generating issue

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-738.
--
Fix Version/s: 0.8.0
   0.7.0
 Assignee: Saisai Shao
   Resolution: Fixed

Issue resolved by pull request 272
https://github.com/apache/incubator-livy/pull/272

> Fix Livy third-party library license generating issue
> -
>
> Key: LIVY-738
> URL: https://issues.apache.org/jira/browse/LIVY-738
> Project: Livy
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 0.8.0
>    Reporter: Saisai Shao
>Assignee: Saisai Shao
>Priority: Minor
> Fix For: 0.7.0, 0.8.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Some dependencies' licenses are missing, so it cannot correctly generate the 
> license information, here propose to fix them manually.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Fwd: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3

[Saisai Shao]

Thanks Alex for your check. I'm not sure if we have a proper place to add
this reference in the document. I'm fine with the current status, and will
add the comment during release vote.

Best regards,
Saisai

Alex Bozarth  于2020年1月7日周二 上午4:04写道：

> On the note of the "missing" licenses:
>
> Since I added those I did a bit of research to refresh my memory. Based on
> https://www.glyphicons.com/license/ we do not need to directly mention
> the licensing for those fonts. This is again backed up by the comment at
> the top of the bootstrap page
> https://getbootstrap.com/docs/3.3/components/ which is where our usage of
> the fonts/icons comes from. Based on these links we are fine as is but
> could also add a mention somewhere in our docs/licenses to the glyphicons
> website. Another reference would be this PR for the retired Apache project
> Falcon which also used these fonts
> https://issues.apache.org/jira/browse/FALCON-2110
>
> tl:dr we're fine as we are, but it might be kind to add a reference
> somewhere in our docs
>
>
> *Alex Bozarth*
> Software Engineer
> Center for Open-Source Data & AI Technologies
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Saisai Shao ---01/05/2020 10:22:40
> PM---Does anyone knows how to handle license issue for fonts like]Saisai
> Shao ---01/05/2020 10:22:40 PM---Does anyone knows how to handle license
> issue for fonts like below? Thanks
>
> From: Saisai Shao 
> To: dev@livy.incubator.apache.org
> Date: 01/05/2020 10:22 PM
> Subject: [EXTERNAL] Fwd: [VOTE] Release Apache Livy 0.7.0 (incubating)
> based on RC3
>
> --
>
>
>
> Does anyone knows how to handle license issue for fonts like below?
>
> Thanks
> Saisai
>
> -- Forwarded message -
> 发件人： Justin Mclean 
> Date: 2020年1月3日周五 上午9:59
> Subject: Re: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3
> To: 
>
>
> Hi,
>
> +1 (binding). There’s a license issue that needs to be fixed before the
> next release.
>
> I checked:
> - incubating in name
> - DISCLAIMER exists
> - LICENSE is missing a few things
> - NOTICE year needs updating (Is it still 2018?)
> - No unexpended binary files
> - all source files have ASF headers
> - can compile from source
>
> License is missing license for [1][2]
>
> Thanks,
> Justin
>
> 1.
>
> ./docs/assets/themes/apache/bootstrap/fonts/glyphicons-halflings-regular.svg
> 2.
>
> ./server/src/main/resources/org/apache/livy/server/ui/static/fonts/glyphicons-halflings-regular.*
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>
>
>

Re: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3

[Saisai Shao]

Hi Luciano,

I've fixed the license version manually, they'll be solved in next RC cut.
But still I cannot find out the license for ldapsdk 4.1, this is a very old
library, I doubt there's no license existed. Would you please help to check
it? Thanks!

And regarding to bootstrap and them files, they do existed in binary distro
as a part of Livy Server.

Thanks
Saisai

Luciano Resende  于2020年1月3日周五 上午6:11写道：

> On Mon, Dec 30, 2019 at 3:43 AM Saisai Shao 
> wrote:
> >
> > The Livy PPMC has voted to release Livy 0.7.0 RC3 as the next Livy
> release.
> >
> > Livy enables programmatic, fault-tolerant, multi-tenant submission of
> > Spark jobs from web/mobile apps (no Spark client needed). So, multiple
> > users can interact with your Spark cluster concurrently and reliably.
> >
> > Vote thread:
> >
> https://lists.apache.org/thread.html/ce78a69fb17b7ae1a04a51cb13949c4f726d03c43c6ebde8bbd0272f%40%3Cdev.livy.apache.org%3E
> >
> > Result thread:
> >
> https://lists.apache.org/thread.html/d030023995e4323c32e8ee68de99e48cb4f20761d5ff784ff2aac558%40%3Cdev.livy.apache.org%3E
> >
> > The RC is based on tag v0.7.0-incubating-rc3:
> >
> https://github.com/apache/incubator-livy/commit/0cc431a7e9236e313f9904ef121cad5df444d4da
> >
> > The release files can be found here:
> >
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc3/
> >
> > The staged maven artifacts can be found here:
> > https://repository.apache.org/content/repositories/orgapachelivy-1012
> >
> > The list of resolved JIRAs in this release can be found here:
> > https://issues.apache.org/jira/projects/LIVY/versions/12345179
> >
> > Vote will be open for at least 72 hours. Thanks!
>
> While looking into the binary distribution, it looks like the
> third-party file might have been generated and has more files that
> actually the ones being distributed, but my bigger issue is with a
> list of dependencies with unknown licenses.
>
> Unknown license:
>
> * ASM Core (asm:asm:3.1 - http://asm.objectweb.org/asm/)
> * Antlr 3.4 Runtime (org.antlr:antlr-runtime:3.4 -
> http://www.antlr.org)
> * Java Transaction API (javax.transaction:jta:1.1 -
> http://java.sun.com/products/jta)
> * Jettison (org.codehaus.jettison:jettison:1.1 - no url defined)
> * commons-beanutils (commons-beanutils:commons-beanutils:1.7.0 -
> no url defined)
> * jsp-api (javax.servlet:jsp-api:2.0 - no url defined)
> * ldapsdk (ldapsdk:ldapsdk:4.1 - no url defined)
> * oro (oro:oro:2.0.8 - no url defined)
> * servlet-api (javax.servlet:servlet-api:2.5 - no url defined)
> * transaction-api (javax.transaction:transaction-api:1.1 - no url
> defined)
> * zookeeper (org.apache.zookeeper:zookeeper:3.4.6 - no url defined)
>
> Also, the main license lists dependencies (e.g. bootstrap and related
> theme files) that I believe is only in source distro.
>
> I would feel more comfortable approving the release after this is
> sorted out, or making this a source release only.
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

[jira] [Created] (LIVY-738) Fix Livy third-party library license generating issue

[Saisai Shao (Jira)]

Saisai Shao created LIVY-738:


 Summary: Fix Livy third-party library license generating issue
 Key: LIVY-738
 URL: https://issues.apache.org/jira/browse/LIVY-738
 Project: Livy
  Issue Type: Bug
  Components: Build
Affects Versions: 0.8.0
Reporter: Saisai Shao


Some dependencies' licenses are missing, so it cannot correctly generate the 
license information, here propose to fix them manually.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Fwd: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3

[Saisai Shao]

Does anyone knows how to handle license issue for fonts like below?

Thanks
Saisai

-- Forwarded message -
发件人： Justin Mclean 
Date: 2020年1月3日周五 上午9:59
Subject: Re: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3
To: 


Hi,

+1 (binding). There’s a license issue that needs to be fixed before the
next release.

I checked:
- incubating in name
- DISCLAIMER exists
- LICENSE is missing a few things
- NOTICE year needs updating (Is it still 2018?)
- No unexpended binary files
- all source files have ASF headers
- can compile from source

License is missing license for [1][2]

Thanks,
Justin

1.
./docs/assets/themes/apache/bootstrap/fonts/glyphicons-halflings-regular.svg
2.
./server/src/main/resources/org/apache/livy/server/ui/static/fonts/glyphicons-halflings-regular.*


-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: Podling Tubemq Report Reminder - January 2020

[Saisai Shao]

I've updated some parts of the report, please help to review, thanks!

Besides, we will follow the Apache way to bring out all the discussions on
the mail list, thanks for your guidance.

Thanks
Saisai


Saisai Shao  于2020年1月3日周五 上午11:27写道：

> Thanks Justin for your reply, I will amend the report soon.
>
> Best regards,
> Saisai
>
> Justin Mclean  于2020年1月3日周五 上午7:03写道：
>
>> Hi,
>>
>> Thanks for submitting the  report but it will not be accepted in it's
>> currently form as it doesn't accurately reflect the current podlings state
>> and what it needs to do. Next time I suggest you work on the report in the
>> open before the due date.
>>
>> You have user the three most important unfinished issues to address
>> before graduating:
>> 1. Grow the community to involve more contributors and increase the
>> diversity.
>> 2. Polish the code and document to satisfy the Apache way.
>> 3. Publish the Apache release.
>>
>> There are far more serious matters that need to be sorted out before
>> those are addressed. Number one being moving conversation to this mailing
>> list.
>>
>> Under "Are there any issues that the IPMC or ASF Board need to be aware
>> of?" you have said "No" this is incorrect.
>>
>> Under "How has the community developed since the last report?" you
>> mention PRs and commits but the repos have not moved to the ASF yet so how
>> is this possible?
>>
>> User "How has the project developed since the last report?" you mention
>> working on the first release and website. I see no conversation of this on
>> list or any work towards those gaols. http://tubemq.apache.org currently
>> gives 404 not found.
>>
>> Please rewrite the report to accurately reflect where the podling
>> currently is and what issues it faces.
>>
>> Thanks,
>> Justin
>>
>>
>>
>>
>>
>>

Re: Podling Tubemq Report Reminder - January 2020

[Saisai Shao]

Thanks Justin for your reply, I will amend the report soon.

Best regards,
Saisai

Justin Mclean  于2020年1月3日周五 上午7:03写道：

> Hi,
>
> Thanks for submitting the  report but it will not be accepted in it's
> currently form as it doesn't accurately reflect the current podlings state
> and what it needs to do. Next time I suggest you work on the report in the
> open before the due date.
>
> You have user the three most important unfinished issues to address before
> graduating:
> 1. Grow the community to involve more contributors and increase the
> diversity.
> 2. Polish the code and document to satisfy the Apache way.
> 3. Publish the Apache release.
>
> There are far more serious matters that need to be sorted out before those
> are addressed. Number one being moving conversation to this mailing list.
>
> Under "Are there any issues that the IPMC or ASF Board need to be aware
> of?" you have said "No" this is incorrect.
>
> Under "How has the community developed since the last report?" you mention
> PRs and commits but the repos have not moved to the ASF yet so how is this
> possible?
>
> User "How has the project developed since the last report?" you mention
> working on the first release and website. I see no conversation of this on
> list or any work towards those gaols. http://tubemq.apache.org currently
> gives 404 not found.
>
> Please rewrite the report to accurately reflect where the podling
> currently is and what issues it faces.
>
> Thanks,
> Justin
>
>
>
>
>
>

Re: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3

[Saisai Shao]

Thanks Justin and Luciano for your check.

I use some maven plugins to extract license files from third party
libraries, the missing of license files are mainly due to the issue of this
jars. I will amend them manually and make another RC release.

Thanks
Saisai



Justin Mclean  于2020年1月3日周五 上午9:59写道：

> Hi,
>
> +1 (binding). There’s a license issue that needs to be fixed before the
> next release.
>
> I checked:
> - incubating in name
> - DISCLAIMER exists
> - LICENSE is missing a few things
> - NOTICE year needs updating (Is it still 2018?)
> - No unexpended binary files
> - all source files have ASF headers
> - can compile from source
>
> License is missing license for [1][2]
>
> Thanks,
> Justin
>
> 1.
> ./docs/assets/themes/apache/bootstrap/fonts/glyphicons-halflings-regular.svg
> 2.
> ./server/src/main/resources/org/apache/livy/server/ui/static/fonts/glyphicons-halflings-regular.*
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3

[Saisai Shao]

There's no any vote yet, I guess it might be due to the holiday season. I
will extend the due date to weekend.

Thanks
Saisai

Saisai Shao  于2019年12月30日周一 上午10:43写道：

> The Livy PPMC has voted to release Livy 0.7.0 RC3 as the next Livy
>  release.
>
> Livy enables programmatic, fault-tolerant, multi-tenant submission of
> Spark jobs from web/mobile apps (no Spark client needed). So, multiple
> users can interact with your Spark cluster concurrently and reliably.
>
> Vote thread:
>
> https://lists.apache.org/thread.html/ce78a69fb17b7ae1a04a51cb13949c4f726d03c43c6ebde8bbd0272f%40%3Cdev.livy.apache.org%3E
>
> Result thread:
>
> https://lists.apache.org/thread.html/d030023995e4323c32e8ee68de99e48cb4f20761d5ff784ff2aac558%40%3Cdev.livy.apache.org%3E
>
> The RC is based on tag v0.7.0-incubating-rc3:
>
> https://github.com/apache/incubator-livy/commit/0cc431a7e9236e313f9904ef121cad5df444d4da
>
> The release files can be found here:
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc3/
>
> The staged maven artifacts can be found here:
> https://repository.apache.org/content/repositories/orgapachelivy-1012
>
> The list of resolved JIRAs in this release can be found here:
> https://issues.apache.org/jira/projects/LIVY/versions/12345179
>
> Vote will be open for at least 72 hours. Thanks!
>

Re: Podling Tubemq Report Reminder - January 2020

[Saisai Shao]

Hi Justin,

Sorry about the late reply, we will work on this today. Thanks a lot for
your reminder.

Best regards,
Saisai

Justin Mclean  于2020年1月2日周四 上午10:05写道：

> Hi,
>
> This is the second report in a row the project has missed, one missed
> report is OK, two are cause for concern, three and you may be kicked out of
> the Incubator. I can see that you are still moving resources to the
> incubator and start up here has been slow. You need to provide the IPMC
> with oversight to how the project is going and what issues it is facing and
> still need to submit reports in the start up phase.
>
> It seems to me that some of your mentors are missing, if this is the case
> please mention this in your report. I've reached out to them (on the
> private list) and if they don't reply they will be removed and other
> mentors found to replace them.
>
> I suggest you discuss on this list what is need to move the project over,
> I'm not seeing a lot of communication on these lists, and it seems that
> some communication is currently occurring elsewhere. Please fix this.
>
> Thanks,
> Justin
>

Re: Podling Livy Report Reminder - January 2020

[Saisai Shao]

I will work on this.

Thanks
Saisai

Justin Mclean  于2020年1月1日周三 下午4:06写道：

> Hi,
> Just a friendly reminder the report is due today is anyone working on it?
> Thanks,
> Justin
>

[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005895#comment-17005895
 ] 

Saisai Shao commented on LIVY-718:
--

IIUC, current JDBC part maintains lots of results/metadata/information on 
LivyServer, I think the key point mentioned by [~bikassaha] is to make Livy 
Server stateless, that would be an ideal solution, but currently it requires a 
large amount of works.

> Support multi-active high availability in Livy
> --
>
> Key: LIVY-718
> URL: https://issues.apache.org/jira/browse/LIVY-718
> Project: Livy
>  Issue Type: Epic
>  Components: RSC, Server
>Reporter: Yiheng Wang
>Priority: Major
>
> In this JIRA we want to discuss how to implement multi-active high 
> availability in Livy.
> Currently, Livy only supports single node recovery. This is not sufficient in 
> some production environments. In our scenario, the Livy server serves many 
> notebook and JDBC services. We want to make Livy service more fault-tolerant 
> and scalable.
> There're already some proposals in the community for high availability. But 
> they're not so complete or just for active-standby high availability. So we 
> propose a multi-active high availability design to achieve the following 
> goals:
> # One or more servers will serve the client requests at the same time.
> # Sessions are allocated among different servers.
> # When one node crashes, the affected sessions will be moved to other active 
> services.
> Here's our design document, please review and comment:
> https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3

[Saisai Shao]

The Livy PPMC has voted to release Livy 0.7.0 RC3 as the next Livy release.

Livy enables programmatic, fault-tolerant, multi-tenant submission of
Spark jobs from web/mobile apps (no Spark client needed). So, multiple
users can interact with your Spark cluster concurrently and reliably.

Vote thread:
https://lists.apache.org/thread.html/ce78a69fb17b7ae1a04a51cb13949c4f726d03c43c6ebde8bbd0272f%40%3Cdev.livy.apache.org%3E

Result thread:
https://lists.apache.org/thread.html/d030023995e4323c32e8ee68de99e48cb4f20761d5ff784ff2aac558%40%3Cdev.livy.apache.org%3E

The RC is based on tag v0.7.0-incubating-rc3:
https://github.com/apache/incubator-livy/commit/0cc431a7e9236e313f9904ef121cad5df444d4da

The release files can be found here:
https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc3/

The staged maven artifacts can be found here:
https://repository.apache.org/content/repositories/orgapachelivy-1012

The list of resolved JIRAs in this release can be found here:
https://issues.apache.org/jira/projects/LIVY/versions/12345179

Vote will be open for at least 72 hours. Thanks!