delete the test module hudi-integ-test
dear all: when i build the project,the following error occurred,could delete the unimport moudle? [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (Setup HUDI_WS) on project hudi-integ-test: Command execution failed.: Cannot run program "\bin\bash" (in directory "D:\code-repository\github\hudi\hudi-integ-test"): CreateProcess error=2, ϵͳ▒Ҳ▒▒▒ָ▒ļ▒▒▒ -> [Help 1]
Re: Apply for JIRA permission
Apache Spark. ? Hefei Li 于2020年5月27日周三 下午11:46写道： > Hi guys, > > I want to contribute to Apache Spark. > > Would you please give me the permission as a contributor ? > > My JIRA username is *lhfei *. > > > > > === > Best Regards > Hefei LiHefei Li > MP: +86 18701581473 > MSN: lh...@live.cn > === >
Re: [DISCUSS] Logos on project front page.
I agree on following the best practices. Bhavani Sudha 于2020年5月13日周三 下午4:30写道： > +1 > > From the doc it seems like we can probably move them to secondary page. > > Thanks, > Sudha > > On Tue, May 12, 2020 at 11:56 PM vino yang wrote: > > > +1 to follow the best practices. > > > > vbal...@apache.org 于2020年5月13日周三 上午10:31写道： > > > > > > > > I agree on following the best practices. > > > Balaji.VOn Tuesday, May 12, 2020, 06:52:59 PM PDT, Vinoth Chandar < > > > vin...@apache.org> wrote: > > > > > > Hello all, > > > > > > This was raised during the graduation discussion. We have been referred > > to > > > [1]. The doc ends saying. "These best practices for linking to outside > > > pages on project websites are meant as suggestions for projects. PMCs > are > > > free to adopt (or not) any of these suggestions for their sites.". > > > > > > But I would prefer to play by the best practices if we can.. > > > > > > Can you all chime in with your thoughts? > > > > > > > > > > > > [1] https://www.apache.org/foundation/marks/linking > > > > > >
Re: [DISCUSS] should we do a 0.5.3 patch set release ?
+1 Bhavani Sudha 于2020年5月13日周三 下午1:19写道： > Thank you all. I created a jira here - > https://jira.apache.org/jira/browse/HUDI-890 that tracks the list of > patches going into this release. So far I am able to cherry pick these > commits and get a successful local run of tests. > > Let us aim to code freeze by end of this week (May 16th EOD) for more > patches that can go into 0.5.3 release. > > Please respond if you think there are more candidate PRs (criteria: perf > improvements/bug fixes) that can be included in 0.5.3. > > Thanks, > Sudha > > On Thu, May 7, 2020 at 4:16 PM Minjeong Noh > wrote: > > > > > > https://github.com/apache/incubator-hudi/commit/dbc9acd23a4eb208c7cd458bb3adaf54731d4145 > > < > > > https://github.com/apache/incubator-hudi/commit/dbc9acd23a4eb208c7cd458bb3adaf54731d4145 > > > > > > > > On 2020/05/06 20:31:00, Vinoth Chandar wrote: > > > Hi Sudha,> > > > > > > +1 on the overall idea.. I tried to pick out few of these PRs that are> > > > > > > - Small enough to apply easily> > > > - Have limited scope, fixing pointed problems> > > > - Have high impact on performance or usability> > > > > > > [HUDI-799] Use appropriate FS when loading configs> > > > > > > https://github.com/apache/incubator-hudi/commit/acb1ada2f756b49d9f9a0aa152f99fcc9e86dde7 > > > > > > > > > > [HUDI-713] Fix conversion of Spark array of struct type to Avro schema> > > > > > > https://github.com/apache/incubator-hudi/commit/ce0a4c64d07d6eea926d1bfb92b69ae387b88f50 > > > > > > > > > > [HUDI-656][Performance] Return a dummy Spark relation after writing> > > > > > > https://github.com/apache/incubator-hudi/commit/c40a0d4e91896dece51969f5308016ecb3aa635c > > > > > > > > > > [HUDI-850] Avoid unnecessary listings in incremental cleaning mode> > > > > > > https://github.com/apache/incubator-hudi/commit/506447fd4fde4cd922f7aa8f4e17a7f0dc97 > > > > > > > > > > [HUDI-724] Parallelize getSmallFiles for partitions> > > > > > > https://github.com/apache/incubator-hudi/commit/1f5b0c77d6c87a936f2d34287ec6a1df1cb18b33 > > > > > > > > > > [HUDI-607] Fix to allow creation/syncing of Hive tables partitioned> > > > > > > https://github.com/apache/incubator-hudi/commit/2d040145810b8b14c59c5882f9115698351039d1 > > > > > > > > > > Add constructor to HoodieROTablePathFilter> > > > > > > https://github.com/apache/incubator-hudi/commit/418f9bb2e91ed6c02077d36e49a47f0c8d08303a > > > > > > > > > > [HUDI-539] Make ROPathFilter conf member serializable> > > > > > > https://github.com/apache/incubator-hudi/commit/e3019031d8fff60df4fec82eac3fd5c044011635 > > > > > > > > > > Add changes for presto mor queries> > > > > > > https://github.com/apache/incubator-hudi/commit/e21441ad8317f302fed947c414e059a332e4d1ef > > > > > > > > > > [HUDI-782] Add support of Aliyun object storage service.> > > > > > > https://github.com/apache/incubator-hudi/commit/5d717a28f45137bea71dffa31b0ae7ccbf1bda00 > > > > > > > > > > > > > Please chime in with your thoughts, as well.> > > > > > > I think there are some bug fixes in the pending PRs as well. esp from > > Alex> > > > and Pratyaksh .> > > > > > > Thanks> > > > Vinoth> > > > > > > > > > On Tue, May 5, 2020 at 9:33 PM Bhavani Sudha > > > > wrote:> > > > > > > > Hello all,> > > > >> > > > > I am wondering if we should do a 0.5.3 release by backporting all > > minor to> > > > > medium bug fixes (that are in master already) to 0.5.2 and do a > minor> > > > > release ? That way we can use some time to reserve 0.6.0 release for > > all> > > > > major features that are upcoming and/or almost there. Please share > > your> > > > > thoughts. If you agree also please share the list of fixes that you > > know of> > > > > that can go into 0.5.3.> > > > >> > > > > Thanks,> > > > > Sudha> > > > >> > > > >
Re: [VOTE] Apache Hudi graduation to top level project
+1 发自我的iPhone -- Original -- From: leesf
Re: [DISCUSS] Support popular metrics reporter
+1 Balaji Varadarajan 于2020年4月23日周四 上午12:12写道： > +1 > On Wednesday, April 22, 2020, 08:35:30 AM PDT, leesf < > leesf0...@gmail.com> wrote: > > +1 > > Vinoth Chandar 于2020年4月22日周三 下午2:24写道： > > > +1 from me as well > > > > On Mon, Apr 20, 2020 at 9:37 PM vino yang wrote: > > > > > Hi Raymond, > > > > > > Thanks for opening this discussion. > > > > > > IMHO, as Hudi's user base grows, we need to enhance our metrics > reporter. > > > From an ecological point of view, this is also very important. > > > > > > So, +1 from my side. > > > > > > Best, > > > Vino > > > > > > Shiyan Xu 于2020年4月21日周二 上午10:59写道： > > > > > > > Hi all, > > > > > > > > I'd like raise the topic of supporting multiple metrics reporters. > > > > > > > > Currently hudi supports graphite and JMX. And there are 2 proposed > > > reporter > > > > types: CSV and Prometheus > > > > https://jira.apache.org/jira/browse/HUDI-210 > > > > https://jira.apache.org/jira/browse/HUDI-361 > > > > > > > > I think supporting multiple metrics backends gives Hudi competitive > > > > advantage on user expansion. It reduces the friction for different > > > > organizations to adopt Hudi. And we only need to support a few > popular > > > ones > > > > to achieve that. > > > > > > > > In terms of determining the list, as mentioned by @vinoyang, flink > has > > a > > > > nice list of supported ones: > > > > > > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#reporter > > > > which can be used as a reference. > > > > > > > > From that list, I'd like to propose supporting Datadog as well, due > to > > > its > > > > popularity. May I get +1 on this? > > > > > > > > Thank you. > > > > > > > > Regards, > > > > Raymond > > > > > > > > >
Re: Table Read fails in Spark Submit , Where as succeeds in spark-shell
hi,periyasamy
Thanks for asking questions or reporting issues, please describe it in
detail by using github issue.
https://github.com/apache/incubator-hudi/issues
cooper
selvaraj periyasamy 于2020年4月22日周三
上午7:35写道：
> Folks,
>
> I am using Apache Hudi 0.5.0. Our hadoop cluster is miix of spark
> version 2.3.0, Scala version 2.11.8 & Hive version 1.2.2.
> There are multiple use cases already working in Hudi.
>
> I need to read one of sequence table, which is continuously inserted on
> new partition by other process using Hive, not by Hudi. And then write
> this DataFrame into another COW table using Hudi.
>
> When I use spark.sql in spark-shell, which was started with Hudi jar, I am
> able to do select as mentioned below.
>
> spark-shell --jars
> /Users/seperiya/Downloads/hudi-spark-bundle-0.5.0-incubating.jar --conf
> 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
>
>
> scala> spark.sql("select * from poc.request_result__ct").show
>
> 2020-04-21 15:56:07 WARN ObjectStore:568 - Failed to get database
> global_temp, returning NoSuchObjectException
>
>
> +--+--++-+---+---+---+---+--+
>
> |request_id|prev_request_id|ref_no|type_code|
> transaction_date|process_ts|
> commit_ts|header__change_oper|header__partition_name|
>
>
> +--+--++-+---+---+---+---+--+
>
> |2020041011| null|null| PA|2020-04-10
> 11:11:23|2020-04-10 11:11:30|2020-04-10 11:11:35| I|
> 20200117T235000_2...|
>
>
> +--+--++-+---+---+---+---+--+
>
>
>
>
>
> Whereas when I convert the same code into scala file and execute it using
> spark-submit , I am getting error. Attached the error logs.
>
>
> spark-submit --jars
> /Users/seperiya/Downloads/hudi-spark-bundle-0.5.0-incubating.jar --conf
> 'spark.serializer=org.apache.spark.serializer.KryoSerializer' --class Test
> /Test-1.0.0-SNAPSHOT-jar-with-dependencies.jar
>
>
> object Test {
>
> def main(args: Array[String]): Unit = {
>
> implicit val sparkSession: SparkSession = SparkUtil.buildSession("Test_"+
>
>
> now.get(Calendar.HOUR_OF_DAY)+now.get(Calendar.MINUTE)+now.get(Calendar.SECOND))
>
> sparkSession.sql( s"select * from poc.request_result__ct").show()
>
> }
> }
>
>
> When I remove Hudi bundle jar and runs the same , it works.
>
>
> spark-submit --class Test /Test-1.0.0-SNAPSHOT-jar-with-dependencies.jar
>
>
> Even though Hudi code will come into picture only when I insert the data
> on other table, for some reason, read fails . Could anyone shed some
> light not his issue?
>
>
>
Re: run example
ok，thank you Bhavani Sudha 于2020年4月15日周三 上午12:49写道： > Hi @cooper, > > Can you please copy paste the issue or create a github issue? Mailing list > does not work well for images attached. > > Thanks, > Sudha > > > > On Tue, Apr 14, 2020 at 4:45 AM cooper wrote: > > > hi,all： > > I try to run the demo HoodieClientExample,I find the > > "hoodie.keep.min.commits" > "hoodie.cleaner.commits.retained",so the > > program runs fail,I don't know whether the logic is correct > > [image: 微信图片_20200414192957.png] > > [image: 微信图片_20200414193042.png] > > >
run example
hi,all： I try to run the demo HoodieClientExample,I find the "hoodie.keep.min.commits" > "hoodie.cleaner.commits.retained",so the program runs fail,I don't know whether the logic is correct [image: 微信图片_20200414192957.png] [image: 微信图片_20200414193042.png]
contributor permission request
Hi, I want to contribute to Apache Hudi. Would you please give me the contributor permission? My JIRA ID is lichangfu
Re: Re: New Committer: lamber-ken
congratulations lamber-ken 于2020年4月10日周五 上午12:04写道： > > > Dear team, > > > Thank you all, let us together strive to work together! > > > Best, > Lamber-Ken > > > > > > At 2020-04-09 03:49:12, "Shiyan Xu" wrote: > >Congrats Lamber-ken! Well deserved! > > > >On Wed, Apr 8, 2020 at 4:52 AM Sivabalan wrote: > > > >> Congrats Lamber! Well deserved. > >> > >> On Wed, Apr 8, 2020 at 5:21 AM Pratyaksh Sharma > >> wrote: > >> > >> > Congratulations lamberken! > >> > > >> > On Wed, Apr 8, 2020 at 11:10 AM Jiayi Liao > >> > wrote: > >> > > >> > > Congratulations! > >> > > > >> > > Best, > >> > > Jiayi Liao > >> > > > >> > > On Wed, Apr 8, 2020 at 12:15 PM tison wrote: > >> > > > >> > > > Congrats lamber! > >> > > > > >> > > > Best, > >> > > > tison. > >> > > > > >> > > > > >> > > > vino yang 于2020年4月8日周三 上午11:45写道： > >> > > > > >> > > > > Congrats lamber! Well deserved! > >> > > > > > >> > > > > Best, > >> > > > > Vino > >> > > > > > >> > > > > leesf 于2020年4月8日周三 上午9:30写道： > >> > > > > > >> > > > > > Congrats lamber-ken, well deserved! > >> > > > > > > >> > > > > > Balaji Varadarajan 于2020年4月8日周三 > >> > > 上午6:45写道： > >> > > > > > > >> > > > > > > Many Congratulations Lamber-Ken. Well deserved !! > >> > > > > > > Balaji.V > >> > > > > > > On Tuesday, April 7, 2020, 02:23:51 PM PDT, Y Ethan Guo > < > >> > > > > > > ethan.guoyi...@gmail.com> wrote: > >> > > > > > > > >> > > > > > > Congrats!!! > >> > > > > > > > >> > > > > > > On Tue, Apr 7, 2020 at 2:22 PM Gary Li < > >> yanjia.gary...@gmail.com > >> > > > >> > > > > wrote: > >> > > > > > > > >> > > > > > > > Congrats lamber! Well deserved! > >> > > > > > > > > >> > > > > > > > On Tue, Apr 7, 2020 at 2:18 PM Vinoth Chandar < > >> > vin...@apache.org > >> > > > > >> > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > Hello Apache Hudi Community, > >> > > > > > > > > > >> > > > > > > > > The Podling Project Management Committee (PPMC) for > Apache > >> > > > > > > > > Hudi (Incubating) has invited lamber-ken (Xie Lei) to > >> become > >> > a > >> > > > > > > committer > >> > > > > > > > > and we are pleased to announce that he has accepted. > >> > > > > > > > > > >> > > > > > > > > lamber-ken has had a large impact by in hudi, with some > >> > > sustained > >> > > > > > > efforts > >> > > > > > > > > in the past several months. He has rebuilt our site > ground > >> > up, > >> > > > > > > automated > >> > > > > > > > > doc workflows, helped fixed a lot of bugs and also been > >> super > >> > > > > helpful > >> > > > > > > for > >> > > > > > > > > the community at large. > >> > > > > > > > > > >> > > > > > > > > Congratulations lamber-ken !! Please join me in > recognizing > >> > his > >> > > > > > > efforts! > >> > > > > > > > > > >> > > > > > > > > On behalf of PPMC, > >> > > > > > > > > Vinoth > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >> -- > >> Regards, > >> -Sivabalan > >> >
