Is this JIRA entry related to what you want?
https://issues.apache.org/jira/browse/SPARK-10399
Regards,
Kazuaki Ishizaki
From: Jia <jacqueline...@gmail.com>
To: Dewful <dew...@gmail.com>
Cc: "user @spark" <u...@spark.apache.org>, dev@spark.apac
great
discussions to exploit GPUs in Spark.
Best Regards,
Kazuaki Ishizaki
I created a new JIRA entry
https://issues.apache.org/jira/browse/SPARK-12620 for this instead of
reopening the existing JIRA based on the suggestion.
Best Regards,
Kazuaki Ishizaki
From: Kazuaki Ishizaki/Japan/IBM@IBMJP
To: dev@spark.apache.org
Date: 2016/01/04 12:54
Subject
pu/blob/dev/examples/src/main/scala/org/apache/spark/examples/SparkGPULR.scala#L89
(Sorry for no documentation now).
We are glad to support requested features or to looking forward to getting
pull requests.
Best Regard,
Kazuaki Ishizaki
From: "Allen Zhang" <allenzhang...@126.co
This is because to build Spark requires maven 3.3.3 or later.
http://spark.apache.org/docs/latest/building-spark.html
Regards,
Kazuaki Ishizaki
From: salexln <sale...@gmail.com>
To: dev@spark.apache.org
Date: 2015/12/25 15:52
Subject:latest Spark build error
Hi all
equality of two Rows that have an Array or complex types (e.g. Map)?
Best Regards,
Kazuaki Ishizaki, @kiszk
iding
binary column storage for data partition. We really appreciate it if you
would give us comments, suggestions, or feedback.
Best Regards
Kazuaki Ishizaki
From: "Ulanov, Alexander" <alexander.ula...@hpe.com>
To: Sam Halliday <sam.halli...@gmail.com>, John Canny
&
exploit GPUs such as BIDMat in Spark.
If we would run BIDMat with our columnar strorage, the performance boost
would be good as others reported.
Best Regards,
Kazuaki Ishizaki,
From: "Allen Zhang" <allenzhang...@126.com>
To: Kazuaki Ishizaki/Japan/IBM@IBMJP
Cc: "
with current Spark. I expect that this
talk would help us.
http://conferences.oreilly.com/strata/hadoop-big-data-ca/public/schedule/detail/47565
We appreciate your great feedback.
Best Regards,
Kazuaki Ishizaki, Ph.D., Senior research staff member, IBM Research -
Tokyo
From: "Ulanov, Alex
sues.apache.org/jira/browse/SPARK-14689
https://issues.apache.org/jira/browse/SPARK-14690
Best regards,
Kazuaki Ishizaki
t;id>10") return correct result.
f(df1).count // output 89 which is incorrect
Regards,
Kazuaki Ishizaki
From: gen tang <gen.tan...@gmail.com>
To: dev@spark.apache.org
Date: 2017/02/22 15:02
Subject:Re: A DataFrame cache bug
Hi All,
I might find a related issue
Congrats!
Kazuaki Ishizaki
From: Reynold Xin <r...@databricks.com>
To: "dev@spark.apache.org" <dev@spark.apache.org>
Date: 2017/02/14 04:18
Subject:welcoming Takuya Ueshin as a new Apache Spark committer
Hi all,
Takuya-san has recently been elected an
these pull requests.
Best Regards,
Kazuaki Ishizaki
From: Maciej Bryński <mac...@brynski.pl>
To: Spark dev list <dev@spark.apache.org>
Date: 2016/08/28 05:40
Subject:Cache'ing performance
Hi,
I did some benchmark of cache function today.
RDD
sc.parallelize(0 unt
pull/11956. This PR shows room to
performance improvement for float/double values that are not compressed.
Kazuaki Ishizaki
From: linguin@gmail.com
To: Maciej Bry��ski <mac...@brynski.pl>
Cc: Spark dev list <dev@spark.apache.org>
Date: 2016/08/28 11:30
Subject
The line where I pointed out would work correctly. This is because a type
of this division is double. d2i correctly handles overflow cases.
Kazuaki Ishizaki
From: Nicholas Chammas <nicholas.cham...@gmail.com>
To: Kazuaki Ishizaki/Japan/IBM@IBMJP, Reynold Xin
<r...@databrick
/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java#L156
Regards,
Kazuaki Ishizaki
From: Reynold Xin <r...@databricks.com>
To: Nicholas Chammas <nicholas.cham...@gmail.com>
Cc: Spark dev list <dev@spark.apache.org>
Date: 2016/12/07 14:27
Subject:Re: R
Hi,
You may find several micro-benchmarks under
https://github.com/apache/spark/tree/master/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark
.
Regards,
Kazuaki Ishizaki
From: Prasun Ratn <prasun.r...@gmail.com>
To: Apache Spark Dev <dev@spark.apache.org>
Dat
is that both applications supports Apache Arrow
APIs.
Other approaches could be.
What approach would be good for all of applications?
Regards,
Kazuaki Ishizaki
if null exists in non-null column.
Any comments are appreciated.
Kazuaki Ishizaki
From: Jason White <jason.wh...@shopify.com>
To: dev@spark.apache.org
Date: 2017/03/21 06:31
Subject:Why are DataFrames always read with nullable=True?
If I create a dataframe in Spark with non-nu
0.002 sec <<< ERROR!
java.lang.IllegalArgumentException: requirement failed: No support for
unaligned Unsafe. Set spark.memory.offHeap.enabled to false.
...
Tests run: 207, Failures: 7, Errors: 16, Skipped: 0
Kazuaki Ishizaki
From: Michael Armbrust <mich...@databricks.com
Thank you. Yes, it is not a regression. 2.1.0 would have this failure,
too.
Regards,
Kazuaki Ishizaki
From: Sean Owen <so...@cloudera.com>
To: Kazuaki Ishizaki/Japan/IBM@IBMJP, Michael Armbrust
<mich...@databricks.com>
Cc: "dev@spark.apache.org" <dev@spark
672M
[INFO]
Regards,
Kazuaki Ishizaki,
From: Michael Armbrust <mich...@databricks.com>
To: "dev@spark.apache.org" <dev@spark.apache.org>
Date: 2017/04/19 04:00
Subject:[VOTE] Apache
06 h
[INFO] Finished at: 2017-07-01T15:20:04+09:00
[INFO] Final Memory: 56M/591M
[INFO]
[WARNING] The requested profile "hive" could not be activated because it
does not exist.
Kazuaki Ishizaki
From: Mi
9T02:23:08+09:00
[INFO] Final Memory: 53M/491M
[INFO]
----
Kazuaki Ishizaki,
From: Michael Armbrust <mich...@databricks.com>
To: "dev@spark.apache.org" <dev@spark.apache.org>
Date: 2017/04/28 03:
576M
[INFO]
Regards,
Kazuaki Ishizaki,
From: Michael Armbrust <mich...@databricks.com>
To: "dev@spark.apache.org" <dev@spark.apache.org>
Date: 2017/04/27 09:30
Subject:[VOTE] Apache
Congratulation, Hyukjin and Sameer, well deserved!!
Kazuaki Ishizaki
From: Matei Zaharia <matei.zaha...@gmail.com>
To: dev <dev@spark.apache.org>
Date: 2017/08/08 00:53
Subject:Welcoming Hyukjin Kwon and Sameer Agarwal as committers
Hi everyone,
The Spark PMC re
9T17:51:04+09:00
[INFO] Final Memory: 53M/514M
[INFO]
[WARNING] The requested profile "hive" could not be activated because it
does not exist.
Kazuaki Ishizaki,
From: Michael Armbrust <mich...@datab
It looked well these days. However, it seems to go down slowly again...
When I tried to see console log (e.g.
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77149/consoleFull
), a server returns "proxy error."
Regards,
Kazuaki Ishizaki
From: shane
6T13:44:48+09:00
[INFO] Final Memory: 53M/510M
[INFO]
[WARNING] The requested profile "hive" could not be activated because it
does not exist.
Kazuaki Ishizaki
From: Michael Armbrust <mich...@datab
Congratulation Tejas!
Kazuaki Ishizaki
From: Matei Zaharia <matei.zaha...@gmail.com>
To: "dev@spark.apache.org" <dev@spark.apache.org>
Date: 2017/09/30 04:58
Subject:Welcoming Tejas Patil as a Spark committer
Hi all,
The Spark PMC recently added Teja
min
[INFO] Finished at: 2017-10-03T23:53:33+09:00
[INFO] Final Memory: 112M/322M
[INFO]
[WARNING] The requested profile "hive" could not be activated because it
does not exist.
Kazuaki Ishizaki
From: Dongj
min
[INFO] Finished at: 2017-09-28T12:16:46+09:00
[INFO] Final Memory: 103M/309M
[INFO]
[WARNING] The requested profile "hive" could not be activated because it
does not exist.
Kazuaki Ishizaki
From: Dongj
Congratulations, Jerry!
Kazuaki Ishizaki
From: Hyukjin Kwon <gurwls...@gmail.com>
To: dev <dev@spark.apache.org>
Date: 2017/08/29 12:24
Subject:Re: Welcoming Saisai (Jerry) Shao as a committer
Congratulations! Very well deserved.
2017-08-29 11:41 GMT+09:00 Li
04 h
[INFO] Finished at: 2017-11-30T01:48:15+09:00
[INFO] Final Memory: 128M/329M
[INFO]
[WARNING] The requested profile "hive" could not be activated because it
does not exist.
Kazuaki Ishizaki
From: Dongj
are appreciated.
Regards,
Kazuaki Ishizaki
From: Joseph Bradley <jos...@databricks.com>
To: dev <dev@spark.apache.org>
Cc: Hossein Falaki <hoss...@databricks.com>
Date: 2018/05/03 07:31
Subject:SparkR test failures in PR builder
Hi all,
Does anyo
+1 for cutting a branch earlier.
In some Asian countries, 1st, 2nd, and 3rd January are off.
https://www.timeanddate.com/holidays/
How about 4th or 5th?
Regards,
Kazuaki Ishizaki
From: Felix Cheung <felixcheun...@hotmail.com>
To: Michael Armbrust <mich...@databricks.com>,
produces
incorrect results. I created a JIRA entry
https://issues.apache.org/jira/browse/SPARK-22934.
Best Regards,
Kazuaki Ishizaki
From: Herman van Hövell tot Westerflier <hvanhov...@databricks.com>
To: Jacek Laskowski <ja...@japila.pl>
Cc: dev <dev@spark.apache.or
updated the JIRA ticket.
I am still working for this.
Kazuaki Ishizaki
From: Herman van Hövell tot Westerflier <hvanhov...@databricks.com>
To: Kazuaki Ishizaki <ishiz...@jp.ibm.com>
Cc: Jacek Laskowski <ja...@japila.pl>, dev <dev@spark.apache.org>
Date:
Of course, we would like to eliminate all of the following tags
"flanky" or "flankytest"
Kazuaki Ishizaki
From: Hyukjin Kwon
To: dev
Cc: Xiao Li , Wenchen Fan
Date: 2018/09/04 14:20
Subject:Re: Spark JIRA tags clarification and management
Thank
value generated by Spark
with it generated by other implementations?
Regards,
Kazuaki Ishizaki
From: Sean Owen
To: jiayuanm
Cc: dev@spark.apache.org
Date: 2018/07/07 15:46
Subject:Re: [SPARK ML] Minhash integer overflow
I think it probably still does its.job; the hash
Thank for you reporting this issue. I think this is a bug regarding
integer overflow. IMHO, it would be good to compute hashes with Long.
Would it be possible to create a JIRA entry? Do you want to submit a pull
request, too?
Regards,
Kazuaki Ishizaki
From: jiayuanm
To: dev
Congratulations to Zhenhua!
Kazuaki Ishizaki
From: sujith chacko <sujithchacko.2...@gmail.com>
To: Denny Lee <denny.g@gmail.com>
Cc: Spark dev list <dev@spark.apache.org>, Wenchen Fan
<cloud0...@gmail.com>, "叶先进" <advance...@gmail.com>
Dat
Congratulations to everyone!
Kazuaki Ishizaki
From: Takeshi Yamamuro <linguin@gmail.com>
To: Spark dev list <dev@spark.apache.org>
Date: 2018/03/03 10:45
Subject:Re: Welcoming some new committers
Congrats, all!
On Sat, Mar 3, 2018 at 10:34 AM, Takuya
/1Jzf56bxpMpSwsGV_hSzl9wQG22hyI731McQcjognqxY/edit?usp=sharing
Looking forward to hear your feedback
Best Regards,
Kazuaki Ishizaki
on this SPIP procedure
http://spark.apache.org/improvement-proposals.html, can I ask one or more
PMCs to become a shepherd of this SPIP?
I would appreciate your kindness and cooperation.
Best Regards,
Kazuaki Ishizaki
From: Takeshi Yamamuro
To: Spark dev list
Cc: ishiz...@jp.ibm.com
Date
Hi Xiao,
Thank you very much for becoming a shepherd.
If you feel the discussion settles, we would appreciate it if you would
start a voting.
Regards,
Kazuaki Ishizaki
From: Xiao Li
To: Kazuaki Ishizaki
Cc: dev , Takeshi Yamamuro
Date: 2018/10/22 16:31
Subject:Re: SPIP
" / " + "c").
It is important for us to learn from failures than learn from successes.
We would appreciate it if you could list up failures that you have seen.
Best Regards,
Kazuaki Ishizaki
From: Reynold Xin
To: Kazuaki Ishizaki
Cc: Xiao Li , dev ,
Takeshi Yam
This entry includes a good figure for support lifecycle.
https://www.azul.com/products/zulu-and-zulu-enterprise/zulu-enterprise-java-support-options/
Kazuaki Ishizaki,
From: Marcelo Vanzin
To: Felix Cheung
Cc: Ryan Blue , sn...@snazy.de, dev
, Cesar Delgado
Date: 2018/11/07 08
forward.
Kazuaki Ishizaki,
From: "Kazuaki Ishizaki"
To: Reynold Xin
Cc: dev , Takeshi Yamamuro
, Xiao Li
Date: 2018/10/31 00:56
Subject:Re: SPIP: SPARK-25728 Structured Intermediate
Representation (Tungsten IR) for generating Java code
Hi Reynold,
Thank yo
that it uses Python UDF support with Arrow in Spark.
P.S. I will give a presentation about in-memory data storages for SPark at
SAIS 2019
https://databricks.com/sparkaisummit/north-america/sessions-single-2019?id=40
:)
Kazuaki Ishizaki
From: Wenchen Fan
To: Bobby Evans
Cc: Spark dev
Congratulations, Jose!
Kazuaki Ishizaki
From: Gengliang Wang
To: dev
Date: 2019/01/31 18:32
Subject:Re: Welcome Jose Torres as a Spark committer
Congrats Jose!
在 2019年1月31日,上午6:51,Bryan Cutler 写道:
Congrats Jose!
On Tue, Jan 29, 2019, 10:48 AM Shixiong Zhu
+1 (non-binding)
Kazuaki Ishizaki
From: Bryan Cutler
To: Bobby Evans
Cc: Thomas graves , Spark dev list
Date: 2019/05/09 03:20
Subject:Re: [VOTE][SPARK-27396] SPIP: Public APIs for extended
Columnar Processing Support
+1 (non-binding)
On Tue, May 7, 2019 at 12:04
Thank you Dongjoon for being a release manager.
If the assumed dates are ok, I would like to volunteer for an 2.3.4
release manager.
Best Regards,
Kazuaki Ishizaki,
From: Dongjoon Hyun
To: dev , "user @spark" ,
Apache Spark PMC
Date: 2019/07/13 07:18
Subject:[EX
Thanks, Dongjoon!
+1
Kazuaki Ishizaki,
From: Hyukjin Kwon
To: Takeshi Yamamuro
Cc: Dongjoon Hyun , dev
, User
Date: 2019/08/14 09:21
Subject:[EXTERNAL] Re: Release Apache Spark 2.4.4
+1
2019년 8월 14일 (수) 오전 9:13, Takeshi Yamamuro 님
이 작성:
Hi,
Thanks for your
)%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC
Best Regards,
Kazuaki Ishizaki
Due to the recent correctness issue at SPARK-28699, I will delay the
release for Spark 2.3.4 RC1 for a while.
https://issues.apache.org/jira/browse/SPARK-28699?focusedCommentId=16910859=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16910859
Regards,
Kazuaki Ishizaki
Hi all,
Thank you. I will prepare RC for 2.3.4 this week in parallel. It will be
in parallel with RC for 2.4.4 managed by Dongjoon.
Regards,
Kazuaki Ishizaki
From: Hyukjin Kwon
To: Dilip Biswal
Cc: jzh...@apache.org, dev , Kazuaki Ishizaki
, Takeshi Yamamuro , Xiao Li
Date
Sure, I will launch a separate e-mail thread for discussing 2.3.4 later.
Regards,
Kazuaki Ishizaki, Ph.D.
From: Dongjoon Hyun
To: Sean Owen , Kazuaki Ishizaki
Cc: dev
Date: 2019/08/16 05:10
Subject:[EXTERNAL] Re: Release Apache Spark 2.4.4
+1 for that.
Kazuaki
Please vote on releasing the following candidate as Apache Spark version
2.3.4.
The vote is open until August 29th 2PM PST and passes if a majority +1 PMC
votes are cast, with
a minimum of 3 +1 votes.
[ ] +1 Release this package as Apache Spark 2.3.4
[ ] -1 Do not release this package because
Thank you for pointing out the problem.
The characters and hyperlink point different URLs.
Could you please access
https://repository.apache.org/content/repositories/orgapachespark-1331/ as
you see characters?
Sorry for your inconvenience.
Kazuaki Ishizaki,
From: Takeshi Yamamuro
+1
Built and tested with `mvn -Pyarn -Phadoop-2.7 -Pkubernetes -Pkinesis-asl
-Phive -Phive-thriftserver test` on OpenJDK 1.8.0_211 on Ubuntu 16.04
x86_64
Regards,
Kazuaki Ishizaki
From: Dongjoon Hyun
To: dev
Date: 2019/08/28 12:14
Subject:[EXTERNAL] Re: [VOTE] Release
*
+0: None
-1: None
Regards,
Kazuaki Ishizaki
ext
Monday.
Regards,
Kazuaki Ishizaki
From: "Kazuaki Ishizaki"
To: "Kazuaki Ishizaki"
Cc: Dilip Biswal , dev ,
Hyukjin Kwon , jzh...@apache.org, Takeshi Yamamuro
, Xiao Li
Date: 2019/08/20 13:12
Subject:[EXTERNAL] RE: Release Spark 2.3.4
Due to the r
Congrats! Well deserved.
Kazuaki Ishizaki,
From: Matei Zaharia
To: dev
Date: 2019/09/10 09:32
Subject:[EXTERNAL] Welcoming some new committers and PMC members
Hi all,
The Spark PMC recently voted to add several new committers and one PMC
member. Join me in welcoming them
.
Kazuaki Ishizaki
+1 if we add them to Alternative config.
Kazuaki Ishizaki
From: Takeshi Yamamuro
To: Wenchen Fan
Cc: Spark dev list
Date: 2020/02/13 16:02
Subject:[EXTERNAL] Re: [DISCUSS] naming policy of Spark configs
+1; the idea sounds reasonable.
Bests,
Takeshi
On Thu, Feb 13
Java 16 will also includes Vector API (incubator), which is a part of
Project Panama, as shown in
https://mail.openjdk.java.net/pipermail/panama-dev/2020-October/011149.html
When the next LTS will be available, we could exploit it in Spark.
Kazuaki Ishizaki
From: Dongjoon Hyun
pache.org/docs/3.1.1/...;
Regards,
Kazuaki Ishizaki,
From: Hyukjin Kwon
To: dev
Cc: Dongjoon Hyun , Jungtaek Lim
, Tom Graves
Date: 2021/03/02 11:20
Subject:Re: Please take a look at the draft of the Spark 3.1.1
release notes
Thanks guys for suggestions and fixes. Now I feel pr
68 matches
Mail list logo