unsubscribe

[????????]

unsubscribe

[刘杰]

unsubscribe

[Kiran B]

Thank you,
Kiran,

unsubscribe

[Chenguang He]

unsubscribe

-- 

Chenguang He

unsubscribe

[Hongbin Liu]

unsubscribe







This message may contain confidential information and is intended for specific 
recipients unless explicitly noted otherwise. If you have reason to believe you 
are not an intended recipient of this message, please delete it and notify the 
sender. This message may not represent the opinion of Intercontinental 
Exchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a 
contract or guarantee. Unencrypted electronic mail is not secure and the 
recipient of this message is expected to provide safeguards from viruses and 
pursue alternate means of communication where privacy or a binding message is 
desired.

unsubscribe

[akram azarm]

-- 
*M Akram Azarm*
*B Eng. in Software Engineering (Reading)*
*UOW,UK / IIT,LK*

Contact | 077-502-0402

unsubscribe

[Hemanth meka]

Re: Unsubscribe

[Zahid Rahman]

To unsubscribe e-mail: dev-unsubscr...@spark.apache.org


Backbutton.co.uk
¯\_(ツ)_/¯
♡۶Java♡۶RMI ♡۶
Make Use Method {MUM}
makeuse.org
<http://www.backbutton.co.uk>


On Sun, 22 Mar 2020 at 16:16, Duan,Bing  wrote:

> Hi:
>  Plz  Unsubscribe me.
>
> Thanks!
>
> ---------
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Unsubscribe

[Duan,Bing]

Hi: 
 Plz  Unsubscribe me.

Thanks!

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Unsubscribe

[chenzhihan]

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Unsubscribe

[Stepan Tuchin]

Unsubscribe
-- 

[image: Brandmark_small.jpg]

Stepan Tuchin, Automation Quality Engineer

Grid Dynamics

Vavilova, 38/114, Saratov

Dir: +7 (902) 047-55-55

Unsubscribe

[Sharanabasappa G Keriwaddi (Sharan, IT & Cloud BL R)]

Re: Unsubscribe

[William R]

Unsubscribe

[Akhil Anil]

-- 
Sent from Gmail Mobile

Unsubscribe

[Akhil Anil]

-- 
Sent from Gmail Mobile

unsubscribe

[Taoufik Dachraoui]

Unsubscribe

[sadhana avasarala]

 

 

From: Dongjoon Hyun 
Date: Wednesday, January 22, 2020 at 1:57 AM
To: Wenchen Fan 
Cc: dev 
Subject: Re: Correctness and data loss issues

 

Thank you for checking, Wenchen! Sure, we need to do that.

 

Another question is "What can we do for 2.4.5 release"?
Some of the fixes cannot be backported due to the technical difficulty like the 
followings.

1. https://issues.apache.org/jira/browse/SPARK-26154
Stream-stream joins - left outer join gives inconsistent output

(Like this, there are eight correctness fixes which lands only at 3.0.0)


2. https://github.com/apache/spark/pull/27233
[SPARK-29701][SQL] Correct behaviours of group analytical queries when 
empty input given
(This is on-going PR which is currently blocking 2.4.5 RC2).

Bests,
Dongjoon.

 

On Tue, Jan 21, 2020 at 11:10 PM Wenchen Fan  wrote:

I think we need to go through them during the 3.0 QA period, and try to fix the 
valid ones.

 

For example, the first ticket should be fixed already in 
https://issues.apache.org/jira/browse/SPARK-28344

 

On Mon, Jan 20, 2020 at 2:07 PM Dongjoon Hyun  wrote:

Hi, All.

 

According to our policy, "Correctness and data loss issues should be considered 
Blockers".

- http://spark.apache.org/contributing.html


Since we are close to branch-3.0 cut,
I want to ask your opinions on the following correctness and data loss issues.

SPARK-30218 Columns used in inequality conditions for joins not resolved 
correctly in case of common lineage
SPARK-29701 Different answers when empty input given in GROUPING SETS
SPARK-29699 Different answers in nested aggregates with window functions
SPARK-29419 Seq.toDS / spark.createDataset(Seq) is not thread-safe
SPARK-28125 dataframes created by randomSplit have overlapping rows
SPARK-28067 Incorrect results in decimal aggregation with whole-stage code 
gen enabled
SPARK-28024 Incorrect numeric values when out of range
SPARK-27784 Alias ID reuse can break correctness when substituting foldable 
expressions
SPARK-27619 MapType should be prohibited in hash expressions
SPARK-27298 Dataset except operation gives different results(dataset count) 
on Spark 2.3.0 Windows and Spark 2.3.0 Linux environment
SPARK-27282 Spark incorrect results when using UNION with GROUP BY clause
SPARK-27213 Unexpected results when filter is used after distinct
SPARK-26836 Columns get switched in Spark SQL using Avro backed Hive table 
if schema evolves
SPARK-25150 Joining DataFrames derived from the same source yields 
confusing/incorrect results
SPARK-21774 The rule PromoteStrings cast string to a wrong data type
SPARK-19248 Regex_replace works in 1.6 but not in 2.0

Some of them are targeted on 3.0.0, but the others are not.
Although we will work on them until 3.0.0,

I'm not sure we can reach a status with no known correctness and data loss 
issue.

 

How do you think about the above issues?

 

Bests,

Dongjoon.

unsubscribe

[Bruno S. de Barros]

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

unsubscribe

[Pingxiao Ye]

unsubscribe

[vijendra rana]

unsubscribe

Re: unsubscribe

[steve goodwin]

unsubscribe

unsubscribe

[Taoufik Dachraoui]

unsubscribe

[Amit Rana]

unsubscribe

unsubscribe

[Taoufik Dachraoui]

unsubscribe

unsubscribe

[William R]

unsubscribe

unsubscribe

[MEETHU MATHEW]

Thanks & Regards, Meethu M

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

unsubscribe

[Taoufik Dachraoui]

unsubscribe

unsubscribe

[Bruno Soares De Barros]

unsubscribe

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

unsubscribe

[vijay krishna]

unsubscribe

[Narendra M]

Unsubscribe

[abel palaty]

Unsubscribe

[Praveen Srivastava]

 

 

From: Jean-Georges Perrin  
Sent: Saturday, December 28, 2019 11:08 PM
To: dev 
Subject: Issue with map Java lambda function with 3.0.0 preview and preview 2

 

Hey guys,

 

This code:

 

Dataset incrementalDf = spark

.createDataset(l, Encoders.INT())

.toDF();

Dataset dotsDs = incrementalDf

.map(status -> {

  double x = Math.random() * 2 - 1;

  double y = Math.random() * 2 - 1;

  counter++;

  if (counter % 10 == 0) {

System.out.println("" + counter + " darts thrown so far");

  }

  return (x * x + y * y <= 1) ? 1 : 0;

}, Encoders.INT());

 

used to work with Spark 2.x, in the two previous, it says:

 

The method map(Function1, Encoder) is ambiguous for the 
type Dataset

 

IfI define my mapping function as a class it works fine. Here is the class:

 

  private final class DartMapper

  implements MapFunction {

private static final long serialVersionUID = 38446L;

 

@Override

public Integer call(Row r) throws Exception {

  double x = Math.random() * 2 - 1;

  double y = Math.random() * 2 - 1;

  counter++;

  if (counter % 1000 == 0) {

System.out.println("" + counter + " operations done so far");

  }

  return (x * x + y * y <= 1) ? 1 : 0;

}

  }

 

Any hint on what/if I did wrong? 

 

jg

 

 

 

unsubscribe

[Deepak Sharma]

unsubscribe;

[余帆帆]

unsubscribe

Unsubscribe

[abel palaty]

Unsubscribe

[Kiran B]

-- 
Thank you, Kiran, 9959311399.

unsubscribe

[Jason Man, CLSA]

unsubscribe
The content of this communication is intended for the recipient and is subject 
to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon 
request.
Please consider before printing. CLSA is ISO14001 certified and committed to 
reducing its impact on the environment.

unsubscribe

[Duan,Bing]

Unsubscribe.



发自我的小米手机
在 2019年9月10日 上午8:32，Matei Zaharia 写道：
Hi all,

The Spark PMC recently voted to add several new committers and one PMC member. 
Join me in welcoming them to their new roles!

New PMC member: Dongjoon Hyun

New committers: Ryan Blue, Liang-Chi Hsieh, Gengliang Wang, Yuming Wang, 
Weichen Xu, Ruifeng Zheng

The new committers cover lots of important areas including ML, SQL, and data 
sources, so it’s great to have them here. All the best,

Matei and the Spark PMC


-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

unsubscribe

[Mario Amatucci]

unsubscribe

send this email to unsubscribe

[Ankit Kumar]

unsubscribe

[Tianji Li]

unsubscribe

[Charan Gupta]

unsubscribe

unsubscribe

[Duan,Bing]

unsubscribe.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: unsubscribe

[Duan,Bing]

unsubscribe

On Aug 1, 2019, at 6:36 PM, abel palaty 
mailto:palatya...@gmail.com>> wrote:

unsubscribe

[abel palaty]

unsubscribe

[Oleksandr Vayda]

unsubscribe

unsubscribe

[Charan Gupta]

unsubscribe

unsubscribe

[Bill Bejeck]

unsubscribe

Re: unsubscribe

[Swapnil M Mane]

Hello Tushar,
You need to help yourself to unsubscribe from the mailing list,
Please send a blank mail on dev-unsubscr...@spark.apache.org


- Best Regards,
Swapnil M Mane

On Wed, Jun 19, 2019 at 3:42 PM Tushar Marne  wrote:

>
>
> --
> Tushar Marne
> 9011062432
>

unsubscribe

[Tushar Marne]

-- 
Tushar Marne
9011062432

unsubscribe

[Matteo Bovetti]

[agilelab_logo]
Matteo Bovetti
Big Data Engineer & DevOps
Mobile: +39 333 290 1242
Email: matteo.bove...@agilelab.it
Site: www.agilelab.it

unsubscribe

[ehsan shams]

unsubscribe

[Raja Pasupuleti]

unsubscribe

[mayur bhole]

Unsubscribe

[Moise]

Unsubscribe
-- 
Kalenda Moise Kabukala

"The newest computer can merely compound, at speed, the oldest problem in
the
relations between human beings, and in the end the communicator will be
confronted
with the old problem, of what to say and how to say it."
—Edward R. Murrow

Unsubscribe

[Arkadiusz Bicz]

>
> unsubscribe
>

Re: Unsubscribe

[William Shen]

Vinod,
You can send an email to dev-unsubscr...@spark.apache.org to unsubscribe.
You should receive an email with instruction to confirm the unsubscribe.

On Sun, Mar 31, 2019 at 7:42 AM Vinod V Rangayyan 
wrote:

> I wish to unsubscribe from dev@spark.apache.org
>
>
>
>
> On Mar 31, 2019, at 10:07 AM, Rubén Berenguel 
> wrote:
>
> I favour using either $”foo” or columnar expressions, but know of several
> developers who prefer single quote syntax and consider it a better practice.
>
> R
>
> On 31 March 2019 at 15:15:00, Sean Owen (sro...@apache.org) wrote:
>
>> FWIW I use "foo" in Pyspark or col("foo") where necessary, and $"foo" in
>> Scala
>>
>> On Sun, Mar 31, 2019 at 1:58 AM Reynold Xin  wrote:
>>
>>> As part of evolving the Scala language, the Scala team is considering
>>> removing single-quote syntax for representing symbols. Single-quote syntax
>>> is one of the ways to represent a column in Spark's DataFrame API. While I
>>> personally don't use them (I prefer just using strings for column names, or
>>> using expr function), I see them used quite a lot by other people's code,
>>> e.g.
>>>
>>> df.select('id, 'name).show()
>>>
>>> I want to bring this to more people's attention, in case they are
>>> depending on this. The discussion thread is:
>>> https://contributors.scala-lang.org/t/proposal-to-deprecate-and-remove-symbol-literals/2953
>>>
>>>
>>>
>>>
>

Unsubscribe

[Vinod V Rangayyan]

I wish to unsubscribe from dev@spark.apache.org




> On Mar 31, 2019, at 10:07 AM, Rubén Berenguel  wrote:
> 
> I favour using either $”foo” or columnar expressions, but know of several 
> developers who prefer single quote syntax and consider it a better practice.
> 
> R
> 
> On 31 March 2019 at 15:15:00, Sean Owen (sro...@apache.org 
> <mailto:sro...@apache.org>) wrote:
> 
>> FWIW I use "foo" in Pyspark or col("foo") where necessary, and $"foo" in 
>> Scala
>> 
>> On Sun, Mar 31, 2019 at 1:58 AM Reynold Xin > <mailto:r...@databricks.com>> wrote:
>> As part of evolving the Scala language, the Scala team is considering 
>> removing single-quote syntax for representing symbols. Single-quote syntax 
>> is one of the ways to represent a column in Spark's DataFrame API. While I 
>> personally don't use them (I prefer just using strings for column names, or 
>> using expr function), I see them used quite a lot by other people's code, 
>> e.g.
>> 
>> df.select <http://df.select/>('id, 'name).show()
>> 
>> I want to bring this to more people's attention, in case they are depending 
>> on this. The discussion thread is: 
>> https://contributors.scala-lang.org/t/proposal-to-deprecate-and-remove-symbol-literals/2953
>>  
>> <https://contributors.scala-lang.org/t/proposal-to-deprecate-and-remove-symbol-literals/2953>
>> 
>> 
>> 

Unsubscribe

[AADIL AHMAD]

Unsubscribe

Re: Unsubscribe

[William Shen]

Please send an email to dev-unsubscr...@spark.apache.org to unsubscribe.
You should receive an email with instruction to confirm the unsubscribe.

On Wed, Feb 20, 2019 at 3:58 PM Reena Agrawal  wrote:

> Unsubscribe pls.
>

Unsubscribe

[Reena Agrawal]

Unsubscribe pls.

Unsubscribe

[northbright]

Unsubscribe pls

Unsubscribe

[Iryna Kharaborkina]

unsubscribe

[Ryan Adams]

unsubscribe

unsubscribe

[Praveen Srivastava]

unsubscribe




Praveen Srivastava 
HYPERLINK 
"mailto:praveen.s.srivast...@oracle.com"praveen.s.srivast...@oracle.com

unsubscribe

[Ryan Adams]

unsubscribe

Ryan Adams
radams...@gmail.com

Unsubscribe

[Happy??????]

Unsubscribe

[Michael Styles]

unsubscribe

[Tarun Kumar]

unsubscribe

[Al Pivonka]

-- 
Those who say it can't be done, are usually interrupted by those doing it.

UNSUBSCRIBE

[sridhararao mutluri]

Unsubscribe

[Prateek Goel]

Unsubscribe

[xu han]

Unsubscribe

[Tripathi, Abhishek]

Unsubscribe
This message contains information that may be privileged or confidential and is 
the property of the Capgemini Group. It is intended only for the person to whom 
it is addressed. If you are not the intended recipient, you are not authorized 
to read, print, retain, copy, disseminate, distribute, or use this message or 
any part thereof. If you receive this message in error, please notify the 
sender immediately and delete all copies of this message.

Re: Unsubscribe

[xu han]

 Unsubscribe

On Fri, Jun 22, 2018 at 4:33 PM, Tarun Kumar  wrote:

> Unsubscribe

Unsubscribe

[Игорь Скоков]

Unsubscribe

Unsubscribe

[Tarun Kumar]

Unsubscribe

Unsubscribe

[Uzi Hadad]

Get Outlook for iOS<https://aka.ms/o0ukef>

From: Thiago 
Sent: Thursday, June 14, 2018 9:28:14 PM
To: dev@spark.apache.org
Subject: Unsubscribe


Unsubscribe

Re: Unsubscribe

[Hadrien Chicault]

Le ven. 15 juin 2018 à 23:17, Mikhail Dubkov  a
écrit :

> Unsubscribe
>
> On Thu, Jun 14, 2018 at 8:38 PM Kumar S, Sajive 
> wrote:
>
>> Unsubscribe
>>
>

Unsubscribe

[Mikhail Dubkov]

Unsubscribe

On Thu, Jun 14, 2018 at 8:38 PM Kumar S, Sajive 
wrote:

> Unsubscribe
>

Unsubscribe

[Kumar S, Sajive]

Unsubscribe

Unsubscribe

[Congxian Qiu]

Unsubscribe

-- 
Blog:http://www.klion26.com
GTalk:qcx978132955
一切随心

Unsubscribe

[Mohamed Gabr]

Unsubscribe

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Unsubscribe

[Thiago]

Unsubscribe

unsubscribe

[Huamin Li]

unsubscribe

unsubscribe

[Vasilis Hadjipanos]

unsubscribe

Unsubscribe

[Al Pivonka]

On Mon, Jun 4, 2018, 1:18 PM Joseph Bradley  wrote:

> +1
>
> On Mon, Jun 4, 2018 at 10:16 AM, Mark Hamstra 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 3:29 PM Marcelo Vanzin 
>> wrote:
>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 2.3.1.
>>>
>>> Given that I expect at least a few people to be busy with Spark Summit
>>> next
>>> week, I'm taking the liberty of setting an extended voting period. The
>>> vote
>>> will be open until Friday, June 8th, at 19:00 UTC (that's 12:00 PDT).
>>>
>>> It passes with a majority of +1 votes, which must include at least 3 +1
>>> votes
>>> from the PMC.
>>>
>>> [ ] +1 Release this package as Apache Spark 2.3.1
>>> [ ] -1 Do not release this package because ...
>>>
>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>
>>> The tag to be voted on is v2.3.1-rc4 (commit 30aaa5a3):
>>> https://github.com/apache/spark/tree/v2.3.1-rc4
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.3.1-rc4-bin/
>>>
>>> Signatures used for Spark RCs can be found in this file:
>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>
>>> The staging repository for this release can be found at:
>>> https://repository.apache.org/content/repositories/orgapachespark-1272/
>>>
>>> The documentation corresponding to this release can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.3.1-rc4-docs/
>>>
>>> The list of bug fixes going into 2.3.1 can be found at the following URL:
>>> https://issues.apache.org/jira/projects/SPARK/versions/12342432
>>>
>>> FAQ
>>>
>>> =
>>> How can I help test this release?
>>> =
>>>
>>> If you are a Spark user, you can help us test this release by taking
>>> an existing Spark workload and running on this release candidate, then
>>> reporting any regressions.
>>>
>>> If you're working in PySpark you can set up a virtual env and install
>>> the current RC and see if anything important breaks, in the Java/Scala
>>> you can add the staging repository to your projects resolvers and test
>>> with the RC (make sure to clean up the artifact cache before/after so
>>> you don't end up building with a out of date RC going forward).
>>>
>>> ===
>>> What should happen to JIRA tickets still targeting 2.3.1?
>>> ===
>>>
>>> The current list of open tickets targeted at 2.3.1 can be found at:
>>> https://s.apache.org/Q3Uo
>>>
>>> Committers should look at those and triage. Extremely important bug
>>> fixes, documentation, and API tweaks that impact compatibility should
>>> be worked on immediately. Everything else please retarget to an
>>> appropriate release.
>>>
>>> ==
>>> But my bug isn't fixed?
>>> ==
>>>
>>> In order to make timely releases, we will typically not hold the
>>> release unless the bug in question is a regression from the previous
>>> release. That being said, if there is something which is a regression
>>> that has not been correctly targeted please ping me or a committer to
>>> help target the issue.
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>
>
>
> --
>
> Joseph Bradley
>
> Software Engineer - Machine Learning
>
> Databricks, Inc.
>
> [image: http://databricks.com] <http://databricks.com/>
>

unsubscribe

[Hadrien Chicault]

 unsubscribe

unsubscribe

[Deepesh Maheshwari]

unsubscribe

[hari haran]

-- 
- Hariharan M K

Unsubscribe

[varma dantuluri]

Unsubscribe

-- 
Regards,

Varma Dantuluri

unsubscribe

[varma dantuluri]

unsubscribe
-- 
Regards,

Varma Dantuluri

unsubscribe

[韩盼]

unsubscribe

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

unsubscribe

[何琪]

unsubscribe

[marco rocchi]

unsubscribe

unsubscribe

[Steve Johnston]

unsubscribe

[Al Pivonka]

unsubscribe

-- 
Those who say it can't be done, are usually interrupted by those doing it.

Unsubscribe

[Pavel Gladkov]

Unsubscribe

-- 
С уважением,
Павел Гладков

Unsubscribe

[Praveen Kumar]

Unsubscribe

[praba karan]

Sent from Yahoo Mail on Android