Re: Apache Spark git repo moved to gitbox.apache.org

2018-12-18 Thread Hyukjin Kwon
Similar issues are going on in spark-website as well. I also filed a ticket at https://issues.apache.org/jira/browse/INFRA-17469. 2018년 12월 12일 (수) 오전 9:02, Reynold Xin 님이 작성: > I filed a ticket: https://issues.apache.org/jira/browse/INFRA-17403 > > Please add your support there. > > > On Tue, De

Re: DataSourceV2 sync notes (#4)

2018-12-18 Thread Srabasti Banerjee
Thanks for sending out the meeting notes from last week's discussion Ryan! For technical unknown reasons, I could not unmute myself and be heard when I was trying to pitch in during one of the topic discussions regarding default value handling for traditional databases. Had posted response in ch

[DISCUSS] Default values and data sources

2018-12-18 Thread Ryan Blue
Hi everyone, This thread is a follow-up to a discussion that we started in the DSv2 community sync last week. The problem I’m trying to solve is that the format I’m using DSv2 to integrate supports schema evolution. Specifically, adding a new optional column so that rows without that column get a

DataSourceV2 sync notes (#4)

2018-12-18 Thread Ryan Blue
Hi everyone, sorry these notes are late. I didn’t have the time to write this up last week. For anyone interested in the next sync, we decided to skip next week and resume in early January. I’ve already sent the invite. As usual, if you have topics you’d like to discuss or would like to be added t

Re: [DISCUSS] Function plugins

2018-12-18 Thread Ryan Blue
I agree that it probably isn’t feasible to support codegen. My goal is to be able to have users code like they can in Scala, but change registration so that they don’t need a SparkSession. This is easy with a SparkSession: In [2]: def plus(a: Int, b: Int): Int = a + b plus: (a: Int, b: Int)Int I

[Standalone Spark] Master Configuration Push-Down

2018-12-18 Thread Sean Po
I am running Spark Standalone mode and I am finding that when I configure ports (i.e. spark.blockManager.port) in both the Spark Master's spark-defaults.conf as well as the Spark Worker's, that the Spark Master's port is the one that will be used in all the workers. Judging by the code, this see

Re: Decimals with negative scale

2018-12-18 Thread Reynold Xin
So why can't we just do validation to fail sources that don't support negative scale, if it is not supported? This way, we don't need to break backward compatibility in anyway and it becomes a strict improvement. On Tue, Dec 18, 2018 at 8:43 AM, Marco Gaido < marcogaid...@gmail.com > wrote: >

Re: Decimals with negative scale

2018-12-18 Thread Marco Gaido
This is at analysis time. On Tue, 18 Dec 2018, 17:32 Reynold Xin Is this an analysis time thing or a runtime thing? > > On Tue, Dec 18, 2018 at 7:45 AM Marco Gaido > wrote: > >> Hi all, >> >> as you may remember, there was a design doc to support operations >> involving decimals with negative sc

Re: Decimals with negative scale

2018-12-18 Thread Reynold Xin
Is this an analysis time thing or a runtime thing? On Tue, Dec 18, 2018 at 7:45 AM Marco Gaido wrote: > Hi all, > > as you may remember, there was a design doc to support operations > involving decimals with negative scales. After the discussion in the design > doc, now the related PR is blocked

Re: [Apache Beam] Custom DataSourceV2 instanciation: parameters passing and Encoders

2018-12-18 Thread Etienne Chauchot
Hi everyone, Does anyone have comments on this question? CCing user ML ThanksEtienne Le mardi 11 décembre 2018 à 19:02 +0100, Etienne Chauchot a écrit : > Hi Spark guys, > I'm Etienne Chauchot and I'm a committer on the Apache Beam project. > We have what we call runners. They are pieces of soft

Decimals with negative scale

2018-12-18 Thread Marco Gaido
Hi all, as you may remember, there was a design doc to support operations involving decimals with negative scales. After the discussion in the design doc, now the related PR is blocked because for 3.0 we have another option which we can explore, ie. forbidding negative scales. This is probably a c

[GitHub] HyukjinKwon closed pull request #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools

2018-12-18 Thread GitBox
HyukjinKwon closed pull request #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools URL: https://github.com/apache/spark-website/pull/162 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below

[GitHub] HyukjinKwon commented on issue #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools

2018-12-18 Thread GitBox
HyukjinKwon commented on issue #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools URL: https://github.com/apache/spark-website/pull/162#issuecomment-448164740 Thanks guys! This is an au