Re: [Pyspark 2.3+] Timeseries with Spark

Time series can mean a lot of different things and algorithms. Can you describe more what you mean by time series use case, ie what is the input, what do you like to do with the input and what is the output? > Am 14.06.2019 um 06:01 schrieb Rishi Shah : > > Hi All, > > I have a time series

[Pyspark 2.3+] Timeseries with Spark

Hi All, I have a time series use case which I would like to implement in Spark... What would be the best way to do so? Any built in libraries? -- Regards, Rishi Shah

Spark on Yarn - Dynamically getting a list of archives from --archives in spark-submit

Hi Is there any way to get a list of the archives submitted with a spark job from the spark context? I see that spark context has a `.files()` function which returns the files included with `--files`, but I don't see an equivalent for `--archives`. Thanks, Tommy

Re: [Spark Core]: What is the release date for Spark 3 ?

next spark summit On Thu, Jun 13, 2019 at 3:58 AM Alex Dettinger wrote: > Follow up on the release date for Spark 3. Any guesstimate or rough > estimation without commitment would be helpful :) > > Cheers, > Alex > > On Mon, Jun 10, 2019 at 5:24 PM Alex Dettinger > wrote: > >> Hi guys, >> >>

Re: Exposing JIRA issue types at GitHub PRs

Thank you for the feedbacks and requirements, Hyukjin, Reynold, Marco. Sure, we can do whatever we want. I'll wait for more feedbacks and proceed to the next steps. Bests, Dongjoon. On Wed, Jun 12, 2019 at 11:51 PM Marco Gaido wrote: > Hi Dongjoon, > Thanks for the proposal! I like the

Re: best docker image to use

Thanks Riccardo. This is useful, and it seems it's maintained by jupyter team. I was hoping I would find some maintained by spark team. Right now, I am using the base images from this repo: https://github.com/big-data-europe/docker-spark/ -Marcelo On Tue, 11 Jun 2019 at 12:19, Riccardo Ferrari

Re: [Spark Core]: What is the release date for Spark 3 ?

Follow up on the release date for Spark 3. Any guesstimate or rough estimation without commitment would be helpful :) Cheers, Alex On Mon, Jun 10, 2019 at 5:24 PM Alex Dettinger wrote: > Hi guys, > > I was not able to find the foreseen release date for Spark 3. > Would one have any

Re: Exposing JIRA issue types at GitHub PRs

Hi Dongjoon, Thanks for the proposal! I like the idea. Maybe we can extend it to component too and to some jira labels such as correctness which may be worth to highlight in PRs too. My only concern is that in many cases JIRAs are created not very carefully so they may be incorrect at the moment

Re: Exposing JIRA issue types at GitHub PRs

Seems like a good idea. Can we test this with a component first? On Thu, Jun 13, 2019 at 6:17 AM Dongjoon Hyun wrote: > Hi, All. > > Since we use both Apache JIRA and GitHub actively for Apache Spark > contributions, we have lots of JIRAs and PRs consequently. One specific > thing I've been

Re: Exposing JIRA issue types at GitHub PRs

Yea, I think we can automate this process via, for instance, https://github.com/apache/spark/blob/master/dev/github_jira_sync.py +1 for such sort of automatic categorizing and matching metadata between JIRA and github Adding Josh and Sean as well. On Thu, 13 Jun 2019, 13:17 Dongjoon Hyun,