Re: [PYSPARK] Python tests organization

2017-01-14 Thread Saikat Kanjilal
iew...@gmail.com> Sent: Thursday, January 12, 2017 10:44 AM To: dev@spark.apache.org Subject: Re: [PYSPARK] Python tests organization Sounds good, but it looks like JIRA is still down. Personally I can look at sql.tests and see what can be done there. Depending on the resolution of https://issues.apac

Re: [PYSPARK] Python tests organization

2017-01-12 Thread Maciej Szymkiewicz
.com > <mailto:mszymkiew...@gmail.com>> > > > > *Sent:* Wednesday, January 11, 2017 10:14 AM > > > *To:* Saikat Kanjilal > > > *Subject:* Re: [PYSPARK] Python tests organization > > > Not yet, I want to see if there is any conse

Re: [PYSPARK] Python tests organization

2017-01-12 Thread Maciej Szymkiewicz
@spark.apache.org > *Subject:* Re: [PYSPARK] Python tests organization > > I'd be happy to help with reviewing Python test improvements. Maybe > make an umbrella JIRA and do one sub components at a time? > > On Thu, Jan 12, 2017 at 12:20 PM Saikat Kanjilal <sxk

Re: [PYSPARK] Python tests organization

2017-01-12 Thread Saikat Kanjilal
Maciej? LGTM, what do you think? I can create a JIRA and drive this. From: Holden Karau <hol...@pigscanfly.ca> Sent: Thursday, January 12, 2017 10:34 AM To: Saikat Kanjilal; dev@spark.apache.org Subject: Re: [PYSPARK] Python tests organization I'd be

Re: [PYSPARK] Python tests organization

2017-01-12 Thread Holden Karau
> > Following up, any thoughts on next steps for this? > > > > > > > > > > > -- > > > *From:* Maciej Szymkiewicz <mszymkiew...@gmail.com> > > > > *Sent:* Wednesday, January 11, 2017 10:14 AM > > > *To:* Saikat Kanjil

Re: [PYSPARK] Python tests organization

2017-01-12 Thread Saikat Kanjilal
Following up, any thoughts on next steps for this? From: Maciej Szymkiewicz <mszymkiew...@gmail.com> Sent: Wednesday, January 11, 2017 10:14 AM To: Saikat Kanjilal Subject: Re: [PYSPARK] Python tests organization Not yet, I want to see if there is any con

Re: [PYSPARK] Python tests organization

2017-01-11 Thread Saikat Kanjilal
ubject: Re: [PYSPARK] Python tests organization Not yet, I want to see if there is any consensus about it. It is a lot of tedious work and I would be shame if someone started working on this just to get it dropped. On 01/11/2017 06:44 PM, Saikat Kanjilal wrote: Hello Maciej, If there's

Re: [PYSPARK] Python tests organization

2017-01-11 Thread Reynold Xin
; > -- > > > *From:* Reynold Xin <r...@databricks.com> > > > *Sent:* Wednesday, January 11, 2017 9:47 AM > > > *To:* Maciej Szymkiewicz; Saikat Kanjilal; dev@spark.apache.org > > > *Subject:* Re: [PYSPARK] Python tests organization >

Re: [PYSPARK] Python tests organization

2017-01-11 Thread Saikat Kanjilal
Is it worth to come up with a proposal for this and float to dev? From: Reynold Xin <r...@databricks.com> Sent: Wednesday, January 11, 2017 9:47 AM To: Maciej Szymkiewicz; Saikat Kanjilal; dev@spark.apache.org Subject: Re: [PYSPARK] Python tests organi

Re: [PYSPARK] Python tests organization

2017-01-11 Thread Reynold Xin
t;mszymkiew...@gmail.com> > > > *Sent:* Wednesday, January 11, 2017 4:18 AM > > > *To:* dev@spark.apache.org > > > *Subject:* [PYSPARK] Python tests organization > > > > > > > > > > > Hi, > > > > > > I can't help but wonder if

Re: [PYSPARK] Python tests organization

2017-01-11 Thread Saikat Kanjilal
ubject: [PYSPARK] Python tests organization Hi, I can't help but wonder if there is any practical reason for keeping monolithic test modules. These things are already pretty large (1500 - 2200 LOCs) and can only grow. Development aside, I assume that many users use tests the same way as me, to

[PYSPARK] Python tests organization

2017-01-11 Thread Maciej Szymkiewicz
Hi, I can't help but wonder if there is any practical reason for keeping monolithic test modules. These things are already pretty large (1500 - 2200 LOCs) and can only grow. Development aside, I assume that many users use tests the same way as me, to check the intended behavior, and largish