[jira] [Commented] (SPARK-25527) Job stuck waiting for last stage to start

[ https://issues.apache.org/jira/browse/SPARK-25527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649990#comment-16649990 ] Ran Haim commented on SPARK-25527: -- Any update? > Job stuck waiting for last stage to start >

[jira] [Commented] (SPARK-25527) Job stuck waiting for last stage to start

[ https://issues.apache.org/jira/browse/SPARK-25527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628392#comment-16628392 ] Ran Haim commented on SPARK-25527: -- Hi, actually as I said there is one task (in the last stage) that

[jira] [Updated] (SPARK-25527) Job stuck waiting for last stage to start

[ https://issues.apache.org/jira/browse/SPARK-25527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ran Haim updated SPARK-25527: - Attachment: threaddumpjob.txt > Job stuck waiting for last stage to start >

[jira] [Created] (SPARK-25527) Job stuck waiting for last stage to start

Ran Haim created SPARK-25527: Summary: Job stuck waiting for last stage to start Key: SPARK-25527 URL: https://issues.apache.org/jira/browse/SPARK-25527 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-22481) CatalogImpl.refreshTable is slow

[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248837#comment-16248837 ] Ran Haim edited comment on SPARK-22481 at 11/12/17 12:31 PM: - Hi, I created

[jira] [Commented] (SPARK-22481) CatalogImpl.refreshTable is slow

[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248837#comment-16248837 ] Ran Haim commented on SPARK-22481: -- Hi, I create this simple test that will show how slow refresh table

[jira] [Commented] (SPARK-22481) CatalogImpl.refreshTable is slow

[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247913#comment-16247913 ] Ran Haim commented on SPARK-22481: -- No, it is not. > CatalogImpl.refreshTable is slow >

[jira] [Commented] (SPARK-22481) CatalogImpl.refreshTable is slow

[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247858#comment-16247858 ] Ran Haim commented on SPARK-22481: -- Ok, I'll create a simple app to reproduce this, later next week. >

[jira] [Commented] (SPARK-22481) CatalogImpl.refreshTable is slow

[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247852#comment-16247852 ] Ran Haim commented on SPARK-22481: -- I will double check on Sunday, but sparksession.table in the end

[jira] [Commented] (SPARK-22481) CatalogImpl.refreshTable is slow

[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247838#comment-16247838 ] Ran Haim commented on SPARK-22481: -- I can check it again on Sunday. I don't know why it is suprising, as

[jira] [Comment Edited] (SPARK-22481) CatalogImpl.refreshTable is slow

[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247828#comment-16247828 ] Ran Haim edited comment on SPARK-22481 at 11/10/17 5:47 PM: It is as I wrote

[jira] [Commented] (SPARK-22481) CatalogImpl.refreshTable is slow

[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247828#comment-16247828 ] Ran Haim commented on SPARK-22481: -- It is as I wrote above. It takes 1 minute in 2.1.1 and 2 seconds in

[jira] [Commented] (SPARK-22481) CatalogImpl.refreshTable is slow

[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246176#comment-16246176 ] Ran Haim commented on SPARK-22481: -- It takes about 2 seconds to create the dataset...i need to refresh

[jira] [Updated] (SPARK-22481) CatalogImpl.refreshTable is slow

[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ran Haim updated SPARK-22481: - Description: CatalogImpl.refreshTable was updated in 2.1.1 and since than it has become really slow.

[jira] [Updated] (SPARK-22481) CatalogImpl.refreshTable is slow

[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ran Haim updated SPARK-22481: - Description: CatalogImpl.refreshTable was updated in 2.1.1 and since than it has become really slow.

[jira] [Updated] (SPARK-22481) CatalogImpl.refreshTable is slow

[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ran Haim updated SPARK-22481: - Description: CatalogImpl.refreshTable was updated in 2.1.1 and since than it has become really slow.

[jira] [Created] (SPARK-22481) CatalogImpl.refreshTable is slow

Ran Haim created SPARK-22481: Summary: CatalogImpl.refreshTable is slow Key: SPARK-22481 URL: https://issues.apache.org/jira/browse/SPARK-22481 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931619#comment-15931619 ] Ran Haim commented on SPARK-17436: -- Hi, I think we need to reopen this. It seems that on

[jira] [Commented] (SPARK-19360) Spark 2.X does not support stored by cluase

[ https://issues.apache.org/jira/browse/SPARK-19360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837503#comment-15837503 ] Ran Haim commented on SPARK-19360: -- Hi, I added my own storage handler actually - and now I cannot use

[jira] [Created] (SPARK-19360) Spark 2.X does not support stored by cluase

Ran Haim created SPARK-19360: Summary: Spark 2.X does not support stored by cluase Key: SPARK-19360 URL: https://issues.apache.org/jira/browse/SPARK-19360 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833367#comment-15833367 ] Ran Haim commented on SPARK-17436: -- Hi, I did not actually use 2.1 yet - so I cannot be 100% sure. I can

[jira] [Comment Edited] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15680994#comment-15680994 ] Ran Haim edited comment on SPARK-17436 at 11/20/16 12:06 PM: - Sure , I

[jira] [Updated] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ran Haim updated SPARK-17436: - Description: update *** It seems that in spark 2.1 code, the sorting issue is resolved. The

[jira] [Updated] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ran Haim updated SPARK-17436: - Description: *** update It seems that in spark 2.0 code, the sorting issue is resolved. The sorter

[jira] [Updated] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ran Haim updated SPARK-17436: - Description: update *** It seems that in spark 2.0 code, the sorting issue is resolved. The

[jira] [Updated] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ran Haim updated SPARK-17436: - Priority: Minor (was: Major) > dataframe.write sometimes does not keep sorting >

[jira] [Updated] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ran Haim updated SPARK-17436: - Description: When using partition by, datawriter can sometimes mess up an ordered dataframe. The

[jira] [Comment Edited] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15680994#comment-15680994 ] Ran Haim edited comment on SPARK-17436 at 11/20/16 11:30 AM: - Sure , I

[jira] [Comment Edited] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15680994#comment-15680994 ] Ran Haim edited comment on SPARK-17436 at 11/20/16 11:29 AM: - Sure -

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15680994#comment-15680994 ] Ran Haim commented on SPARK-17436: -- Sure. Basically I propose to stop using UnsafeKVExternalSorter, and

[jira] [Comment Edited] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15679204#comment-15679204 ] Ran Haim edited comment on SPARK-17436 at 11/19/16 12:44 PM: - Hi, When you

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15679204#comment-15679204 ] Ran Haim commented on SPARK-17436: -- Hi, When you want to write your data to orc files or perquet files,

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674032#comment-15674032 ] Ran Haim commented on SPARK-17436: -- I have basiaclly cloned the repository from

[jira] [Comment Edited] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674032#comment-15674032 ] Ran Haim edited comment on SPARK-17436 at 11/17/16 3:48 PM: I have basiaclly

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673230#comment-15673230 ] Ran Haim commented on SPARK-17436: -- I am running thee build on linux, so this is not it. It is something

[jira] [Comment Edited] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664233#comment-15664233 ] Ran Haim edited comment on SPARK-17436 at 11/14/16 3:45 PM: So... Can you

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664233#comment-15664233 ] Ran Haim commented on SPARK-17436: -- So... Can you help? Or at least point me to someone who can? BTW On

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663358#comment-15663358 ] Ran Haim commented on SPARK-17436: -- Hey, I'll do a pull and try again How to I gain access to create

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663345#comment-15663345 ] Ran Haim commented on SPARK-17436: -- Look, I think this is a serious bug - it makes parquet files and

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663343#comment-15663343 ] Ran Haim commented on SPARK-17436: -- but the whole thing is that the partitioning here actually happens a

[jira] [Comment Edited] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661640#comment-15661640 ] Ran Haim edited comment on SPARK-17436 at 11/13/16 3:37 PM: Hi, I only got a

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661640#comment-15661640 ] Ran Haim commented on SPARK-17436: -- Hi, I only got a chance to work on it now. I saw that the whole

[jira] [Comment Edited] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611483#comment-15611483 ] Ran Haim edited comment on SPARK-17436 at 10/27/16 10:35 AM: - Usually you

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611483#comment-15611483 ] Ran Haim commented on SPARK-17436: -- usually you partition the data, and then you order it - this way you

[jira] [Comment Edited] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611457#comment-15611457 ] Ran Haim edited comment on SPARK-17436 at 10/27/16 10:32 AM: - Of course it

[jira] [Comment Edited] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611457#comment-15611457 ] Ran Haim edited comment on SPARK-17436 at 10/27/16 10:22 AM: - Of course it

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611457#comment-15611457 ] Ran Haim commented on SPARK-17436: -- Of course it does, every technology that supports partitioning

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611331#comment-15611331 ] Ran Haim commented on SPARK-17436: -- anyone? > dataframe.write sometimes does not keep sorting >

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15471291#comment-15471291 ] Ran Haim commented on SPARK-17436: -- I tries it under 1.6.1, but I did not see anything that fixes that

[jira] [Created] (SPARK-17436) dataframe.write sometimes does not keep sorting

Ran Haim created SPARK-17436: Summary: dataframe.write sometimes does not keep sorting Key: SPARK-17436 URL: https://issues.apache.org/jira/browse/SPARK-17436 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-15731) orc writer directory permissions

[ https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15354851#comment-15354851 ] Ran Haim edited comment on SPARK-15731 at 6/29/16 9:14 AM: --- I have tested it

[jira] [Commented] (SPARK-15731) orc writer directory permissions

[ https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15354851#comment-15354851 ] Ran Haim commented on SPARK-15731: -- I have tested it under 1.6.1, and writing the data under the local

[jira] [Commented] (SPARK-15731) orc writer directory permissions

[ https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351026#comment-15351026 ] Ran Haim commented on SPARK-15731: -- Oh, OK - I will try to do this on 1.6 locally and then on 2.x

[jira] [Commented] (SPARK-15731) orc writer directory permissions

[ https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350996#comment-15350996 ] Ran Haim commented on SPARK-15731: -- Yes, I will try it locally. But this might be the problem, I am

[jira] [Commented] (SPARK-15731) orc writer directory permissions

[ https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351003#comment-15351003 ] Ran Haim commented on SPARK-15731: -- wait a minute, you are not addressing any issues on 1.6.1 which is

[jira] [Commented] (SPARK-15731) orc writer directory permissions

[ https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350983#comment-15350983 ] Ran Haim commented on SPARK-15731: -- You did not answer any of my questions, it will help me build a

[jira] [Reopened] (SPARK-15731) orc writer directory permissions

[ https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ran Haim reopened SPARK-15731: -- Hi, MapR upgraded Spark to 1.6.1 and I still get this behavior described. Did you test it on a file

[jira] [Commented] (SPARK-15731) orc writer directory permissions

[ https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315887#comment-15315887 ] Ran Haim commented on SPARK-15731: -- Fine, you can close it - if it works. I will use a work around. >

[jira] [Commented] (SPARK-15731) orc writer directory permissions

[ https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315884#comment-15315884 ] Ran Haim commented on SPARK-15731: -- I am using Mapr distribution and they provide spark 1.5.1 :/ - so

[jira] [Commented] (SPARK-15731) orc writer directory permissions

[ https://issues.apache.org/jira/browse/SPARK-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315782#comment-15315782 ] Ran Haim commented on SPARK-15731: -- Hi Kevin. it is pretty much the same, the only difference I see is

[jira] [Created] (SPARK-15731) orc writer directory permissions

Ran Haim created SPARK-15731: Summary: orc writer directory permissions Key: SPARK-15731 URL: https://issues.apache.org/jira/browse/SPARK-15731 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-10872) Derby error (XSDB6) when creating new HiveContext after restarting SparkContext

[ https://issues.apache.org/jira/browse/SPARK-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15307775#comment-15307775 ] Ran Haim commented on SPARK-10872: -- is there a work around for this? I need this for unit tests. >

[jira] [Comment Edited] (SPARK-15348) Hive ACID

[ https://issues.apache.org/jira/browse/SPARK-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284698#comment-15284698 ] Ran Haim edited comment on SPARK-15348 at 5/16/16 3:09 PM: --- This means that if

[jira] [Commented] (SPARK-15348) Hive ACID

[ https://issues.apache.org/jira/browse/SPARK-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284698#comment-15284698 ] Ran Haim commented on SPARK-15348: -- If I have a transnational table in hive, I cannot use spark job to

[jira] [Closed] (SPARK-15349) Hive ACID

[ https://issues.apache.org/jira/browse/SPARK-15349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ran Haim closed SPARK-15349. Resolution: Duplicate > Hive ACID > - > > Key: SPARK-15349 > URL:

[jira] [Created] (SPARK-15349) Hive ACID

Ran Haim created SPARK-15349: Summary: Hive ACID Key: SPARK-15349 URL: https://issues.apache.org/jira/browse/SPARK-15349 Project: Spark Issue Type: New Feature Reporter: Ran Haim

[jira] [Created] (SPARK-15348) Hive ACID

Ran Haim created SPARK-15348: Summary: Hive ACID Key: SPARK-15348 URL: https://issues.apache.org/jira/browse/SPARK-15348 Project: Spark Issue Type: New Feature Reporter: Ran Haim