[jira] [Commented] (SPARK-12717) pyspark broadcast fails when using multiple threads

[ https://issues.apache.org/jira/browse/SPARK-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326006#comment-16326006 ] Jianfei Wang commented on SPARK-12717: -- [~bryanc] I use pyspark 2.2.0, got the same error. which

[jira] [Commented] (SPARK-18735) Why don't we destroy the broadcast variable after each iteration?

[ https://issues.apache.org/jira/browse/SPARK-18735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724605#comment-15724605 ] Jianfei Wang commented on SPARK-18735: -- oh no,I just see in Kmeans and GaussianMixture they both use

[jira] [Created] (SPARK-18735) Why don't we destroy the broadcast variable after each iteration?

Jianfei Wang created SPARK-18735: Summary: Why don't we destroy the broadcast variable after each iteration? Key: SPARK-18735 URL: https://issues.apache.org/jira/browse/SPARK-18735 Project: Spark

[jira] [Commented] (SPARK-18463) I think it's necessary to have an overrided method of smaple

[ https://issues.apache.org/jira/browse/SPARK-18463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670521#comment-15670521 ] Jianfei Wang commented on SPARK-18463: -- thank you very much ,some misunderstanding about this case

[jira] [Commented] (SPARK-18463) I think it's necessary to have an overrided method of smaple

[ https://issues.apache.org/jira/browse/SPARK-18463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670522#comment-15670522 ] Jianfei Wang commented on SPARK-18463: -- thank you very much ,some misunderstanding about this case

[jira] [Issue Comment Deleted] (SPARK-18463) I think it's necessary to have an overrided method of smaple

[ https://issues.apache.org/jira/browse/SPARK-18463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-18463: - Comment: was deleted (was: ok ,what you mean is rdd1.zip(rdd2).sample() won't use more memory to

[jira] [Commented] (SPARK-18463) I think it's necessary to have an overrided method of smaple

[ https://issues.apache.org/jira/browse/SPARK-18463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670504#comment-15670504 ] Jianfei Wang commented on SPARK-18463: -- ok ,what you mean is rdd1.zip(rdd2).sample() won't use more

[jira] [Closed] (SPARK-18463) I think it's necessary to have an overrided method of smaple

[ https://issues.apache.org/jira/browse/SPARK-18463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang closed SPARK-18463. Resolution: Invalid > I think it's necessary to have an overrided method of smaple >

[jira] [Commented] (SPARK-18463) I think it's necessary to have an overrided method of smaple

[ https://issues.apache.org/jira/browse/SPARK-18463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670222#comment-15670222 ] Jianfei Wang commented on SPARK-18463: -- So, maybe we can imp a sample that sample the two rdds's

[jira] [Created] (SPARK-18463) I think it's necessary to have an overrided method of smaple

Jianfei Wang created SPARK-18463: Summary: I think it's necessary to have an overrided method of smaple Key: SPARK-18463 URL: https://issues.apache.org/jira/browse/SPARK-18463 Project: Spark

[jira] [Comment Edited] (SPARK-17969) I think it's user unfriendly to process standard json file with DataFrame

[ https://issues.apache.org/jira/browse/SPARK-17969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581357#comment-15581357 ] Jianfei Wang edited comment on SPARK-17969 at 10/17/16 6:27 AM: I can do

[jira] [Commented] (SPARK-17969) I think it's user unfriendly to process standard json file with DataFrame

[ https://issues.apache.org/jira/browse/SPARK-17969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581357#comment-15581357 ] Jianfei Wang commented on SPARK-17969: -- I can do this mini job. thank you! > I think it's user

[jira] [Created] (SPARK-17969) I think it's user unfriendly to process standard json file with DataFrame

Jianfei Wang created SPARK-17969: Summary: I think it's user unfriendly to process standard json file with DataFrame Key: SPARK-17969 URL: https://issues.apache.org/jira/browse/SPARK-17969 Project:

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508257#comment-15508257 ] Jianfei Wang commented on SPARK-17562: -- that sounds reasonable, thank you Josh Rosen! > I think a

[jira] [Commented] (SPARK-17579) Exception When the Main object extends Encoder in cluster mode but ok in local mode

[ https://issues.apache.org/jira/browse/SPARK-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501927#comment-15501927 ] Jianfei Wang commented on SPARK-17579: -- {code} 16/09/19 08:49:41 INFO TaskSetManager: Starting task

[jira] [Comment Edited] (SPARK-17579) Exception When the Main object extends Encoder in cluster mode but ok in local mode

[ https://issues.apache.org/jira/browse/SPARK-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501910#comment-15501910 ] Jianfei Wang edited comment on SPARK-17579 at 9/19/16 12:41 AM: Yeah,if i

[jira] [Commented] (SPARK-17579) Exception When the Main object extends Encoder in cluster mode but ok in local mode

[ https://issues.apache.org/jira/browse/SPARK-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501910#comment-15501910 ] Jianfei Wang commented on SPARK-17579: -- Yeah,if i change A[T:Encoder] to A[T],it will work both in

[jira] [Created] (SPARK-17579) Exception When the Main object extends Encoder in cluster mode but ok in local mode

Jianfei Wang created SPARK-17579: Summary: Exception When the Main object extends Encoder in cluster mode but ok in local mode Key: SPARK-17579 URL: https://issues.apache.org/jira/browse/SPARK-17579

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500059#comment-15500059 ] Jianfei Wang commented on SPARK-17562: -- [~joshrosen] please check this. thank you very much! > I

[jira] [Updated] (SPARK-17573) The FileInputStream may be uncloseed when some exceptions occurs

[ https://issues.apache.org/jira/browse/SPARK-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-17573: - Component/s: (was: SQL) Spark Core > The FileInputStream may be uncloseed

[jira] [Commented] (SPARK-17573) The FileInputStream may be uncloseed when some exceptions occurs

[ https://issues.apache.org/jira/browse/SPARK-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498996#comment-15498996 ] Jianfei Wang commented on SPARK-17573: -- fileStream may never be closed when some exceptions

[jira] [Issue Comment Deleted] (SPARK-17573) The FileInputStream may be uncloseed when some exceptions occurs

[ https://issues.apache.org/jira/browse/SPARK-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-17573: - Comment: was deleted (was: if some exceptions happen the fileStream may never be closed in

[jira] [Comment Edited] (SPARK-17573) The FileInputStream may be uncloseed when some exceptions occurs

[ https://issues.apache.org/jira/browse/SPARK-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498958#comment-15498958 ] Jianfei Wang edited comment on SPARK-17573 at 9/17/16 12:50 PM: if some

[jira] [Commented] (SPARK-17573) The FileInputStream may be uncloseed when some exceptions occurs

[ https://issues.apache.org/jira/browse/SPARK-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498958#comment-15498958 ] Jianfei Wang commented on SPARK-17573: -- if some exceptions happen the fileStream may never be closed

[jira] [Updated] (SPARK-17573) The FileInputStream may be uncloseed when some exceptions occurs

[ https://issues.apache.org/jira/browse/SPARK-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-17573: - Description: I think that the InputStream may never be closed when some exceptions occur, we

[jira] [Updated] (SPARK-17573) The FileInputStream may be uncloseed when some exceptions occurs

[ https://issues.apache.org/jira/browse/SPARK-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-17573: - Priority: Trivial (was: Major) > The FileInputStream may be uncloseed when some exceptions

[jira] [Reopened] (SPARK-17573) The FileInputStream may be uncloseed when some exceptions occurs

[ https://issues.apache.org/jira/browse/SPARK-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang reopened SPARK-17573: -- the issue has been changed > The FileInputStream may be uncloseed when some exceptions occurs >

[jira] [Updated] (SPARK-17573) The FileInputStream may be uncloseed when some exceptions occurs

[ https://issues.apache.org/jira/browse/SPARK-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-17573: - Description: I think that the InputStream may never be closed when some exceptions occur, we

[jira] [Updated] (SPARK-17573) The FileInputStream may be uncloseed when some exceptions occurs

[ https://issues.apache.org/jira/browse/SPARK-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-17573: - Summary: The FileInputStream may be uncloseed when some exceptions occurs (was: Why don't we

[jira] [Commented] (SPARK-17573) Why don't we close the input/output Streams

[ https://issues.apache.org/jira/browse/SPARK-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498871#comment-15498871 ] Jianfei Wang commented on SPARK-17573: -- Thank you sir! I've learned much from you,i will be careful

[jira] [Closed] (SPARK-17573) Why don't we close the input/output Streams

[ https://issues.apache.org/jira/browse/SPARK-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang closed SPARK-17573. Resolution: Invalid > Why don't we close the input/output Streams >

[jira] [Updated] (SPARK-17573) Why don't we close the input/output Streams

[ https://issues.apache.org/jira/browse/SPARK-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-17573: - Description: I find that there are many places in spark that we don't close the input/output

[jira] [Updated] (SPARK-17573) Why don't we close the input/output Streams

[ https://issues.apache.org/jira/browse/SPARK-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-17573: - Description: I find that there are many places in spark that we don't close the input/output

[jira] [Updated] (SPARK-17573) Why don't we close the input/output Streams

[ https://issues.apache.org/jira/browse/SPARK-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-17573: - Description: I find that there are many places in spark that we don't close the input/output

[jira] [Created] (SPARK-17573) Why don't we close the input/output Streams

Jianfei Wang created SPARK-17573: Summary: Why don't we close the input/output Streams Key: SPARK-17573 URL: https://issues.apache.org/jira/browse/SPARK-17573 Project: Spark Issue Type:

[jira] [Issue Comment Deleted] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-17562: - Comment: was deleted (was: 中秋快乐！谢谢。) > I think a little code is unnecessary to exist in >

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496324#comment-15496324 ] Jianfei Wang commented on SPARK-17562: -- 中秋快乐！谢谢。 > I think a little code is unnecessary to exist in

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496323#comment-15496323 ] Jianfei Wang commented on SPARK-17562: -- 中秋快乐！谢谢。 > I think a little code is unnecessary to exist in

[jira] [Issue Comment Deleted] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-17562: - Comment: was deleted (was: 中秋快乐！谢谢。) > I think a little code is unnecessary to exist in >

[jira] [Comment Edited] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496283#comment-15496283 ] Jianfei Wang edited comment on SPARK-17562 at 9/16/16 1:03 PM: --- this func

[jira] [Comment Edited] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496283#comment-15496283 ] Jianfei Wang edited comment on SPARK-17562 at 9/16/16 1:03 PM: --- this func

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496283#comment-15496283 ] Jianfei Wang commented on SPARK-17562: -- this func is to revert writes that haven't been committed

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496183#comment-15496183 ] Jianfei Wang commented on SPARK-17562: -- if 0 object is written, we should just set the success flag

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496147#comment-15496147 ] Jianfei Wang commented on SPARK-17562: -- [~cloud_fan] can you check this? thank you! > I think a

[jira] [Updated] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-17562: - Description: In ExternalSorter.spillMemoryIteratorToDisk, I think the code below will never be

[jira] [Created] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

Jianfei Wang created SPARK-17562: Summary: I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk Key: SPARK-17562 URL: https://issues.apache.org/jira/browse/SPARK-17562

[jira] [Commented] (SPARK-17552) Doubt about the double Synchronized in Object SparkSession.getOrCreate()

[ https://issues.apache.org/jira/browse/SPARK-17552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492684#comment-15492684 ] Jianfei Wang commented on SPARK-17552: -- of course not the same one ,but only one thread can get into

[jira] [Created] (SPARK-17552) Doubt about the double Synchronized in Object SparkSession.getOrCreate()

Jianfei Wang created SPARK-17552: Summary: Doubt about the double Synchronized in Object SparkSession.getOrCreate() Key: SPARK-17552 URL: https://issues.apache.org/jira/browse/SPARK-17552 Project: