newbie does python del, gc.collect() release all resources?

I am new to python. I find my self working with several data frames at the same time. I have run into some driver memory problems and want to make sure I release all resource as soon as possible. 1. should I be calling del and gc.collect() ? 2. If a dataframe was cached do I need to explicitly

Re: performance problem when reading lots of small files created by spark streaming.

Opps sorry , wrong mail list From: Andrew Davidson Date: Wednesday, July 27, 2016 at 7:10 PM To: "users@kafka.apache.org" Subject: performance problem when reading lots of small files created by spark streaming. > I have a relatively

performance problem when reading lots of small files created by spark streaming.

I have a relatively small data set however it is split into many small JSON files. Each file is between maybe 4K and 400K This is probably a very common issue for anyone using spark streaming. My streaming app works fine, how ever my batch application takes several hours to run. All I am doing

newbie: trouble configuring topic default retention (i.e. time to live)

Hi I am new to kafka and need to pick up admin responsibilities for our brokers. I am using kafka_2.11-0.10.0.0. I want to set up my broker so that topics that are created automatically have a SLA of 1 hr. I.E. I want the data to removed after 1 hr. I added the following to my

FW: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Does anyone know if spark plans to upgrade? I think the current version is 0.8x? Kind regards Andy From: Gwen Shapira Reply-To: Date: Tuesday, May 24, 2016 at 9:24 AM To: , ,

Re: newbie: kafka 0.9.0.0 producer does not terminate after producer.close()

Kafka 0.9.0 server supports the previous versions of the clients (0.8.2, > 0.8.1..). > But, new clients won't work properly with the older version of Kafka server. > > You should upgrade your server / broker first. > > --Kamal > > On Fri, May 20, 2016 at 10:58 PM, Andy Davidson

Re: newbie: kafka 0.9.0.0 producer does not terminate after producer.close()

.org> Date: Friday, May 20, 2016 at 7:55 AM To: <users@kafka.apache.org> Subject: Re: newbie: kafka 0.9.0.0 producer does not terminate after producer.close() > You can take a thread dump (using "jstack ") when > the program doesn't terminate and post that output here. That

newbie: kafka 0.9.0.0 producer does not terminate after producer.close()

I wrote a little test client that reads from a file an publishes using the 0.9.0.0 API. I am contacting to an older 0.8.x sever. I am able to send messages how ever I noticed that once I am done reading the input file my test program hangs Any idea what I am doing wrong? Kind regards Andy