My apologies to the list. I replied to Manu's question and it went directly to him rather than the list.
In case anyone else has this issue here is my reply and Manu's reply to me. This also answers Ian's question. --------------------------------------- Hi Manu, The dataset is 7.5 million rows and 500 columns. In parquet form it's about 1.1 Gig. It was created with Spark and copied up to s3. It has about 4600 parts (which I'd also like to gain some control over). I can try a smaller dataset, however it works when I run it locally, even with the file out on s3. It just takes a while. I can try copying it to HDFS first but that wont help longer term. Thanks Jim ----------------------------------------- Manu's response: ----------------------------------------- I am pretty sure it is due to the number of parts you have.. I have a parquet data set that is 250M rows and 924 columns and it is ~2500 files... I recommend creating a tables in HIve with that data set and doing an insert overwrite so you can get a data set with more manageable files.. Why I think its the number of files is that I believe that a all of those or large part of those files are read when you run sqlContext.parquetFile() and the time it would take in s3 for that to happen is a lot so something internally is timing out.. -Manu -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Querying-a-parquet-file-in-s3-with-an-ec2-install-tp13737p13790.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
