[ https://issues.apache.org/jira/browse/SPARK-22189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-22189. ------------------------------- Resolution: Invalid Questions should go to the mailing list, please. > Number of jobs created while querying partitioned table in hive using spark > --------------------------------------------------------------------------- > > Key: SPARK-22189 > URL: https://issues.apache.org/jira/browse/SPARK-22189 > Project: Spark > Issue Type: Question > Components: SQL > Affects Versions: 1.6.0 > Reporter: Astha Arya > > I am using Spark SQL > Spark version - 1.6.0 > Hive 1.1.0-cdh5.9.0 > When I run hiveContext.sql, creates 2 another job for my case i.e. 3 jobs in > total for querying hive for a partitioned table. Whereas when i run the same > query on hive using spark as execution engine, it makes only one job. > Also, the driver logs show that it lists all the partitions which most likely > shouldnt happen because it slows down my execution. > Is this a bug? Is there any way to reduce the number of jobs and also not > list all the partitions each time I query the same table ? -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org