I've filed this as https://issues.apache.org/jira/browse/SPARK-6554
On Thu, Mar 26, 2015 at 6:29 AM, Jon Chase <jon.ch...@gmail.com> wrote:
> Spark 1.3.0, Parquet
>
> I'm having trouble referencing partition columns in my queries.
>
> In the following example, 'probeTypeId' is a partition column. For
> example, the directory structure looks like this:
>
> /mydata
> /probeTypeId=1
> ...files...
> /probeTypeId=2
> ...files...
>
> I see the column when I reference load a DF using the /mydata directory
> and call df.printSchema():
>
> ...
> |-- probeTypeId: integer (nullable = true)
> ...
>
> Parquet is also aware of the column:
> optional int32 probeTypeId;
>
> And this works fine:
>
> sqlContext.sql("select probeTypeId from df limit 1");
>
> ...as does df.show() - it shows the correct values for the partition
> column.
>
>
> However, when I try to use a partition column in a where clause, I get an
> exception stating that the column was not found in the schema:
>
> sqlContext.sql("select probeTypeId from df where probeTypeId = 1 limit 1");
>
>
>
> ...
> ...
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
> in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage
> 0.0 (TID 0, localhost): java.lang.IllegalArgumentException: Column
> [probeTypeId] was not found in schema!
> at parquet.Preconditions.checkArgument(Preconditions.java:47)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.getColumnDescriptor(SchemaCompatibilityValidator.java:172)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:160)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:142)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:76)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:41)
> at parquet.filter2.predicate.Operators$Eq.accept(Operators.java:162)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:46)
> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:41)
> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:22)
> at
> parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:108)
> at
> parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:28)
> at
> parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:158)
> ...
> ...
>
>
>
> What am I doing wrong?
>
>
>
>
>
>
>
> Here's the full stack trace:
>
> using local[*] for master
> 06:05:55,675 |-INFO in
> ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute
> not set
> 06:05:55,683 |-INFO in ch.qos.logback.core.joran.action.AppenderAction -
> About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
> 06:05:55,694 |-INFO in ch.qos.logback.core.joran.action.AppenderAction -
> Naming appender as [STDOUT]
> 06:05:55,721 |-INFO in
> ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default
> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder]
> property
> 06:05:55,768 |-INFO in
> ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of
> ROOT logger to INFO
> 06:05:55,768 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction
> - Attaching appender named [STDOUT] to Logger[ROOT]
> 06:05:55,769 |-INFO in
> ch.qos.logback.classic.joran.action.ConfigurationAction - End of
> configuration.
> 06:05:55,770 |-INFO in
> ch.qos.logback.classic.joran.JoranConfigurator@6aaceffd - Registering
> current configuration as safe fallback point
>
> INFO org.apache.spark.SparkContext Running Spark version 1.3.0
> WARN o.a.hadoop.util.NativeCodeLoader Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> INFO org.apache.spark.SecurityManager Changing view acls to: jon
> INFO org.apache.spark.SecurityManager Changing modify acls to: jon
> INFO org.apache.spark.SecurityManager SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(jon); users
> with modify permissions: Set(jon)
> INFO akka.event.slf4j.Slf4jLogger Slf4jLogger started
> INFO Remoting Starting remoting
> INFO Remoting Remoting started; listening on addresses :[akka.tcp://
> sparkDriver@192.168.1.134:62493]
> INFO org.apache.spark.util.Utils Successfully started service
> 'sparkDriver' on port 62493.
> INFO org.apache.spark.SparkEnv Registering MapOutputTracker
> INFO org.apache.spark.SparkEnv Registering BlockManagerMaster
> INFO o.a.spark.storage.DiskBlockManager Created local directory at
> /var/folders/x7/9hdp8kw9569864088tsl4jmm0000gn/T/spark-150e23b2-ff19-4a51-8cfc-25fb8e1b3f2b/blockmgr-6eea286c-7473-4bda-8886-7250156b68f4
> INFO org.apache.spark.storage.MemoryStore MemoryStore started with
> capacity 1966.1 MB
> INFO org.apache.spark.HttpFileServer HTTP File server directory is
> /var/folders/x7/9hdp8kw9569864088tsl4jmm0000gn/T/spark-cf4687bd-1563-4ddf-b697-21c96fd95561/httpd-6343b9c9-bb66-43ac-ac43-6da80c7a1f95
> INFO org.apache.spark.HttpServer Starting HTTP Server
> INFO o.spark-project.jetty.server.Server jetty-8.y.z-SNAPSHOT
> INFO o.s.jetty.server.AbstractConnector Started
> SocketConnector@0.0.0.0:62494
> INFO org.apache.spark.util.Utils Successfully started service 'HTTP file
> server' on port 62494.
> INFO org.apache.spark.SparkEnv Registering OutputCommitCoordinator
> INFO o.spark-project.jetty.server.Server jetty-8.y.z-SNAPSHOT
> INFO o.s.jetty.server.AbstractConnector Started
> SelectChannelConnector@0.0.0.0:4040
> INFO org.apache.spark.util.Utils Successfully started service 'SparkUI'
> on port 4040.
> INFO org.apache.spark.ui.SparkUI Started SparkUI at
> http://192.168.1.134:4040
> INFO org.apache.spark.executor.Executor Starting executor ID <driver> on
> host localhost
> INFO org.apache.spark.util.AkkaUtils Connecting to HeartbeatReceiver:
> akka.tcp://sparkDriver@192.168.1.134:62493/user/HeartbeatReceiver
> INFO o.a.s.n.n.NettyBlockTransferService Server created on 62495
> INFO o.a.spark.storage.BlockManagerMaster Trying to register BlockManager
> INFO o.a.s.s.BlockManagerMasterActor Registering block manager
> localhost:62495 with 1966.1 MB RAM, BlockManagerId(<driver>, localhost,
> 62495)
> INFO o.a.spark.storage.BlockManagerMaster Registered BlockManager
> INFO o.a.h.conf.Configuration.deprecation mapred.max.split.size is
> deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
> INFO o.a.h.conf.Configuration.deprecation
> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.reduce.speculative
> INFO o.a.h.conf.Configuration.deprecation
> mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use
> mapreduce.job.committer.setup.cleanup.needed
> INFO o.a.h.conf.Configuration.deprecation mapred.min.split.size.per.rack
> is deprecated. Instead, use
> mapreduce.input.fileinputformat.split.minsize.per.rack
> INFO o.a.h.conf.Configuration.deprecation mapred.min.split.size is
> deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
> INFO o.a.h.conf.Configuration.deprecation mapred.min.split.size.per.node
> is deprecated. Instead, use
> mapreduce.input.fileinputformat.split.minsize.per.node
> INFO o.a.h.conf.Configuration.deprecation mapred.reduce.tasks is
> deprecated. Instead, use mapreduce.job.reduces
> INFO o.a.h.conf.Configuration.deprecation mapred.input.dir.recursive is
> deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
> INFO o.a.h.hive.metastore.HiveMetaStore 0: Opening raw store with
> implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
> INFO o.a.h.hive.metastore.ObjectStore ObjectStore, initialize called
> INFO DataNucleus.Persistence Property
> hive.metastore.integral.jdo.pushdown unknown - will be ignored
> INFO DataNucleus.Persistence Property datanucleus.cache.level2 unknown -
> will be ignored
> INFO o.a.h.hive.metastore.ObjectStore Setting MetaStore object pin
> classes with
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> INFO o.a.h.h.metastore.MetaStoreDirectSql MySQL check failed, assuming we
> are not on mysql: Lexical error at line 1, column 5. Encountered: "@"
> (64), after : "".
> INFO DataNucleus.Datastore The class
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
> "embedded-only" so does not have its own datastore table.
> INFO DataNucleus.Datastore The class
> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
> "embedded-only" so does not have its own datastore table.
> INFO DataNucleus.Datastore The class
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
> "embedded-only" so does not have its own datastore table.
> INFO DataNucleus.Datastore The class
> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
> "embedded-only" so does not have its own datastore table.
> INFO DataNucleus.Query Reading in results for query
> "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used
> is closing
> INFO o.a.h.hive.metastore.ObjectStore Initialized ObjectStore
> INFO o.a.h.hive.metastore.HiveMetaStore Added admin role in metastore
> INFO o.a.h.hive.metastore.HiveMetaStore Added public role in metastore
> INFO o.a.h.hive.metastore.HiveMetaStore No user is added in admin role,
> since config is empty
> INFO o.a.h.hive.ql.session.SessionState No Tez session required at this
> point. hive.execution.engine=mr.
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> root
> |-- clientMarketId: integer (nullable = true)
> |-- clientCountryId: integer (nullable = true)
> |-- clientRegionId: integer (nullable = true)
> |-- clientStateId: integer (nullable = true)
> |-- clientAsnId: integer (nullable = true)
> |-- reporterZoneId: integer (nullable = true)
> |-- reporterCustomerId: integer (nullable = true)
> |-- responseCode: integer (nullable = true)
> |-- measurementValue: integer (nullable = true)
> |-- year: integer (nullable = true)
> |-- month: integer (nullable = true)
> |-- day: integer (nullable = true)
> |-- providerOwnerZoneId: integer (nullable = true)
> |-- providerOwnerCustomerId: integer (nullable = true)
> |-- providerId: integer (nullable = true)
> |-- probeTypeId: integer (nullable = true)
>
> ======================================================
> INFO hive.ql.parse.ParseDriver Parsing command: select probeTypeId from
> df where probeTypeId = 1 limit 1
> INFO hive.ql.parse.ParseDriver Parse Completed
> ==== results for select probeTypeId from df where probeTypeId = 1 limit 1
> ======================================================
> INFO o.a.s.sql.parquet.ParquetRelation2 Reading 33.33333333333333% of
> partitions
> INFO org.apache.spark.storage.MemoryStore ensureFreeSpace(191336) called
> with curMem=0, maxMem=2061647216
> INFO org.apache.spark.storage.MemoryStore Block broadcast_0 stored as
> values in memory (estimated size 186.9 KB, free 1966.0 MB)
> INFO org.apache.spark.storage.MemoryStore ensureFreeSpace(27530) called
> with curMem=191336, maxMem=2061647216
> INFO org.apache.spark.storage.MemoryStore Block broadcast_0_piece0 stored
> as bytes in memory (estimated size 26.9 KB, free 1965.9 MB)
> INFO o.a.spark.storage.BlockManagerInfo Added broadcast_0_piece0 in
> memory on localhost:62495 (size: 26.9 KB, free: 1966.1 MB)
> INFO o.a.spark.storage.BlockManagerMaster Updated info of block
> broadcast_0_piece0
> INFO org.apache.spark.SparkContext Created broadcast 0 from NewHadoopRDD
> at newParquet.scala:447
> INFO o.a.h.conf.Configuration.deprecation mapred.max.split.size is
> deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
> INFO o.a.h.conf.Configuration.deprecation mapred.min.split.size is
> deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
> INFO o.a.s.s.p.ParquetRelation2$$anon$1$$anon$2 Using Task Side Metadata
> Split Strategy
> INFO org.apache.spark.SparkContext Starting job: runJob at
> SparkPlan.scala:121
> INFO o.a.spark.scheduler.DAGScheduler Got job 0 (runJob at
> SparkPlan.scala:121) with 1 output partitions (allowLocal=false)
> INFO o.a.spark.scheduler.DAGScheduler Final stage: Stage 0(runJob at
> SparkPlan.scala:121)
> INFO o.a.spark.scheduler.DAGScheduler Parents of final stage: List()
> INFO o.a.spark.scheduler.DAGScheduler Missing parents: List()
> INFO o.a.spark.scheduler.DAGScheduler Submitting Stage 0
> (MapPartitionsRDD[3] at map at SparkPlan.scala:96), which has no missing
> parents
> INFO org.apache.spark.storage.MemoryStore ensureFreeSpace(5512) called
> with curMem=218866, maxMem=2061647216
> INFO org.apache.spark.storage.MemoryStore Block broadcast_1 stored as
> values in memory (estimated size 5.4 KB, free 1965.9 MB)
> INFO org.apache.spark.storage.MemoryStore ensureFreeSpace(3754) called
> with curMem=224378, maxMem=2061647216
> INFO org.apache.spark.storage.MemoryStore Block broadcast_1_piece0 stored
> as bytes in memory (estimated size 3.7 KB, free 1965.9 MB)
> INFO o.a.spark.storage.BlockManagerInfo Added broadcast_1_piece0 in
> memory on localhost:62495 (size: 3.7 KB, free: 1966.1 MB)
> INFO o.a.spark.storage.BlockManagerMaster Updated info of block
> broadcast_1_piece0
> INFO org.apache.spark.SparkContext Created broadcast 1 from broadcast at
> DAGScheduler.scala:839
> INFO o.a.spark.scheduler.DAGScheduler Submitting 1 missing tasks from
> Stage 0 (MapPartitionsRDD[3] at map at SparkPlan.scala:96)
> INFO o.a.s.scheduler.TaskSchedulerImpl Adding task set 0.0 with 1 tasks
> INFO o.a.spark.scheduler.TaskSetManager Starting task 0.0 in stage 0.0
> (TID 0, localhost, PROCESS_LOCAL, 1687 bytes)
> INFO org.apache.spark.executor.Executor Running task 0.0 in stage 0.0
> (TID 0)
> INFO o.a.s.s.p.ParquetRelation2$$anon$1 Input split:
> ParquetInputSplit{part:
> file:/Users/jon/Downloads/sparksql/1partitionsminusgeo/year=2015/month=1/day=14/providerOwnerZoneId=0/providerOwnerCustomerId=0/providerId=287/probeTypeId=1/part-r-00001.parquet
> start: 0 end: 8851183 length: 8851183 hosts: [] requestedSchema: message
> root {
> optional int32 probeTypeId;
> }
> readSupportMetadata:
> {org.apache.spark.sql.parquet.row.requested_schema={"type":"struct","fields":[{"name":"probeTypeId","type":"integer","nullable":true,"metadata":{}}]},
> org.apache.spark.sql.parquet.row.metadata={"type":"struct","fields":[{"name":"clientMarketId","type":"integer","nullable":true,"metadata":{}},{"name":"clientCountryId","type":"integer","nullable":true,"metadata":{}},{"name":"clientRegionId","type":"integer","nullable":true,"metadata":{}},{"name":"clientStateId","type":"integer","nullable":true,"metadata":{}},{"name":"clientAsnId","type":"integer","nullable":true,"metadata":{}},{"name":"reporterZoneId","type":"integer","nullable":true,"metadata":{}},{"name":"reporterCustomerId","type":"integer","nullable":true,"metadata":{}},{"name":"responseCode","type":"integer","nullable":true,"metadata":{}},{"name":"measurementValue","type":"integer","nullable":true,"metadata":{}},{"name":"year","type":"integer","nullable":true,"metadata":{}},{"name":"month","type":"integer","nullable":true,"metadata":{}},{"name":"day","type":"integer","nullable":true,"metadata":{}},{"name":"providerOwnerZoneId","type":"integer","nullable":true,"metadata":{}},{"name":"providerOwnerCustomerId","type":"integer","nullable":true,"metadata":{}},{"name":"providerId","type":"integer","nullable":true,"metadata":{}},{"name":"probeTypeId","type":"integer","nullable":true,"metadata":{}}]}}}
> ERROR org.apache.spark.executor.Executor Exception in task 0.0 in stage
> 0.0 (TID 0)
> java.lang.IllegalArgumentException: Column [probeTypeId] was not found in
> schema!
> at parquet.Preconditions.checkArgument(Preconditions.java:47)
> ~[parquet-common-1.6.0rc3.jar:na]
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.getColumnDescriptor(SchemaCompatibilityValidator.java:172)
> ~[parquet-column-1.6.0rc3.jar:na]
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:160)
> ~[parquet-column-1.6.0rc3.jar:na]
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:142)
> ~[parquet-column-1.6.0rc3.jar:na]
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:76)
> ~[parquet-column-1.6.0rc3.jar:na]
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:41)
> ~[parquet-column-1.6.0rc3.jar:na]
> at parquet.filter2.predicate.Operators$Eq.accept(Operators.java:162)
> ~[parquet-column-1.6.0rc3.jar:na]
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:46)
> ~[parquet-column-1.6.0rc3.jar:na]
> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:41)
> ~[parquet-hadoop-1.6.0rc3.jar:na]
> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:22)
> ~[parquet-hadoop-1.6.0rc3.jar:na]
> at
> parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:108)
> ~[parquet-column-1.6.0rc3.jar:na]
> at
> parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:28)
> ~[parquet-hadoop-1.6.0rc3.jar:na]
> at
> parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:158)
> ~[parquet-hadoop-1.6.0rc3.jar:na]
> at
> parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:138)
> ~[parquet-hadoop-1.6.0rc3.jar:na]
> at
> org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at
> org.apache.spark.rdd.NewHadoopRDD$NewHadoopMapPartitionsWithSplitRDD.compute(NewHadoopRDD.scala:244)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> ~[spark-core_2.10-1.3.0.jar:1.3.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_31]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_31]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]
> WARN o.a.spark.scheduler.TaskSetManager Lost task 0.0 in stage 0.0 (TID
> 0, localhost): java.lang.IllegalArgumentException: Column [probeTypeId] was
> not found in schema!
> at parquet.Preconditions.checkArgument(Preconditions.java:47)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.getColumnDescriptor(SchemaCompatibilityValidator.java:172)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:160)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:142)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:76)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:41)
> at parquet.filter2.predicate.Operators$Eq.accept(Operators.java:162)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:46)
> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:41)
> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:22)
> at
> parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:108)
> at
> parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:28)
> at
> parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:158)
> at
> parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:138)
> at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133)
> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104)
> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at
> org.apache.spark.rdd.NewHadoopRDD$NewHadoopMapPartitionsWithSplitRDD.compute(NewHadoopRDD.scala:244)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> ERROR o.a.spark.scheduler.TaskSetManager Task 0 in stage 0.0 failed 1
> times; aborting job
> INFO o.a.s.scheduler.TaskSchedulerImpl Removed TaskSet 0.0, whose tasks
> have all completed, from pool
> INFO o.a.s.scheduler.TaskSchedulerImpl Cancelling stage 0
> INFO o.a.spark.scheduler.DAGScheduler Job 0 failed: runJob at
> SparkPlan.scala:121, took 0.132538 s
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/metrics/json,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/static,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/executors/threadDump,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/executors/json,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/executors,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/environment/json,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/environment,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/storage/rdd,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/storage/json,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/storage,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/stages/pool/json,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/stages/pool,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/stages/stage/json,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/stages/stage,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/stages/json,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/stages,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/jobs/job/json,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/jobs/job,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/jobs/json,null}
> INFO o.s.j.server.handler.ContextHandler stopped
> o.s.j.s.ServletContextHandler{/jobs,null}
> INFO org.apache.spark.ui.SparkUI Stopped Spark web UI at
> http://192.168.1.134:4040
> INFO o.a.spark.scheduler.DAGScheduler Stopping DAGScheduler
> INFO o.a.s.MapOutputTrackerMasterActor MapOutputTrackerActor stopped!
> INFO org.apache.spark.storage.MemoryStore MemoryStore cleared
> INFO o.apache.spark.storage.BlockManager BlockManager stopped
> INFO o.a.spark.storage.BlockManagerMaster BlockManagerMaster stopped
> INFO o.a.s.s.OutputCommitCoordinator$OutputCommitCoordinatorActor
> OutputCommitCoordinator stopped!
> INFO org.apache.spark.SparkContext Successfully stopped SparkContext
>
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
> in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage
> 0.0 (TID 0, localhost): java.lang.IllegalArgumentException: Column
> [probeTypeId] was not found in schema!
> at parquet.Preconditions.checkArgument(Preconditions.java:47)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.getColumnDescriptor(SchemaCompatibilityValidator.java:172)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:160)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:142)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:76)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:41)
> at parquet.filter2.predicate.Operators$Eq.accept(Operators.java:162)
> at
> parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:46)
> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:41)
> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:22)
> at
> parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:108)
> at
> parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:28)
> at
> parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:158)
> at
> parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:138)
> at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133)
> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104)
> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at
> org.apache.spark.rdd.NewHadoopRDD$NewHadoopMapPartitionsWithSplitRDD.compute(NewHadoopRDD.scala:244)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> Driver stacktrace:
> at org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
> at scala.Option.foreach(Option.scala:236)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
>
> Mar 26, 2015 6:06:02 AM INFO: parquet.filter2.compat.FilterCompat:
> Filtering using predicate: eq(probeTypeId, 1)
> Mar 26, 2015 6:06:02 AM WARNING: parquet.hadoop.ParquetRecordReader: Can
> not initialize counter due to context is not a instance of
> TaskInputOutputContext, but is
> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
> Mar 26, 2015 6:06:02 AM INFO: parquet.filter2.compat.FilterCompat:
> Filtering using predicate: eq(probeTypeId, 1)
>
> Process finished with exit code 255
>
>
