Re: ORC 'BETWEEN' Error
Cool, figured you guys were on it but couldn't locate the JIRA for some reason. I'll blame user error :) Thanks for the fix! Nick Sent from my iPhone On Feb 27, 2014, at 6:09 PM, "Prasanth Jayachandran" mailto:pjayachand...@hortonworks.com>> wrote: Hi Martin This is an known issue and its fixed in hive trunk. It should be available in 0.13 release. https://issues.apache.org/jira/browse/HIVE-5601 Thanks Prasanth Jayachandran On Feb 26, 2014, at 8:55 AM, Martin, Nick mailto:nimar...@pssd.com>> wrote: Hi all, (Running Hive 12.0) I have two tables and both are stored as ORC. I attempted to insert via select from tbl1 to tbl2 using ‘BETWEEN’ in my where clause to narrow down some dates. Something like so: “Insert into tbl1 select col1, col2 from tbl2 where col1 between 2 and 4” I kept hitting the error pasted below. So, I switched to a different approach to see if it would work: “Insert into tbl1 select col1,col2 from tbl2 where col1>=2 and col1<=4” Hit the same error. When I just use “where col1>=2” in the where clause the insert will run fine. Is this expected? 2014-02-26 11:22:53,755 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2014-02-26 11:22:53,782 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2014-02-26 11:22:53,902 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2014-02-26 11:22:53,930 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink ganglia started 2014-02-26 11:22:53,975 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2014-02-26 11:22:53,975 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started 2014-02-26 11:22:53,985 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: 2014-02-26 11:22:53,985 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1392147432508_1108, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@249c2715) 2014-02-26 11:22:54,057 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2014-02-26 11:22:54,352 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2014-02-26 11:22:54,363 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2014-02-26 11:22:54,409 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /hdfs/01/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/02/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/03/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/04/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/05/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/06/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/07/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/08/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/09/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/10/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/11/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/12/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108 2014-02-26 11:22:54,481 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2014-02-26 11:22:54,486 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2014-02-26 11:22:54,542 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 2014-02-26 11:22:54,542 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap 2014-02-26 11:22:54,543 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir 2014-02-26 11:22:54,543 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.cache.localFiles is deprecated. Instead, use mapreduce.job.cache.local.files 2014-02-26 11:22:54,543 INFO [main] org.apache.hadoop
Re: ORC 'BETWEEN' Error
Hi Martin This is an known issue and its fixed in hive trunk. It should be available in 0.13 release. https://issues.apache.org/jira/browse/HIVE-5601 Thanks Prasanth Jayachandran On Feb 26, 2014, at 8:55 AM, Martin, Nick wrote: > Hi all, > > (Running Hive 12.0) > > I have two tables and both are stored as ORC. I attempted to insert via > select from tbl1 to tbl2 using ‘BETWEEN’ in my where clause to narrow down > some dates. Something like so: > > “Insert into tbl1 select col1, col2 from tbl2 where col1 between 2 and 4” > > I kept hitting the error pasted below. So, I switched to a different approach > to see if it would work: > > “Insert into tbl1 select col1,col2 from tbl2 where col1>=2 and col1<=4” > > Hit the same error. When I just use “where col1>=2” in the where clause the > insert will run fine. > > Is this expected? > > > > 2014-02-26 11:22:53,755 WARN [main] org.apache.hadoop.conf.Configuration: > job.xml:an attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > 2014-02-26 11:22:53,782 WARN [main] org.apache.hadoop.conf.Configuration: > job.xml:an attempt to override final parameter: > mapreduce.job.end-notification.max.attempts; Ignoring. > 2014-02-26 11:22:53,902 INFO [main] > org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from > hadoop-metrics2.properties > 2014-02-26 11:22:53,930 INFO [main] > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink ganglia started > 2014-02-26 11:22:53,975 INFO [main] > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period > at 10 second(s). > 2014-02-26 11:22:53,975 INFO [main] > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system > started > 2014-02-26 11:22:53,985 INFO [main] org.apache.hadoop.mapred.YarnChild: > Executing with tokens: > 2014-02-26 11:22:53,985 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: > mapreduce.job, Service: job_1392147432508_1108, Ident: > (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@249c2715) > 2014-02-26 11:22:54,057 INFO [main] org.apache.hadoop.mapred.YarnChild: > Sleeping for 0ms before retrying again. Got null now. > 2014-02-26 11:22:54,352 WARN [main] org.apache.hadoop.conf.Configuration: > job.xml:an attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > 2014-02-26 11:22:54,363 WARN [main] org.apache.hadoop.conf.Configuration: > job.xml:an attempt to override final parameter: > mapreduce.job.end-notification.max.attempts; Ignoring. > 2014-02-26 11:22:54,409 INFO [main] org.apache.hadoop.mapred.YarnChild: > mapreduce.cluster.local.dir for child: > /hdfs/01/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/02/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/03/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/04/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/05/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/06/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/07/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/08/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/09/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/10/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/11/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/12/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108 > 2014-02-26 11:22:54,481 WARN [main] org.apache.hadoop.conf.Configuration: > job.xml:an attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > 2014-02-26 11:22:54,486 WARN [main] org.apache.hadoop.conf.Configuration: > job.xml:an attempt to override final parameter: > mapreduce.job.end-notification.max.attempts; Ignoring. > 2014-02-26 11:22:54,542 INFO [main] > org.apache.hadoop.conf.Configuration.deprecation: mapred.task.id is > deprecated. Instead, use mapreduce.task.attempt.id > 2014-02-26 11:22:54,542 INFO [main] > org.apache.hadoop.conf.Configuration.deprecation: mapred.task.is.map is > deprecated. Instead, use mapreduce.task.ismap > 2014-02-26 11:22:54,543 INFO [main] > org.apache.hadoop.conf.Configuration.deprecation: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > 2014-02-26 11:22:54,543 INFO [main] > org.apache.hadoop.conf.Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 2014-02-26 11:22:54,543 INFO [main] > org.apache.hadoop.conf.Configuration.deprecation: mapred.job.id is > deprecated. Instead, use mapreduce.job.id > 2014-02-26 11:22:54,544 INF
ORC 'BETWEEN' Error
Hi all, (Running Hive 12.0) I have two tables and both are stored as ORC. I attempted to insert via select from tbl1 to tbl2 using 'BETWEEN' in my where clause to narrow down some dates. Something like so: "Insert into tbl1 select col1, col2 from tbl2 where col1 between 2 and 4" I kept hitting the error pasted below. So, I switched to a different approach to see if it would work: "Insert into tbl1 select col1,col2 from tbl2 where col1>=2 and col1<=4" Hit the same error. When I just use "where col1>=2" in the where clause the insert will run fine. Is this expected? 2014-02-26 11:22:53,755 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2014-02-26 11:22:53,782 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2014-02-26 11:22:53,902 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2014-02-26 11:22:53,930 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink ganglia started 2014-02-26 11:22:53,975 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2014-02-26 11:22:53,975 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started 2014-02-26 11:22:53,985 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: 2014-02-26 11:22:53,985 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1392147432508_1108, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@249c2715) 2014-02-26 11:22:54,057 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2014-02-26 11:22:54,352 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2014-02-26 11:22:54,363 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2014-02-26 11:22:54,409 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /hdfs/01/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/02/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/03/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/04/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/05/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/06/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/07/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/08/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/09/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/10/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/11/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/12/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108 2014-02-26 11:22:54,481 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2014-02-26 11:22:54,486 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2014-02-26 11:22:54,542 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 2014-02-26 11:22:54,542 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap 2014-02-26 11:22:54,543 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir 2014-02-26 11:22:54,543 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.cache.localFiles is deprecated. Instead, use mapreduce.job.cache.local.files 2014-02-26 11:22:54,543 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id 2014-02-26 11:22:54,544 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id 2014-02-26 11:22:54,544 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition 2014-02-26 11:22:54,545 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: job.local.dir is deprecated.
