Re: Weird Error on Inserting in Table [ORC, MESOS, HIVE]
hi
I faced a similar situation in my dev cluster CDH distribution 5.1.3
See the thread details with log files
https://groups.google.com/a/cloudera.org/forum/#!mydiscussions/scm-users/MpcpHj5mWT8
thanks
sanjay From: John Omernik j...@omernik.com
To: user@hive.apache.org
Sent: Tuesday, September 9, 2014 12:10 PM
Subject: Re: Weird Error on Inserting in Table [ORC, MESOS, HIVE]
Well, here is me talking to myself: but in case someone else runs across this,
I changed the hive metastore connect timeout to 600 seconds (per the JIRA below
for Hive 0.14) and now my problem has gone away. It looks like the timeout was
causing some craziness.
https://issues.apache.org/jira/browse/HIVE-7140
On Tue, Sep 9, 2014 at 1:00 PM, John Omernik j...@omernik.com wrote:
I ran with debug logging, and this is interesting, there was a loss of
connection to the metastore client RIGHT before the partition mention above...
as data was looking to be moved around... I wonder if the timing on that is bad?
14/09/09 12:47:37 [main]: INFO exec.MoveTask: Partition is: {day=null,
source=null}14/09/09 12:47:38 [main]: INFO metadata.Hive: Renaming
src:maprfs:/user/hive/scratch/hive-mapr/hive_2014-09-09_12-38-30_860_3555291990145206535-1/-ext-1/day=2012-11-30/source=20121119_SWAirlines_Spam/04_0;dest:
maprfs:/user/hive/warehouse/intel_flow.db/pcaps/day=2012-11-30/source=20121119_SWAirlines_Spam/04_0;Status:true14/09/09
12:48:02 [main]: WARN metastore.RetryingMetaStoreClient: MetaStoreClient lost
connection. Attempting to
reconnect.org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
On Tue, Sep 9, 2014 at 11:02 AM, John Omernik j...@omernik.com wrote:
I am doing a dynamic partition load in Hive 0.13 using ORC files. This has
always worked in the past both with MapReduce V1 and YARN. I am working with
Mesos now, and trying to trouble shoot this weird error:
Failed with exception AlreadyExistsException(message:Partition already exists
What's odd is is my insert is an insert (without Overwrite) so it's like two
different reducers have data to go into the same partition, but then there is a
collision of some sort? Perhaps there is a situation where the partition
doesn't exist prior to the run, but when two reducers have data, they both
think they should be the one to create the partition? Shouldn't if a partition
already exists, the reducer just copies it's file into the partition? I am
struggling to see why this would be an issue with Mesos, but not on Yarn, or
MRv1.
Any thoughts would be welcome.
John
Weird Error on Inserting in Table [ORC, MESOS, HIVE]
I am doing a dynamic partition load in Hive 0.13 using ORC files. This has always worked in the past both with MapReduce V1 and YARN. I am working with Mesos now, and trying to trouble shoot this weird error: Failed with exception AlreadyExistsException(message:Partition already exists What's odd is is my insert is an insert (without Overwrite) so it's like two different reducers have data to go into the same partition, but then there is a collision of some sort? Perhaps there is a situation where the partition doesn't exist prior to the run, but when two reducers have data, they both think they should be the one to create the partition? Shouldn't if a partition already exists, the reducer just copies it's file into the partition? I am struggling to see why this would be an issue with Mesos, but not on Yarn, or MRv1. Any thoughts would be welcome. John
Re: Weird Error on Inserting in Table [ORC, MESOS, HIVE]
I ran with debug logging, and this is interesting, there was a loss of
connection to the metastore client RIGHT before the partition mention
above... as data was looking to be moved around... I wonder if the timing
on that is bad?
14/09/09 12:47:37 [main]: INFO exec.MoveTask: Partition is: {day=null,
source=null}
14/09/09 12:47:38 [main]: INFO metadata.Hive: Renaming
src:maprfs:/user/hive/scratch/hive-mapr/hive_2014-09-09_12-38-30_860_3555291990145206535-1/-ext-1/day=2012-11-30/source=20121119_SWAirlines_Spam/04_0;dest:
maprfs:/user/hive/warehouse/intel_flow.db/pcaps/day=2012-11-30/source=20121119_SWAirlines_Spam/04_0;Status:true
14/09/09 12:48:02 [main]: WARN metastore.RetryingMetaStoreClient:
MetaStoreClient lost connection. Attempting to reconnect.
org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
On Tue, Sep 9, 2014 at 11:02 AM, John Omernik j...@omernik.com wrote:
I am doing a dynamic partition load in Hive 0.13 using ORC files. This has
always worked in the past both with MapReduce V1 and YARN. I am working
with Mesos now, and trying to trouble shoot this weird error:
Failed with exception AlreadyExistsException(message:Partition already
exists
What's odd is is my insert is an insert (without Overwrite) so it's like
two different reducers have data to go into the same partition, but then
there is a collision of some sort? Perhaps there is a situation where the
partition doesn't exist prior to the run, but when two reducers have data,
they both think they should be the one to create the partition? Shouldn't
if a partition already exists, the reducer just copies it's file into the
partition? I am struggling to see why this would be an issue with Mesos,
but not on Yarn, or MRv1.
Any thoughts would be welcome.
John
Re: Weird Error on Inserting in Table [ORC, MESOS, HIVE]
Well, here is me talking to myself: but in case someone else runs across
this, I changed the hive metastore connect timeout to 600 seconds (per the
JIRA below for Hive 0.14) and now my problem has gone away. It looks like
the timeout was causing some craziness.
https://issues.apache.org/jira/browse/HIVE-7140
On Tue, Sep 9, 2014 at 1:00 PM, John Omernik j...@omernik.com wrote:
I ran with debug logging, and this is interesting, there was a loss of
connection to the metastore client RIGHT before the partition mention
above... as data was looking to be moved around... I wonder if the timing
on that is bad?
14/09/09 12:47:37 [main]: INFO exec.MoveTask: Partition is: {day=null,
source=null}
14/09/09 12:47:38 [main]: INFO metadata.Hive: Renaming
src:maprfs:/user/hive/scratch/hive-mapr/hive_2014-09-09_12-38-30_860_3555291990145206535-1/-ext-1/day=2012-11-30/source=20121119_SWAirlines_Spam/04_0;dest:
maprfs:/user/hive/warehouse/intel_flow.db/pcaps/day=2012-11-30/source=20121119_SWAirlines_Spam/04_0;Status:true
14/09/09 12:48:02 [main]: WARN metastore.RetryingMetaStoreClient:
MetaStoreClient lost connection. Attempting to reconnect.
org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
On Tue, Sep 9, 2014 at 11:02 AM, John Omernik j...@omernik.com wrote:
I am doing a dynamic partition load in Hive 0.13 using ORC files. This
has always worked in the past both with MapReduce V1 and YARN. I am working
with Mesos now, and trying to trouble shoot this weird error:
Failed with exception AlreadyExistsException(message:Partition already
exists
What's odd is is my insert is an insert (without Overwrite) so it's like
two different reducers have data to go into the same partition, but then
there is a collision of some sort? Perhaps there is a situation where the
partition doesn't exist prior to the run, but when two reducers have data,
they both think they should be the one to create the partition? Shouldn't
if a partition already exists, the reducer just copies it's file into the
partition? I am struggling to see why this would be an issue with Mesos,
but not on Yarn, or MRv1.
Any thoughts would be welcome.
John
