I cannot find the base directory in Hive warehouse in HDFS, so I raised this
question. Can you find the base directory in your Hive?
Wantao
At 2014-12-02 15:50:47, unmesha sreeveni unmeshab...@gmail.com wrote:
Check the base directory in Hive warehouse.
On Tue, Dec 2, 2014 at 12:42
hi,i use git checkout a spark branch of hive today,and i think it's the
latest version.and it's pom.xml file include:
commons-dbcp.version1.4/commons-dbcp.version
derby.version10.11.1.1/derby.version
after i built ,a derby-10.11.1.1.jar in
apache-hive-0.15.0-SNAPSHOT-bin/lib ,but when i
hi,i checkout a spark 1.2 branch from spark github,and built,then copy
spark assembly jar into Hive lib directory,but when i run this qeury ,it
still give me this error.
i am very confused,how can i let hive on spark work!
On 2014/12/2 13:39, Xuefu Zhang wrote:
You need to build your spark
Hi Olaf,
Do you have a patch now?
Xiaobo
-- Original --
From: Olaf Flebbe;o.fle...@science-computing.de;
Send time: Monday, Dec 1, 2014 3:51 PM
To: user@hive.apache.orguser@hive.apache.org;
Subject: AW: Build hwi of hive 0.14.0 failed
I filed a Bug
Why hive UPADTE is not reusing the blocks.
the update is not written to same block, why is it so ?
On Tue, Dec 2, 2014 at 10:50 AM, unmesha sreeveni unmeshab...@gmail.com
wrote:
I tried to update my record in hive previous version and also tried out
update in hive 0.14.0. The newer version
The document describes how transaction works and what the data layout is:
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions. See the
Basic design section. HDFS is immutable. Hive creates a delta directory for
every transaction and merges it when read, so it does not written
So that block will not be reused right? If we are updating the entire
block..and at some point we dont need that record...The block will be
wasted right?
They need to relize the blocks for further writes, right?
Am I correct?
On Tue, Dec 2, 2014 at 2:36 PM, vic0777 vic0...@163.com wrote:
The
The compact operation will merge the data and then the blocks may be resued.
At 2014-12-02 17:10:43, unmesha sreeveni unmeshab...@gmail.com wrote:
So that block will not be reused right? If we are updating the entire
block..and at some point we dont need that record...The block will be
Hi Xiaobo,
you can find the pom.xml and update the dependency of
calcite-avatica:jar:0.9.2-incubating-SNAPSHOT to
calcite-avatica:jar:1.0.0-incubating-SNAPSHOT. Then you can build it.
Hi Wentao,
But I didn't find the dependency entry either in the pom.xml for hwi nor the
pom.xml for hive, which one do you mean?
Xiaobo
-- Original --
From: vic0777;vic0...@163.com;
Send time: Tuesday, Dec 2, 2014 5:19 PM
To: useruser@hive.apache.org;
It is in the ql/pom.xml.
At 2014-12-02 17:58:41, guxiaobo1982 guxiaobo1...@qq.com wrote:
Hi Wentao,
But I didn't find the dependency entry either in the pom.xml for hwi nor the
pom.xml for hive, which one do you mean?
Xiaobo
-- Original --
From:
Hi Xiaono,
That dependency is in hive-0.14.0/ql/pom.xml
If I remember correctly the API of calcite is changed so hive does not build
with calcite-1.0.0 . The 0.14 branch uses calcite 0.9.2-incubating.
And there is another broken dependency to tez-0.5.2-SNAPSHOT (But that one is
obvious
hi,XueFu
i checkout a spark branch from sparkgithub(tags:v1.2.0-snapshot0)and i
compare this spark's pom.xml with spark-parent-1.2.0-SNAPSHOT.pom(get
from
http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark_2.10-1.2-SNAPSHOT/org/apache/spark/spark-parent/1.2.0-SNAPSHOT/),and
Hi All,
I am trying to merge gzip compressed RCFile output to one single file per
partition. Hive version is 0.10:
SET hive.exec.compress.intermediate=true;
SET mapred.compress.map.output=true;
SET
mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET
I built Hive0.14 with calcite-1.0.0 successfully a couple of days ago, but I
did not check the code, so I am not sure if its internal logic is affected by
calcite-1.0.0.
Wantao
在 2014-12-02 18:22:33，Olaf Flebbe o.fle...@science-computing.de 写道：
Hi Xiaono,
That dependency is in
I made the chage to ql/pom.xml, but still got the error, maybe there is another
dependence with it.
[xiaobogu@lix3 hwi]$ mvn package
[INFO] Scanning for projects...
[INFO]
[INFO] Using the builder
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder
with a
On a high level we (hive) should really avoid building releases based on
snapshots. That is bad form generally and against the maven model.
On Tue, Dec 2, 2014 at 4:19 AM, vic0777 vic0...@163.com wrote:
Hi Xiaobo,
you can find the pom.xml and update the dependency of
Could you provide details on how to reproduce the issue? such as the exact
spark branch, the command to build Spark, how you build Hive, and what
queries/commands you run.
We are running Hive on Spark all the time. Our pre-commit test runs without
any issue.
Thanks,
Xuefu
On Tue, Dec 2, 2014 at
I am running Hive 0.12 in production, I have a table that ha 1100
partitions, (flat, no multi level partitions) and in those partitions some
have a small number of files (5- 10) and others have quite a few files (up
to 120). The total table size is not huge around 285 GB.
While this is not
Trying again, any pointers is really appreciated !
Thanks
- Atul
From: Atul Paldhikar (apaldhik)
Sent: Monday, December 01, 2014 1:50 PM
To: user@hive.apache.org
Subject: JDBC/beeline client hangs with HiverServer2 (0.13.1)
Hi,
I am trying to connect to Hive via JDBC client (Java code) but the
Hi,
I have a bash script that runs a hive query and I would like it to do
something if the query succeeds and something else if it fails.
My testings show that a query failure does not change Hive's exit code,
what's the right way to achieve this ?
Thanks,
Daniel
By default the sasl auth is enabled, you don't need to set 'auth=noSasl'
unless you configured non-sasl auth.
What errors did you see in the server log initially when noSasl option was
not passed ?
thanks
Prasad
On Tue, Dec 2, 2014 at 10:08 AM, Atul Paldhikar (apaldhik)
apald...@cisco.com
Hi User,
I want to know the difference of query execution time in hive if I use SSD
for HDFS and HDD for HDFS.
Thanks,
Amit
-- Forwarded message --
From: John Omernik j...@omernik.com
Date: Tue, Dec 2, 2014 at 1:58 PM
Subject: Re: Files Per Partition Causing Slowness
To: user@hive.apache.org
Thank you Edward, I knew the number of partitions mattered, but I
didn't think 1000 would be to much.
unsubscribe
-Original Message-
From: John Omernik [mailto:j...@omernik.com]
Sent: Tuesday, December 2, 2014 1:01 PM
To: user@hive.apache.org
Subject: Fwd: Files Per Partition Causing Slowness
-- Forwarded message --
From: John Omernik j...@omernik.com
Date: Tue, Dec 2,
That's not what I've found:
$ hive -e show tables
table1
table2
$ echo $?
0
$ hive -e show partitions notable
FAILED: SemanticException [Error 10001]: Table not found notable
$ echo $?
17
In a bash script:
hive -e show partitions notable
hiveresult=`echo $?`
if [ $hiveresult -ne 0 ]; then
Hi,
I am trying to get Tez sessions enabled with HS2. I start the HiveServer2
instance with the flag -hiveconf hive.execution.engine=tez and then try
to submit multiple queries one after another, as the same user, to the HS2
instance.
When i check the YARN UI, i find that each query of mine is
BCC’ed user@tez.
This question belongs to either the hive user list or the Hortonworks user
forums.
thanks
— Hitesh
On Dec 2, 2014, at 1:28 PM, Pala M Muthaia mchett...@rocketfuelinc.com wrote:
Hi,
I am trying to get Tez sessions enabled with HS2. I start the HiveServer2
instance with
Ok.
Turns out that if i use the same JDBC connection, each successive query
from the same user gets submitted to the same Tez application. If i create
a different JDBC connection for each query, then each query runs in its own
Tez application.
On Tue, Dec 2, 2014 at 2:42 PM, Hitesh Shah
hi,XueFu,thanks a lot for your help,now i will provide more detail to
reproduce this ssue:
1),i checkout a spark branch from hive
github(https://github.com/apache/hive/tree/spark on Nov 29,becasue of
for version now it will give something wrong about:Caused by:
java.lang.RuntimeException:
Hi Team,
I am creating this table:
CREATE TABLE IF NOT EXISTS orctest2 (
id string,
id2 string,
id3 string,
id4 string
)
STORED AS ORC tblproperties
(orc.stripe.size=1048576,orc.row.index.stride=”);
The stripe size is set to 1MB.
After loading data, the table file is about 60MB:
-rwxr-xr-x
i checkout a spark branch of hive today,and built with commad:
mvn clean package -DskipTests -Phadoop-2 -Pdist
after i run commad in hive bin:
/opt/hispark/apache-hive-0.15.0-SNAPSHOT-bin/bin # ./hive --auxpath
/opt/hispark/spark/assembly/target/scala-2.10/spark-assembly-1.2.0-hadoop2.4.0.jar
it
When you build Spark, remove -Phive as well as -Pyarn. When you run hive
queries, you may need to run set spark.home=/path/to/spark/dir;
Thanks,
Xuefu
On Tue, Dec 2, 2014 at 6:29 PM, yuemeng1 yueme...@huawei.com wrote:
hi,XueFu,thanks a lot for your help,now i will provide more detail to
You have to use ORC as the table format (it's the only format that
supports update and delete at this point), and you have to explicitly
mark the table as transactional. You must also bucket the table. For
example:
create table HiveTest (...) clustered by (_col_) into _num_ buckets
stored
Usually this means the metastore database was not properly upgraded.
You can check this by logging into your RDBMS and listing all the
tables. If you see a table named TXNS then the proper tables exist.
Did you install Hive 0.14 fresh or upgrade from a previous version? If
you upgraded did
The base directories will only exist after compaction has run. When you
say you set hive.compactor.initiator.on (=true I hope) and
hive.compactor.worker.threads, did you did that in your metastore
process? If so, did you restart the metastore after changing the config
values?
Alan.
vic0777
Thanks Alan Gates :) . My issue is solved.
http://unmeshasreeveni.blogspot.in/2014/11/updatedeleteinsert-in-hive-0140.html
On Wed, Dec 3, 2014 at 7:41 AM, Alan Gates ga...@hortonworks.com wrote:
You have to use ORC as the table format (it's the only format that
supports update and delete at
Hi Pala,
Can you share the settings you have used for those mentioned in the
document? Are you trying to use the tez session pools? As the document
mentions, you need to disable doAs to ensure usage of the session
pool. Also, the hive server 2 settings need to be in place at the time
of starting
Hi Alan,
Thans for your help. I set the hive.compactor.initiator.on(= true) and
hive.compactor.worker.threads(=2) in hive-site.xml. After the configuration, I
started Hive for its first run. what do you mean by When you say you set
hive.compactor.initiator.on (=true I hope) and
Thanks
Once hive.compactor.initiator.on property is set to true whether the
merge operation takes place and reuse the blocks after each update or do we
need to do Alter statement mensioned
in
41 matches
Mail list logo
