But tachyon is inherently unreliable with an LRU eviction policy which removes
blocks from Tachyon (i.e final tier eviction).
I found tachyon change file status to pinned. When a file is pinned, its blocks
will not be evicted.
r7raul1...@163.com
From: Gopal Vijayaraghavan
Date: 2015-10-28
Yes
r7raul1...@163.com
From: Bikas Saha
Date: 2015-11-02 14:54
To: user@tez.apache.org
Subject: RE: RE: Allow session-level temp-tables store in tachyon?
So you are planning to run Tachyon on a separate host on which jobs do not run?
From: r7raul1...@163.com [mailto:r7raul1...@163.com
HDFS in-memory storage need this host with datanode installed. I worry about
HDFS in-memory will cousume the memory on that host which will cause job fail.
r7raul1...@163.com
From: Bikas Saha
Date: 2015-10-28 11:00
To: user@tez.apache.org
Subject: RE: Allow session-level temp-tables store in
I have a idea to put temp table in tachyon to speed query.
I found the jira which is similar to my idea.
https://issues.apache.org/jira/browse/HIVE-7313
But how to put temp-tables into tachyon ? Any advice ?
r7raul1...@163.com
th or either one of them to ensure "hive.tez.java.opts" is
smaller then "hive.tez.container.size" or
"mapreduce.{map|reduce}.java.opts" is smaller then
"mapreduce.{map|reduce}.memory.mb".
r7raul1...@163.com
From: Juho Autio
Date: 2015-10-26 18:46
To: user
S
Try hive hook
http://stackoverflow.com/questions/17461932/hive-execution-hook
r7raul1...@163.com
From: Raajay
Date: 2015-09-02 05:12
To: user
Subject: DAG serialization
Hi,
Currently, I am running Hive on Tez. I wish to make some changes to the DAGs
generated by Hive before running it
Thank you!
r7raul1...@163.com
From: Rajesh Balamohan
Date: 2015-08-25 17:04
To: user
Subject: Re: How to profile tez task?
You can use "tez.task-specific.launch.cmd-opts.list" and
"tez.task-specific.launch.cmd-opts" to specify which tasks in which vertices
shou
/libjprofilerti.so=port=8849,nowait;
In tez how to enable profile?
r7raul1...@163.com
oint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
15/07/24 10:31:55 DEBUG mortbay.log: EOF
15/07/24 10:31:55 DEBUG mortbay.log: EXCEPTION
r7raul1...@163.com
From: Hitesh Shah
Date: 2015-07-22 23:12
To: user
Subject: Re: tez use timeline server get exception
Hello
cope "Singleton"
15/07/23 10:30:49 INFO timeline.LeveldbTimelineStore: Discarded 0 entities for
timestamp 1437013849959 and earlier in 0.001 seconds
15/07/23 10:35:49 INFO timeline.LeveldbTimelineStore: Discarded 0 entities for
timestamp 1437014149960 and earlier in 0.001 seconds
r7raul1...@163.com
I fire a jira for hadoop common.
https://issues.apache.org/jira/browse/HADOOP-12253
r7raul1...@163.com
From: Hitesh Shah
Date: 2015-07-22 13:32
To: user
Subject: Re: tez throw error when enable hdfs federation
Thanks for digging into the issue.
Would you mind filing a Hadoop jira for this
, creationTime, creationTime,
PERMISSION_555, ugi.getUserName(), ugi.getGroupNames()[0],
new Path(theInternalDir.fullPath).makeQualified(
myUri, ROOT_PATH));
}
Then I create user tangjijun on hiveserver2 node. The exception gone.
From: r7raul1...@163.com
Date: 2015-07-21 14:29
To: user
Subject: Re
Error, return code 1
from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1)
r7raul1...@163.com
From: r7raul1...@163.com
Date: 2015-07-21 08:33
To: user
Subject: Re: Re: tez throw error when enable hdfs federation
Thanks
r7raul1...@163.com
From: Rajesh Balamohan
Date: 2015-07-20
Thanks
r7raul1...@163.com
From: Rajesh Balamohan
Date: 2015-07-20 18:59
To: user
Subject: Re: tez throw error when enable hdfs federation
https://issues.apache.org/jira/browse/TEZ-2600
~Rajesh.B
On Mon, Jul 20, 2015 at 2:45 PM, jjtang wrote:
INFO : Tez session hasn't been create
log like this hdfs-audit.log.9 ?
r7raul1...@163.com
From: Hitesh Shah
Date: 2015-06-19 02:28
To: user
Subject: Re: hive 1.1.0 on tez0.53 error
Also, if you have access to the name node audit logs, can you search for all
accesses of
"/tmp/hive/lujian/_tez_session_dir/86bc0010-4816-4251
maybe first successfule job delete tez-conf.pb ?
r7raul1...@163.com
From: Hitesh Shah
Date: 2015-06-18 10:46
To: user
Subject: Re: hive 1.1.0 on tez0.53 error
That particular log is a red herring and not really an issue that is causing
the failure.
The main problem based on the log is this
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3002)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2982)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:626)
More detail log see my attach.
r7raul1...@163.com
categ.categ_lvl1_id,
categ.categ_lvl1_name
Tez UI can pick out some sql (dag graph) when user search for
TESTTMP.cust_sku_vis_detl1.
r7raul1...@163.com
From: Jeff Zhang
Date: 2015-06-12 08:38
To: user
Subject: Re: some advice to tez ui
This might need hive add some app specific info on the dag/vertex.
I think tez ui can add some feature like data governance capabilities such as
- which source table generate the result table,provide lineage search for dag.
r7raul1...@163.com
sorry, the command is
yarn logs -applicationId
r7raul1...@163.com
From: r7raul1...@163.com
Date: 2015-06-10 17:06
To: user
Subject: Re: Re: hive on tez error
check log
yarn -logs applicationId
r7raul1...@163.com
From: Sateesh Karuturi
Date: 2015-06-10 17:10
To: user
Subject: Re: Re
check log
yarn -logs applicationId
r7raul1...@163.com
From: Sateesh Karuturi
Date: 2015-06-10 17:10
To: user
Subject: Re: Re: hive on tez error
tez.lib.uris
${fs.defaultFS}/tez,${fs.defaultFS}/tez/lib
On Wed, Jun 10, 2015 at 2:02 PM, r7raul1...@163.com wrote:
show me
show me tez-site.xml?
r7raul1...@163.com
From: Sateesh Karuturi
Date: 2015-06-10 16:50
To: user
Subject: Re: hive on tez error
tez 0.5.2
On Wed, Jun 10, 2015 at 1:49 PM, r7raul1...@163.com wrote:
tez version??
r7raul1...@163.com
From: Sateesh Karuturi
Date: 2015-06-10 16:09
To: user
tez version??
r7raul1...@163.com
From: Sateesh Karuturi
Date: 2015-06-10 16:09
To: user
Subject: hive on tez error
whenever iam perform a simple insert operation on hive
1.0.0(hive.execution.engine=tez) its shows following error:
FAILED: Execution Error, return code 1 from
I replace the jackson jars by 1.9.13 in tez-0.7.0.tar.gz . Now TEZ UI can
capture log from timeline server.
r7raul1...@163.com
From: Hitesh Shah
Date: 2015-06-05 09:55
To: user
Subject: Re: hive 1.1.0+tez0.7+hadoop2.5.0-cdh5.2.0 when use TEZ UI throw
exception
There might be a better
/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hadoop-yarn/lib/jackson-xc-1.8.8.jar
r7raul1...@163.com
From: Prakash Ramachandran
Date: 2015-06-04 17:21
To: user
Subject: Re: hive 1.1.0+tez0.7+hadoop2.5.0-cdh5.2.0 when use TEZ UI throw
exception
I have faced this issue when there was a
tez-plugins/pom.xml.
r7raul1...@163.com
eeded: 1 Running: 0 Failed: 0 Killed: 0
15/06/04 15:45:41 INFO client.DAGClientImpl: DAG completed.
FinalState=SUCCEEDED
r7raul1...@163.com
From: r7raul1...@163.com
Date: 2015-06-04 15:13
To: user
Subject: Re: Re: hive 1.1.0 tez0.7 hadoop 2.5.0 run query NoClassDefFoundError
Thank you!
r7
Thank you!
r7raul1...@163.com
From: Jianfeng (Jeff) Zhang
Date: 2015-06-04 14:41
To: user
Subject: Re: hive 1.1.0 tez0.7 hadoop 2.5.0 run query NoClassDefFoundError
Maybe this blog is helpful for you
http://www.swiss-scalability.com/2015/05/hive-on-tez-on-cdh-love-everywhere.html
Best
run using
/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/jars/hadoop-common-2.5.0-cdh5.2.0.jar
r7raul1...@163.com
From: Jianfeng (Jeff) Zhang
Date: 2015-06-04 14:00
To: user
Subject: Re: hive 1.1.0 tez0.7 hadoop 2.5.0 run query NoClassDefFoundError
I believe you are using CDH. I remember
(DAGAppMaster.java:2220)
at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2038)
r7raul1...@163.com
From: r7raul1...@163.com
Date: 2015-06-04 13:19
To: user
Subject: hive 1.1.0 tez0.7 hadoop 2.5.0 run query NoClassDefFoundError
hive> set hive.execution.engine=tez;
hive&g
-101 from
org.apache.hadoop.hive.ql.exec.tez.TezTask.
org/apache/hadoop/yarn/api/records/timeline/TimelineDomain
hive>
r7raul1...@163.com
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the
command
[ERROR] mvn -rf :tez-ui
r7raul1...@163.com
Thank you! I wiil try hive 1.1.0 on tez 0.7.
r7raul1...@163.com
From: Rajesh Balamohan
Date: 2015-06-04 09:14
To: user
Subject: Re: Re: What is the difference between PipelinedSorter and
DefaultSorter?
Support for > 2GB & PipelinedShuffle got added in 0.7.x. If you are using
0.5.
ppers end up generating huge amount of dataset.
~Rajesh.B
On Wed, Jun 3, 2015 at 7:52 AM, r7raul1...@163.com wrote:
Thank you!
r7raul1...@163.com
From: Rajesh Balamohan
Date: 2015-06-03 10:43
To: user
Subject: Re: What is the difference between PipelinedSorter and DefaultSorter?
DefaultSorter is the
Maybe meet https://issues.apache.org/jira/browse/TEZ-704
r7raul1...@163.com
From: r7raul1...@163.com
Date: 2015-06-03 17:26
To: user
Subject: Re: How to debug hanging hive on tez job
What is the value of mapreduce.reduce.cpu.vcores?
r7raul1...@163.com
From: Thomas Larsson
Date: 2015-06
What is the value of mapreduce.reduce.cpu.vcores?
r7raul1...@163.com
From: Thomas Larsson
Date: 2015-06-03 16:45
To: user
Subject: How to debug hanging hive on tez job
Hello,
I'm not sure if this question belongs to the hive or tez community so bear with
me.
I am running some hive qu
Thank you!
r7raul1...@163.com
From: Rajesh Balamohan
Date: 2015-06-03 10:43
To: user
Subject: Re: What is the difference between PipelinedSorter and DefaultSorter?
DefaultSorter is the same sorter implementation used in MapReduce world and is
single threaded. PipelinedSorter on the other
uts(),
memoryUpdateCallbackHandler.getMemoryAssigned());
} else {
sorter = new DefaultSorter(getContext(), conf, getNumPhysicalOutputs(),
memoryUpdateCallbackHandler.getMemoryAssigned());
}
When set tez.runtime.sort.threads >1 will choose PipelinedSorter .
r7raul1...@163.com
Congratulations !
r7raul1...@163.com
From: Hitesh Shah
Date: 2015-06-01 12:22
To: dev; user
Subject: [ANNOUNCE] New Apache Tez PMC Member - Jeff Zhang
Hi all,
I am very pleased to announce that Jeff Zhang has been voted in as a member of
the Apache Tez PMC. We appreciate all the work Jeff
information to Timeline
server.
tez.runtime.convert.user-payload.to.history-text
true
tez.am.launch.cmd-opts
-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA
-XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/
r7raul1...@163.com
From: Jianfeng
mapreduce.map.java.opts = -Djava.net.preferIPv4Stack=true -XX:+PrintGCDetails
-verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC
Map Task Maximum Heap Size
mapreduce.map.java.opts.max.heap
825955249 B
default value
r7raul1...@163.com
From: r7raul1...@163.com
Date: 2015-06-01 10
:48 AM, Jianfeng (Jeff) Zhang
wrote:
>
> Could you check the yarn app logs to see what the error is ? If there’s
> still no useful info, you may refer the yarn RM/NN logs
>
>
>
>
> Best Regard,
> Jeff Zhang
>
>
> From: "r7raul1...@163.com"
configuration information to Timeline server
tez.runtime.convert.user-payload.to.history-text
true
tez.am.launch.cmd-opts
-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA
-XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/
r7raul1...@163.com
From
sec HDFS Read: 1985399
HDFS Write: 20068 SUCCESS
Stage-Stage-2: Map: 1 Reduce: 1 Cumulative CPU: 2.35 sec HDFS Read: 24481 HDFS
Write: 6 SUCCESS
Total MapReduce CPU Time Spent: 5 seconds 310 msec
That's ok.
r7raul1...@163.com
How to find query log when I use hiveserver2 on tez? My enviroment is hive
1.1.0+tez0.53 use beeline to connect hiveserver2.
r7raul1...@163.com
I use hive1.1.0 beeline remote connect to hiveserver2. When I submit query ,
only query result return. No log return. Where can I find the execution log?
r7raul1...@163.com
when one reduce process too many data(skew join) set
hive.tez.auto.reducer.parallelism =true can slove this problem?
r7raul1...@163.com
When one reduce or map process too much data. Can vertexManager create some
reduce or map to help process data dynamically?
How to slove data skew problem by tez or hive on tez ?
r7raul1...@163.com
Thank you! I think this should be well doucumented.
r7raul1...@163.com
发件人： Vikram Dixit
发送时间： 2015-05-13 09:38
收件人： user
主题： Re: 回复: Re: hive on tez not convert map join to broadcast join
It does not have any effect in tez.
From: "r7raul1...@163.com"
Reply-To: user
Date: Tuesd
t)
sort order: ++
Map-reduce partition columns: parnt_ordr_code (type: string), comb_prod_id
(type: bigint)
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
value expressions: end_user_id (type: bigint), actl_sale_amt (type: double),
sale_num (type: bigint), ds (type: string)
r7raul1...@163.com
skewtablequeryplan.TXT
Description: Binary data
set hive.mapjoin.smalltable.filesize is no effect fot hive tez?
r7raul1...@163.com
发件人： Vikram Dixit
发送时间： 2015-05-13 09:26
收件人： user
主题： Re: hive on tez not convert map join to broadcast join
From the explain output, it looks like the small table is of size 57 MB whereas
the no
I usehive 1.1.0 + tez0.53
set hive.mapjoin.smalltable.filesize = 6000;
set hive.auto.convert.join=true;
set hive.auto.convert.join.noconditionaltask=true
set hive.auto.convert.join.noconditionaltask.size=1000
r7raul1...@163.com
From: Vikram Dixit
Date: 2015-05-13 08:47
To: user
string), comb_prod_id (type: bigint)
r7raul1...@163.com
Reducer 8 .. SUCCEEDED 203 203 0 0 0 0
VERTICES: 11/13 [==>>] 55% ELAPSED TIME: 307.54 s
What is the root cause ?
r7raul1...@163.com
sql.txt
Description: Binary data
queryplan.TXT
Description: Binary data
It's OK Thank you ~~
r7raul1...@163.com
r7raul1...@163.com
?? 2015-04-29 09:57
user
?? : Re: Tez taskcount log visualization
Thank you ~ I will try
r7raul1...@163.com
Hitesh Shah
?? 2015-04-29 08:59
user
?? Re
Thank you ~ I will try
r7raul1...@163.com
Hitesh Shah
?? 2015-04-29 08:59
user
?? Re: Tez taskcount log visualization
I think Gopal and/or Prakash had use a simple proxying script that could be
used to get around the CORS issue. @Gopal/@Prakash ?
As for
I should install Ambari first? My cluster is cdh...
r7raul1...@163.com
发件人： Sreenath Somarajapuram
发送时间： 2015-04-28 18:29
收件人： user
主题： Re: 回复: 回复: Tez taskcount log visualization
As mentioned by Prakash, CORS handling is not supported in 2.5. Guess you might
be able to get the UI working
yarn.timeline-service.http-cross-origin.enabled Hadoop 2.5.0 not support.
r7raul1...@163.com
发件人： r7raul1...@163.com
发送时间： 2015-04-28 16:27
收件人： user
主题： 回复: Re: Tez taskcount log visualization
I found tez UI cross domain problem. How to fix it?
XMLHttpRequest cannot load
http
e not allowed access.
r7raul1...@163.com
发件人： r7raul1...@163.com
发送时间： 2015-04-28 11:06
收件人： user
主题： Re: Re: Tez taskcount log visualization
I setup UI using the tez 6.0 build with the UI configured to point to the ATS
server.
My hadoop version is hadoop 2.5.0 and Tez is 0.53.
I try http://192
I setup UI using the tez 6.0 build with the UI configured to point to the ATS
server.
My hadoop version is hadoop 2.5.0 and Tez is 0.53.
I try http://192.168.117.117:8288/ws/v1/timeline/TEZ_DAG_ID?limit=11 will
retrun result ;
But Tez UI always display .
r7raul1...@163.com
From: Hitesh
Thank you! UmmmFirstlly , I should upgrade my hadoop2.3.0 cluster to
hadoop 2.6.0 and hive 1.1.0 to use tez 0.60..But hive docs say support
tez0.52
r7raul1...@163.com
From: Rajesh Balamohan
Date: 2015-04-27 14:28
To: user
Subject: Re: Tez taskcount log visualization
Thank you!
r7raul1...@163.com
From: Rajesh Balamohan
Date: 2015-04-24 17:24
To: user
Subject: Re: How to Tuning Tez Task Performance
Listing some details at very high level,
- Set "tez.task.generate.counters.per.io=true" to get more details on the task
counters. Basically t
I want to Tuning Tez Task Performance. This Tez Task is created by Hive. How
to Tuning Tez Task Performance?
Analyze performance by Tez Task Counts of Tez Log ? Any Suggestion?
r7raul1...@163.com
I didn't config yarn.scheduler.capacity.resource-calculator ,So I am on YARN-CS
+ DefaultResourceCalculator
r7raul1...@163.com
From: Gopal V
Date: 2015-02-04 01:25
To: r7raul1...@163.com; user
Subject: Re: tez map task and reduce task stay pending forerver
On 1/27/15, 9:24 PM, r7
Hello
My cluster mapreduce.map.cpu.vcores setting is 3.
r7raul1...@163.com
From: Hitesh Shah
Date: 2015-01-29 05:36
To: user
Subject: Re: tez map task and reduce task stay pending forerver
Hello
Thanks for tracking down the issue to the vcores setting. Let me dig into that.
Some
I test again, I found if I set mapreduce.map.cpu.vcores >1 ,the job will hang .
Very similar to https://issues.apache.org/jira/browse/TEZ-704
r7raul1...@163.com
From: r7raul1...@163.com
Date: 2015-01-28 10:29
To: user
Subject: Re: Re: tez map task and reduce task stay pending forerver
o y
${fs.defaultFS}/apps/tez-0.5.3/tez-0.5.3-minimal.tar.gz
tez.use.cluster.hadoop-libs
true
Every thing is ok.
I think some config in my cluster is too larger .
r7raul1...@163.com
From: r7raul1...@163.com
Date: 2015-01-28 10:24
To: user
Subject: Re: Re: tez map task and reduce task stay pending
No . set hive.execution.engine=mr , still hang...
r7raul1...@163.com
发件人： Jianfeng (Jeff) Zhang
发送时间： 2015-01-28 10:11
收件人： user
主题： Re: 回复: tez map task and reduce task stay pending forerver
Can you run this query successfully using hive on mr ?
Best Regards,
Jeff Zhang
On Wed, Jan
mismatch will cause the problem ?
r7raul1...@163.com
发件人： r7raul1...@163.com
发送时间： 2015-01-27 17:59
收件人： user
主题： 回复: 回复: tez map task and reduce task stay pending forerver
Sorry Gopal V, I made a mistake, My config mapreduce.map.memory.mb is 2867 .
r7raul1...@163.com
发件人： r7raul1...@163
Sorry Gopal V, I made a mistake, My config mapreduce.map.memory.mb is 2867 .
r7raul1...@163.com
发件人： r7raul1...@163.com
发送时间： 2015-01-27 17:58
收件人： user
主题： 回复: 回复: tez map task and reduce task stay pending forerver
Hello Gopal V,
I check my cdh config ,I found
Hello Gopal V,
I check my cdh config ,I found mapreduce.map.memory.mb is 2876.
r7raul1...@163.com
发件人： r7raul1...@163.com
发送时间： 2015-01-27 17:31
收件人： user
主题： 回复: Re: tez map task and reduce task stay pending forerver
I check the hivetez.log . No kill request trigger by hive
I check the hivetez.log . No kill request trigger by hive.
r7raul1...@163.com
发件人： Gopal V
发送时间： 2015-01-27 17:17
收件人： user
抄送： r7raul1...@163.com
主题： Re: tez map task and reduce task stay pending forerver
On 1/27/15, 12:50 AM, r7raul1...@163.com wrote:
> hive 0.14.0 tez 0.53 hadoop 2.
72 matches
Mail list logo
