[jira] [Commented] (HADOOP-11935) Provide optional native implementation of stat syscall.
[
https://issues.apache.org/jira/browse/HADOOP-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16827965#comment-16827965
]
Yeliang Cang commented on HADOOP-11935:
---
I think this has been resolved by HADOOP-14600
> Provide optional native implementation of stat syscall.
> ---
>
> Key: HADOOP-11935
> URL: https://issues.apache.org/jira/browse/HADOOP-11935
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, native
>Reporter: Chris Nauroth
>Priority: Major
> Attachments: HADOOP-11935-NativeIO-prelim.patch
>
>
> Currently,
> {{RawLocalFileSystem.DeprecatedRawLocalFileStatus#loadPermissionInfo}} is
> implemented as forking an {{ls}} command and parsing the output. This was
> observed to be a bottleneck in YARN-3491. This issue proposes an optional
> native implementation of a {{stat}} syscall through JNI. We would maintain
> the existing code as a fallback for systems where the native code is not
> available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15913) xml parsing error in a heavily multi-threaded environment
[
https://issues.apache.org/jira/browse/HADOOP-15913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yeliang Cang updated HADOOP-15913:
--
Description:
We met this problem in a production environment, the stack trace like this:
{code}ERROR org.apache.hadoop.hive.ql.exec.Task: Ended Job =
job_1541600895081_0580 with exception 'java.lang.NullPointerException(Inflater
has been closed)'
java.lang.NullPointerException: Inflater has been closed
at java.util.zip.Inflater.ensureOpen(Inflater.java:389)
at java.util.zip.Inflater.inflate(Inflater.java:257)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:154)
at java.io.BufferedReader.readLine(BufferedReader.java:317)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at
javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:319)
at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
at
javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2524)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2501)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2407)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:983)
at
org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2007)
at org.apache.hadoop.mapred.JobConf.(JobConf.java:479)
at org.apache.hadoop.mapred.JobConf.(JobConf.java:469)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:188)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:601)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:599)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at
org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:599)
at org.apache.hadoop.mapred.JobClient.getJobInner(JobClient.java:609)
at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:639)
at
org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:294)
at
org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:558)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:457)
at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:141)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197){code}
and can reproduce it in our test environment by steps below:
1. set configs:
{code}
hive.server2.async.exec.threads = 50
hive.server2.async.exec.wait.queue.size = 100
{code}
2. open 4 beeline terminates in 4 different nodes.
3. create 30 queries in each beeline terminate. Each query include "add jar
xxx.jar" like this:
{code}
add jar mykeytest-1.0-SNAPSHOT.jar;
create temporary function ups as 'com.xxx.manager.GetCommentNameOrId';
insert into test partition(tjrq = ${my_no}, ywtx = '${my_no2}' )
select dt.d_year as i_brand
,item.i_brand_id as i_item_sk
,ups(item.i_brand) as i_product_name
,sum(ss_ext_sales_price) as i_category_id
from date_dim dt
,store_sales
,item
where dt.d_date_sk = store_sales.ss_sold_date_sk
and store_sales.ss_item_sk = item.i_item_sk
and item.i_manufact_id = 436
and dt.d_moy=12
group by dt.d_year
,item.i_brand
,item.i_brand_id
order by dt.d_year
{code}
and all these 120 queries connect to one hiveserver2
Run all the query concurrently, and will see the stack trace abover in
hiveserver2 log
was:
We met this problem in a production environment, the stack trace like this:
{code}ERROR org.apache.hadoop.hive.ql.exec.Task: Ended Job =
job_1541600895081_0580 with exception 'java.lang.NullPointerException(Inflater
has been closed)'
java.lang.NullPointerException: Inflater has been closed
at java.util.zip.Inflater.ensureOpen(Inflater.java:389)
at java.util.zip.Inflater.inflate(Inflater.java:257)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(
[jira] [Updated] (HADOOP-15913) xml parsing error in a heavily multi-threaded environment
[ https://issues.apache.org/jira/browse/HADOOP-15913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yeliang Cang updated HADOOP-15913: -- Priority: Critical (was: Major) > xml parsing error in a heavily multi-threaded environment > - > > Key: HADOOP-15913 > URL: https://issues.apache.org/jira/browse/HADOOP-15913 > Project: Hadoop Common > Issue Type: Bug >Reporter: Yeliang Cang >Priority: Critical > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15913) xml parsing error in a heavily multi-threaded environment
[
https://issues.apache.org/jira/browse/HADOOP-15913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yeliang Cang updated HADOOP-15913:
--
Description:
We met this problem in a production environment, the stack trace like this:
{code}ERROR org.apache.hadoop.hive.ql.exec.Task: Ended Job =
job_1541600895081_0580 with exception 'java.lang.NullPointerException(Inflater
has been closed)'
java.lang.NullPointerException: Inflater has been closed
at java.util.zip.Inflater.ensureOpen(Inflater.java:389)
at java.util.zip.Inflater.inflate(Inflater.java:257)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:154)
at java.io.BufferedReader.readLine(BufferedReader.java:317)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at
javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:319)
at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
at
javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2524)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2501)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2407)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:983)
at
org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2007)
at org.apache.hadoop.mapred.JobConf.(JobConf.java:479)
at org.apache.hadoop.mapred.JobConf.(JobConf.java:469)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:188)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:601)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:599)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at
org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:599)
at org.apache.hadoop.mapred.JobClient.getJobInner(JobClient.java:609)
at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:639)
at
org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:294)
at
org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:558)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:457)
at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:141)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197){code}
and can reproduce it in our test environment by steps below:
1. set configs:
{code}
hive.server2.async.exec.threads = 50
hive.server2.async.exec.wait.queue.size = 100
{code}
2. open 4 beeline terminates in 4 different nodes.
3. create 30 queries in each beeline terminate, and all these 120 queries
connect to one hiveserver2
> xml parsing error in a heavily multi-threaded environment
> -
>
> Key: HADOOP-15913
> URL: https://issues.apache.org/jira/browse/HADOOP-15913
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.7.3
>Reporter: Yeliang Cang
>Priority: Critical
>
> We met this problem in a production environment, the stack trace like this:
> {code}ERROR org.apache.hadoop.hive.ql.exec.Task: Ended Job =
> job_1541600895081_0580 with exception
> 'java.lang.NullPointerException(Inflater has been closed)'
> java.lang.NullPointerException: Inflater has been closed
> at java.util.zip.Inflater.ensureOpen(Inflater.java:389)
> at java.util.zip.Inflater.inflate(Inflater.java:257)
> at
> java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.
[jira] [Updated] (HADOOP-15913) xml parsing error in a heavily multi-threaded environment
[ https://issues.apache.org/jira/browse/HADOOP-15913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yeliang Cang updated HADOOP-15913: -- Component/s: common > xml parsing error in a heavily multi-threaded environment > - > > Key: HADOOP-15913 > URL: https://issues.apache.org/jira/browse/HADOOP-15913 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.7.3 >Reporter: Yeliang Cang >Priority: Critical > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15913) xml parsing error in a heavily multi-threaded environment
Yeliang Cang created HADOOP-15913: - Summary: xml parsing error in a heavily multi-threaded environment Key: HADOOP-15913 URL: https://issues.apache.org/jira/browse/HADOOP-15913 Project: Hadoop Common Issue Type: Bug Reporter: Yeliang Cang -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15913) xml parsing error in a heavily multi-threaded environment
[
https://issues.apache.org/jira/browse/HADOOP-15913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679875#comment-16679875
]
Yeliang Cang commented on HADOOP-15913:
---
We have already applied https://issues.apache.org/jira/browse/HADOOP-12404
and still see the error.
Based on comments in https://github.com/mikiobraun/jblas/issues/103, it
suggests that it is misuse ZipFile object between multiple threads
> xml parsing error in a heavily multi-threaded environment
> -
>
> Key: HADOOP-15913
> URL: https://issues.apache.org/jira/browse/HADOOP-15913
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.7.3
>Reporter: Yeliang Cang
>Priority: Critical
>
> We met this problem in a production environment, the stack trace like this:
> {code}ERROR org.apache.hadoop.hive.ql.exec.Task: Ended Job =
> job_1541600895081_0580 with exception
> 'java.lang.NullPointerException(Inflater has been closed)'
> java.lang.NullPointerException: Inflater has been closed
> at java.util.zip.Inflater.ensureOpen(Inflater.java:389)
> at java.util.zip.Inflater.inflate(Inflater.java:257)
> at
> java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:319)
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
> at
> javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2524)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2501)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2407)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:983)
> at
> org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2007)
> at org.apache.hadoop.mapred.JobConf.(JobConf.java:479)
> at org.apache.hadoop.mapred.JobConf.(JobConf.java:469)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:188)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:601)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:599)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at
> org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:599)
> at org.apache.hadoop.mapred.JobClient.getJobInner(JobClient.java:609)
> at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:639)
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:294)
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:558)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:457)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:141)
> at
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197){code}
> and can reproduce it in our test environment by steps below:
> 1. set configs:
> {code}
> hive.server2.async.exec.threads = 50
> hive.server2.async.exec.wait.queue.size = 100
> {code}
> 2. open 4 beeline terminates in 4 different nodes.
> 3. create 30 queries in each beeline terminate. Each query include "add jar
> xxx.jar" like this:
> {code}
> add jar mykeytest-1.0-SNAPSHOT.jar;
> create temporary function ups as 'com.xxx.manager.GetCommentNameOrId';
> insert into test partition(tjrq = ${my_no}, ywtx = '${my_no2}' )
> select dt.d_year as i_brand
>,item.i_brand_id as i_item_sk
>,ups(item.i_brand) as i_product_name
>,sum(ss_ext_sales_price) as i_category_id
> from date_dim dt
> ,store_sales
> ,item
> where dt.d_date_sk = store_sales.ss_sold_date_sk
>and store_sales.ss_item_sk = item.i_item_sk
>and item.i_manufact_id = 436
>and dt.d_moy=12
> group by dt.d_year
> ,item.i_brand
> ,item.i_b
[jira] [Updated] (HADOOP-15913) xml parsing error in a heavily multi-threaded environment
[ https://issues.apache.org/jira/browse/HADOOP-15913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yeliang Cang updated HADOOP-15913: -- Affects Version/s: 2.7.3 > xml parsing error in a heavily multi-threaded environment > - > > Key: HADOOP-15913 > URL: https://issues.apache.org/jira/browse/HADOOP-15913 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.7.3 >Reporter: Yeliang Cang >Priority: Critical > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15913) xml parsing error in a heavily multi-threaded environment
[
https://issues.apache.org/jira/browse/HADOOP-15913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679882#comment-16679882
]
Yeliang Cang commented on HADOOP-15913:
---
[~jeagles], [~zxu], [~arun.sur...@gmail.com], [~ajisakaa], what do you think
about this bug I encountered? Any thoughts will be great appreciated!
> xml parsing error in a heavily multi-threaded environment
> -
>
> Key: HADOOP-15913
> URL: https://issues.apache.org/jira/browse/HADOOP-15913
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.7.3
>Reporter: Yeliang Cang
>Priority: Critical
>
> We met this problem in a production environment, the stack trace like this:
> {code}ERROR org.apache.hadoop.hive.ql.exec.Task: Ended Job =
> job_1541600895081_0580 with exception
> 'java.lang.NullPointerException(Inflater has been closed)'
> java.lang.NullPointerException: Inflater has been closed
> at java.util.zip.Inflater.ensureOpen(Inflater.java:389)
> at java.util.zip.Inflater.inflate(Inflater.java:257)
> at
> java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:319)
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
> at
> javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2524)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2501)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2407)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:983)
> at
> org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2007)
> at org.apache.hadoop.mapred.JobConf.(JobConf.java:479)
> at org.apache.hadoop.mapred.JobConf.(JobConf.java:469)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:188)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:601)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:599)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at
> org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:599)
> at org.apache.hadoop.mapred.JobClient.getJobInner(JobClient.java:609)
> at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:639)
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:294)
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:558)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:457)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:141)
> at
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197){code}
> and can reproduce it in our test environment by steps below:
> 1. set configs:
> {code}
> hive.server2.async.exec.threads = 50
> hive.server2.async.exec.wait.queue.size = 100
> {code}
> 2. open 4 beeline terminates in 4 different nodes.
> 3. create 30 queries in each beeline terminate. Each query include "add jar
> xxx.jar" like this:
> {code}
> add jar mykeytest-1.0-SNAPSHOT.jar;
> create temporary function ups as 'com.xxx.manager.GetCommentNameOrId';
> insert into test partition(tjrq = ${my_no}, ywtx = '${my_no2}' )
> select dt.d_year as i_brand
>,item.i_brand_id as i_item_sk
>,ups(item.i_brand) as i_product_name
>,sum(ss_ext_sales_price) as i_category_id
> from date_dim dt
> ,store_sales
> ,item
> where dt.d_date_sk = store_sales.ss_sold_date_sk
>and store_sales.ss_item_sk = item.i_item_sk
>and item.i_manufact_id = 436
>and dt.d_moy=12
> group by dt.d_year
> ,item.i_brand
> ,item.i_brand_id
> order by dt.d_year
> {code}
> and all these 120 queries connect to one hiveserve
[jira] [Commented] (HADOOP-14635) Javadoc correction for AccessControlList#buildACL
[
https://issues.apache.org/jira/browse/HADOOP-14635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079836#comment-16079836
]
Yeliang Cang commented on HADOOP-14635:
---
[~bibinchundatt], I have modified the javadoc. Please have a look, thank you!
> Javadoc correction for AccessControlList#buildACL
> -
>
> Key: HADOOP-14635
> URL: https://issues.apache.org/jira/browse/HADOOP-14635
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-14635-001.patch
>
>
> {{AccessControlList#buildACL}}
> {code}
> /**
>* Build ACL from the given two Strings.
>* The Strings contain comma separated values.
>*
>* @param aclString build ACL from array of Strings
>*/
> private void buildACL(String[] userGroupStrings) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14635) Javadoc correction for AccessControlList#buildACL
[
https://issues.apache.org/jira/browse/HADOOP-14635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yeliang Cang updated HADOOP-14635:
--
Status: Patch Available (was: Open)
> Javadoc correction for AccessControlList#buildACL
> -
>
> Key: HADOOP-14635
> URL: https://issues.apache.org/jira/browse/HADOOP-14635
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-14635-001.patch
>
>
> {{AccessControlList#buildACL}}
> {code}
> /**
>* Build ACL from the given two Strings.
>* The Strings contain comma separated values.
>*
>* @param aclString build ACL from array of Strings
>*/
> private void buildACL(String[] userGroupStrings) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14635) Javadoc correction for AccessControlList#buildACL
[
https://issues.apache.org/jira/browse/HADOOP-14635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yeliang Cang reassigned HADOOP-14635:
-
Assignee: Yeliang Cang
> Javadoc correction for AccessControlList#buildACL
> -
>
> Key: HADOOP-14635
> URL: https://issues.apache.org/jira/browse/HADOOP-14635
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Yeliang Cang
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-14635-001.patch
>
>
> {{AccessControlList#buildACL}}
> {code}
> /**
>* Build ACL from the given two Strings.
>* The Strings contain comma separated values.
>*
>* @param aclString build ACL from array of Strings
>*/
> private void buildACL(String[] userGroupStrings) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14635) Javadoc correction for AccessControlList#buildACL
[
https://issues.apache.org/jira/browse/HADOOP-14635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yeliang Cang updated HADOOP-14635:
--
Attachment: HADOOP-14635-001.patch
> Javadoc correction for AccessControlList#buildACL
> -
>
> Key: HADOOP-14635
> URL: https://issues.apache.org/jira/browse/HADOOP-14635
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-14635-001.patch
>
>
> {{AccessControlList#buildACL}}
> {code}
> /**
>* Build ACL from the given two Strings.
>* The Strings contain comma separated values.
>*
>* @param aclString build ACL from array of Strings
>*/
> private void buildACL(String[] userGroupStrings) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14690) RetryInvocationHandler$RetryInfo should override toString()
[
https://issues.apache.org/jira/browse/HADOOP-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yeliang Cang updated HADOOP-14690:
--
Attachment: HADOOP-14690-001.patch
Hi [~ajisakaa], I submit a patch like you suggested. Please see it!
> RetryInvocationHandler$RetryInfo should override toString()
> ---
>
> Key: HADOOP-14690
> URL: https://issues.apache.org/jira/browse/HADOOP-14690
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Priority: Minor
> Labels: newbie, supportability
> Attachments: HADOOP-14690-001.patch
>
>
> {code:title=RetryInvocationHandler.java}
> LOG.trace("#{} processRetryInfo: retryInfo={}, waitTime={}",
> callId, retryInfo, waitTime);
> {code}
> RetryInfo is used for logging but it does not output useful information.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14690) RetryInvocationHandler$RetryInfo should override toString()
[
https://issues.apache.org/jira/browse/HADOOP-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yeliang Cang updated HADOOP-14690:
--
Assignee: Yeliang Cang
Status: Patch Available (was: Open)
> RetryInvocationHandler$RetryInfo should override toString()
> ---
>
> Key: HADOOP-14690
> URL: https://issues.apache.org/jira/browse/HADOOP-14690
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Assignee: Yeliang Cang
>Priority: Minor
> Labels: newbie, supportability
> Attachments: HADOOP-14690-001.patch
>
>
> {code:title=RetryInvocationHandler.java}
> LOG.trace("#{} processRetryInfo: retryInfo={}, waitTime={}",
> callId, retryInfo, waitTime);
> {code}
> RetryInfo is used for logging but it does not output useful information.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14690) RetryInvocationHandler$RetryInfo should override toString()
[
https://issues.apache.org/jira/browse/HADOOP-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16106827#comment-16106827
]
Yeliang Cang commented on HADOOP-14690:
---
Thank you for the review, [~ajisakaa]!
> RetryInvocationHandler$RetryInfo should override toString()
> ---
>
> Key: HADOOP-14690
> URL: https://issues.apache.org/jira/browse/HADOOP-14690
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Assignee: Yeliang Cang
>Priority: Minor
> Labels: newbie, supportability
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HADOOP-14690-001.patch
>
>
> {code:title=RetryInvocationHandler.java}
> LOG.trace("#{} processRetryInfo: retryInfo={}, waitTime={}",
> callId, retryInfo, waitTime);
> {code}
> RetryInfo is used for logging but it does not output useful information.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14784) [KMS] Improve KeyAuthorizationKeyProvider#toString()
[
https://issues.apache.org/jira/browse/HADOOP-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yeliang Cang updated HADOOP-14784:
--
Status: Patch Available (was: Open)
> [KMS] Improve KeyAuthorizationKeyProvider#toString()
>
>
> Key: HADOOP-14784
> URL: https://issues.apache.org/jira/browse/HADOOP-14784
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Priority: Trivial
> Labels: newbie
> Attachments: HADOOP-14784.001.patch
>
>
> When KMS server starts, it loads KeyProviderCryptoExtension and print the
> following message:
> {noformat}
> 2017-08-17 04:57:13,348 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSWebApp: Initialized
> KeyProviderCryptoExtension EagerKeyGeneratorKeyProviderCryptoExtension:
> KeyProviderCryptoExtension: CachingKeyProvider:
> jceks://file@/var/lib/kms/kms.keystore
> {noformat}
> However, this is confusing as KeyAuthorizationKeyProvider is loaded by not
> shown in this message. KeyAuthorizationKeyProvider#toString should be
> improved, so that in addition to its internal provider, also print its own
> class name when loaded.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14784) [KMS] Improve KeyAuthorizationKeyProvider#toString()
[
https://issues.apache.org/jira/browse/HADOOP-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yeliang Cang updated HADOOP-14784:
--
Attachment: HADOOP-14784.001.patch
> [KMS] Improve KeyAuthorizationKeyProvider#toString()
>
>
> Key: HADOOP-14784
> URL: https://issues.apache.org/jira/browse/HADOOP-14784
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Priority: Trivial
> Labels: newbie
> Attachments: HADOOP-14784.001.patch
>
>
> When KMS server starts, it loads KeyProviderCryptoExtension and print the
> following message:
> {noformat}
> 2017-08-17 04:57:13,348 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSWebApp: Initialized
> KeyProviderCryptoExtension EagerKeyGeneratorKeyProviderCryptoExtension:
> KeyProviderCryptoExtension: CachingKeyProvider:
> jceks://file@/var/lib/kms/kms.keystore
> {noformat}
> However, this is confusing as KeyAuthorizationKeyProvider is loaded by not
> shown in this message. KeyAuthorizationKeyProvider#toString should be
> improved, so that in addition to its internal provider, also print its own
> class name when loaded.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14784) [KMS] Improve KeyAuthorizationKeyProvider#toString()
[
https://issues.apache.org/jira/browse/HADOOP-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131627#comment-16131627
]
Yeliang Cang commented on HADOOP-14784:
---
Hi, [~jojochuang], I submit a patch like you suggested. Please check it!
> [KMS] Improve KeyAuthorizationKeyProvider#toString()
>
>
> Key: HADOOP-14784
> URL: https://issues.apache.org/jira/browse/HADOOP-14784
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Priority: Trivial
> Labels: newbie
> Attachments: HADOOP-14784.001.patch
>
>
> When KMS server starts, it loads KeyProviderCryptoExtension and print the
> following message:
> {noformat}
> 2017-08-17 04:57:13,348 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSWebApp: Initialized
> KeyProviderCryptoExtension EagerKeyGeneratorKeyProviderCryptoExtension:
> KeyProviderCryptoExtension: CachingKeyProvider:
> jceks://file@/var/lib/kms/kms.keystore
> {noformat}
> However, this is confusing as KeyAuthorizationKeyProvider is loaded by not
> shown in this message. KeyAuthorizationKeyProvider#toString should be
> improved, so that in addition to its internal provider, also print its own
> class name when loaded.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14271) Correct spelling of 'occurred' and variants
[ https://issues.apache.org/jira/browse/HADOOP-14271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15955042#comment-15955042 ] Yeliang Cang commented on HADOOP-14271: --- [~chris.douglas]], thanks for your review! > Correct spelling of 'occurred' and variants > --- > > Key: HADOOP-14271 > URL: https://issues.apache.org/jira/browse/HADOOP-14271 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha2 >Reporter: Yeliang Cang >Assignee: Yeliang Cang >Priority: Trivial > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6427.001.patch > > > I have find some spelling mistakes in both hdfs and yarn components. The word > "occured" should be "occurred". -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
