[jira] [Commented] (TEZ-529) Hive communicates state from RecordReader to Processor via JobConf
[ https://issues.apache.org/jira/browse/TEZ-529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255333#comment-14255333 ] Gunther Hagleitner commented on TEZ-529: No there's a workaround in place that doesn't rely on map.input.file afaik. This can be closed I think - [~sseth]? > Hive communicates state from RecordReader to Processor via JobConf > -- > > Key: TEZ-529 > URL: https://issues.apache.org/jira/browse/TEZ-529 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.2.0 >Reporter: Gunther Hagleitner >Assignee: Siddharth Seth > > Hive currently switches between operator pipelines + partition descriptors > via a map.input.file > In the CombineFileInputFormat case Hive relies on the fact that > CombineFileRecordReader sets this field every time a new file is processed. > This file will then be read in the processor to setup the correct processing > pipeline. > After the Tez refactor RecordReader and TezProcessor use different job conf > instances. Because of that Hive will fail since map.input.file isn't set and > updated in the processor's conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-478) Remove the OUTPUT_CONSUMABLE state from TaskAttemptImpl
[ https://issues.apache.org/jira/browse/TEZ-478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255374#comment-14255374 ] Jeff Zhang commented on TEZ-478: It has been resolved in TEZ-1627, close it > Remove the OUTPUT_CONSUMABLE state from TaskAttemptImpl > --- > > Key: TEZ-478 > URL: https://issues.apache.org/jira/browse/TEZ-478 > Project: Apache Tez > Issue Type: Task >Reporter: Siddharth Seth > > This was introduced for in-mem shuffle, and is liekly not required anymore. > Before removing this though, need to consider how streaming will be handled, > and whether this set of transitions in it's current state is adequate for > streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (TEZ-478) Remove the OUTPUT_CONSUMABLE state from TaskAttemptImpl
[ https://issues.apache.org/jira/browse/TEZ-478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-478: --- Comment: was deleted (was: It has been resolved in TEZ-1627, close it) > Remove the OUTPUT_CONSUMABLE state from TaskAttemptImpl > --- > > Key: TEZ-478 > URL: https://issues.apache.org/jira/browse/TEZ-478 > Project: Apache Tez > Issue Type: Task >Reporter: Siddharth Seth > > This was introduced for in-mem shuffle, and is liekly not required anymore. > Before removing this though, need to consider how streaming will be handled, > and whether this set of transitions in it's current state is adequate for > streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-478) Remove the OUTPUT_CONSUMABLE state from TaskAttemptImpl
[ https://issues.apache.org/jira/browse/TEZ-478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255375#comment-14255375 ] Jeff Zhang commented on TEZ-478: It has been resolved in TEZ-1627, close it > Remove the OUTPUT_CONSUMABLE state from TaskAttemptImpl > --- > > Key: TEZ-478 > URL: https://issues.apache.org/jira/browse/TEZ-478 > Project: Apache Tez > Issue Type: Task >Reporter: Siddharth Seth > > This was introduced for in-mem shuffle, and is liekly not required anymore. > Before removing this though, need to consider how streaming will be handled, > and whether this set of transitions in it's current state is adequate for > streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TEZ-478) Remove the OUTPUT_CONSUMABLE state from TaskAttemptImpl
[ https://issues.apache.org/jira/browse/TEZ-478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang resolved TEZ-478. Resolution: Duplicate > Remove the OUTPUT_CONSUMABLE state from TaskAttemptImpl > --- > > Key: TEZ-478 > URL: https://issues.apache.org/jira/browse/TEZ-478 > Project: Apache Tez > Issue Type: Task >Reporter: Siddharth Seth > > This was introduced for in-mem shuffle, and is liekly not required anymore. > Before removing this though, need to consider how streaming will be handled, > and whether this set of transitions in it's current state is adequate for > streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1875) dropdown filters do not work on vertices and task attempts page
[ https://issues.apache.org/jira/browse/TEZ-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255382#comment-14255382 ] Jeff Zhang commented on TEZ-1875: - Thanks [~pramachandran] [~Sreenath] Committed to master commit 793fd72c8f799123841311c9c5357c8ddc04f9c1 (HEAD, origin/master, origin/HEAD, master) Author: Jeff Zhang Date: Mon Dec 22 11:26:09 2014 +0800 TEZ-1875. dropdown filters do not work on vertices and task attempts page (Prakash Ramachandran via zjffdu) > dropdown filters do not work on vertices and task attempts page > --- > > Key: TEZ-1875 > URL: https://issues.apache.org/jira/browse/TEZ-1875 > Project: Apache Tez > Issue Type: Bug >Reporter: Prakash Ramachandran >Assignee: Prakash Ramachandran >Priority: Blocker > Labels: tez-ui > Fix For: 0.6.0 > > Attachments: TEZ-1875.1.patch, TEZ-1875.2.patch > > > The Status dropdown list in the tab of vertex & taskattempt tab always > been "ALL" even I change it. The actual list gets filtered however the > dropdown is not updated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1874) AM Inline mode property is deprecated
[ https://issues.apache.org/jira/browse/TEZ-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255385#comment-14255385 ] Jeff Zhang commented on TEZ-1874: - [~hitesh], I think inline is already supported in local mode, Do you mean we may support it in non-local mode ? If that's yes, then we should still keep it there. > AM Inline mode property is deprecated > - > > Key: TEZ-1874 > URL: https://issues.apache.org/jira/browse/TEZ-1874 > Project: Apache Tez > Issue Type: Bug >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-1874.patch > > > Looks like a legacy code for local mode, now we have new property > TEZ_AM_INLINE_TASK_EXECUTION_MAX_TASKS for that, should remove it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-285) Complete 'terminationCause' tracking to TaskImpl
[ https://issues.apache.org/jira/browse/TEZ-285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255388#comment-14255388 ] Jeff Zhang commented on TEZ-285: [~hitesh], We already have TaskTerminationCause, but haven't set it as part of TaskImpl ( no getTaskTerminationCause and trySetTerminationCause). If later UI needs to display the TaskTermincauseCause, we couldn't get it. I think we can use this jira to consolidate the terminationCause for DAG/Vertex/Task/TaskAttempt. > Complete 'terminationCause' tracking to TaskImpl > > > Key: TEZ-285 > URL: https://issues.apache.org/jira/browse/TEZ-285 > Project: Apache Tez > Issue Type: Bug >Reporter: Mike Liddell > Labels: TEZ-0.2.0 > > As for Tez-284, but this time for tracking root cause of Task termination. > Changeset for Tez-141 includes the preliminaries for TaskTerminationCause. > We will need to add: > OWN_TASKATTEMPTS_FAILED and perhaps other causes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1884) Some places still use job ( should use dag )
[ https://issues.apache.org/jira/browse/TEZ-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-1884: Summary: Some places still use job ( should use dag ) (was: Lots of places still use job ( should use dag )) > Some places still use job ( should use dag ) > > > Key: TEZ-1884 > URL: https://issues.apache.org/jira/browse/TEZ-1884 > Project: Apache Tez > Issue Type: Bug >Reporter: Jeff Zhang > Labels: newbie++ > > Legacy code issue because some code are borrowed from mapreduce, should > rename job to dag. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1884) Lots of places still use job ( should use dag )
Jeff Zhang created TEZ-1884: --- Summary: Lots of places still use job ( should use dag ) Key: TEZ-1884 URL: https://issues.apache.org/jira/browse/TEZ-1884 Project: Apache Tez Issue Type: Bug Reporter: Jeff Zhang Legacy code issue because some code are borrowed from mapreduce, should rename job to dag. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1885) Should check whether AM is shutting down first in submitDAGToAppMaster
Jeff Zhang created TEZ-1885:
---
Summary: Should check whether AM is shutting down first in
submitDAGToAppMaster
Key: TEZ-1885
URL: https://issues.apache.org/jira/browse/TEZ-1885
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Jeff Zhang
{code}
2014-12-16 06:26:54,372 INFO [Timer-0] org.apache.tez.dag.app.DAGAppMaster: No
current running DAG, shutting down the AM
2014-12-16 06:26:54,372 INFO [Timer-0] org.apache.tez.dag.app.DAGAppMaster:
Handling DAGAppMaster shutdown
2014-12-16 06:26:56,465 INFO [IPC Server handler 0 on 37746]
org.apache.hadoop.ipc.Server: IPC Server handler 0 on 37746, call
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB.submitDAG from
10.66.7.123:35917 Call#7994 Retry#0: error:
org.apache.tez.dag.api.TezException: App master already running a DAG
org.apache.tez.dag.api.TezException: App master already running a DAG
at
org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:931)
at
org.apache.tez.dag.app.DAGAppMaster$DAGClientHandler.submitDAG(DAGAppMaster.java:1065)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:131)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:8292)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:599)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
2014-12-16 06:26:56,491 INFO [IPC Server handler 0 on 37746]
org.apache.tez.dag.app.DAGAppMaster: Received message to shutdown AM
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-1885) Should check whether AM is shutting down first in submitDAGToAppMaster
[
https://issues.apache.org/jira/browse/TEZ-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Zhang updated TEZ-1885:
Description:
Here's logs which demonstrate the scenario: Shutdown AM -> submit new DAG
But the error message is not correct (" App master already running a DAG" ), it
should be "AM unable to accept new DAG submissions, In the process of shutting
down"
{code}
2014-12-16 06:26:54,372 INFO [Timer-0] org.apache.tez.dag.app.DAGAppMaster: No
current running DAG, shutting down the AM
2014-12-16 06:26:54,372 INFO [Timer-0] org.apache.tez.dag.app.DAGAppMaster:
Handling DAGAppMaster shutdown
2014-12-16 06:26:56,465 INFO [IPC Server handler 0 on 37746]
org.apache.hadoop.ipc.Server: IPC Server handler 0 on 37746, call
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB.submitDAG from
10.66.7.123:35917 Call#7994 Retry#0: error:
org.apache.tez.dag.api.TezException: App master already running a DAG
org.apache.tez.dag.api.TezException: App master already running a DAG
at
org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:931)
at
org.apache.tez.dag.app.DAGAppMaster$DAGClientHandler.submitDAG(DAGAppMaster.java:1065)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:131)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:8292)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:599)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
2014-12-16 06:26:56,491 INFO [IPC Server handler 0 on 37746]
org.apache.tez.dag.app.DAGAppMaster: Received message to shutdown AM
{code}
was:
{code}
2014-12-16 06:26:54,372 INFO [Timer-0] org.apache.tez.dag.app.DAGAppMaster: No
current running DAG, shutting down the AM
2014-12-16 06:26:54,372 INFO [Timer-0] org.apache.tez.dag.app.DAGAppMaster:
Handling DAGAppMaster shutdown
2014-12-16 06:26:56,465 INFO [IPC Server handler 0 on 37746]
org.apache.hadoop.ipc.Server: IPC Server handler 0 on 37746, call
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB.submitDAG from
10.66.7.123:35917 Call#7994 Retry#0: error:
org.apache.tez.dag.api.TezException: App master already running a DAG
org.apache.tez.dag.api.TezException: App master already running a DAG
at
org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:931)
at
org.apache.tez.dag.app.DAGAppMaster$DAGClientHandler.submitDAG(DAGAppMaster.java:1065)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:131)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:8292)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:599)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
2014-12-16 06:26:56,491 INFO [IPC Server handler 0 on 37746]
org.apache.tez.dag.app.DAGAppMaster: Received message to shutdown AM
{code}
> Should check whether AM is shutting down first in submitDAGToAppMaster
> --
>
> Key: TEZ-1885
> URL: https://issues.apache.org/jira/browse/TEZ-1885
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>
> Here's logs which demonstrate the scenario: Shutdown AM -> submit new DAG
> But the error message is not correct (" App master already running a DAG" ),
> it should be "AM unable to accept new DAG submissions, In the process of
> shu
[jira] [Resolved] (TEZ-1358) Display better diagnostics when tasks fail to launch
[ https://issues.apache.org/jira/browse/TEZ-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang resolved TEZ-1358. - Resolution: Not a Problem Close it as not a problem > Display better diagnostics when tasks fail to launch > - > > Key: TEZ-1358 > URL: https://issues.apache.org/jira/browse/TEZ-1358 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Jeff Zhang > > Tasks could fail to launch due to various issues - bad environment settings, > localization failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1843) AM should go to KILLED state when it is killed in RUNNING state
[ https://issues.apache.org/jira/browse/TEZ-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255474#comment-14255474 ] Jeff Zhang commented on TEZ-1843: - [~hitesh], First I think we should make user understand what the YarnApplicationStatus mean. IMO, that should mean the final status of this Tez Application rather than status of any dags. Because there may be multiple dags in one TezSession, we can't build 1-to-1 relationship between DAG and Tez Application. So I think if the session is killed when a dag is not running, that means Tez Application has already done its work and user want to stop it, it should be normal behavior which should result in SUCCEEDED. While if the session is shut down when a dag is running. That means user want to interrupt the Tez Application which resulting in KILLED make more sense. BTW. I think finally we should encourage user to use the tez-ui to rather than the yarn ui. > AM should go to KILLED state when it is killed in RUNNING state > --- > > Key: TEZ-1843 > URL: https://issues.apache.org/jira/browse/TEZ-1843 > Project: Apache Tez > Issue Type: Bug >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-1843.patch > > > Currently, AM will go to SUCCEEDED state when it is killed in RUNNING state, > the yarn application also show SUCCEEDED state. But it should go to KILLED > state instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1886) remove deprecation warnings for tez-ui on the console
Prakash Ramachandran created TEZ-1886:
-
Summary: remove deprecation warnings for tez-ui on the console
Key: TEZ-1886
URL: https://issues.apache.org/jira/browse/TEZ-1886
Project: Apache Tez
Issue Type: Bug
Reporter: Prakash Ramachandran
Assignee: Prakash Ramachandran
after the update to bower.json ember-i18n has the following warnings
- CLDR.js has been deprecated; use Ember-I18n's i18n-plurals.js instead.
- Ember.I18n t helper called with unquoted key: common.applicationId. In the
future, this will be treated as a bound property, not a string literal
- DEPRECATION: The {{t}} helper will no longer use a tag in future
versions of Ember.I18n. Set Ember.FEATURES.I18N_TRANSLATE_HELPER_SPAN to false
to quiet these warnings and maintain older behavior.
This pollutes the debug/info messages on the console.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-1886) remove deprecation warnings for tez-ui on the console
[
https://issues.apache.org/jira/browse/TEZ-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prakash Ramachandran updated TEZ-1886:
--
Attachment: TEZ-1886.1.patch
straightforward patch
[~Sreenath] review please.
> remove deprecation warnings for tez-ui on the console
> -
>
> Key: TEZ-1886
> URL: https://issues.apache.org/jira/browse/TEZ-1886
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Prakash Ramachandran
>Assignee: Prakash Ramachandran
> Labels: tez-ui
> Attachments: TEZ-1886.1.patch
>
>
> after the update to bower.json ember-i18n has the following warnings
> - CLDR.js has been deprecated; use Ember-I18n's i18n-plurals.js instead.
> - Ember.I18n t helper called with unquoted key: common.applicationId. In the
> future, this will be treated as a bound property, not a string literal
> - DEPRECATION: The {{t}} helper will no longer use a tag in future
> versions of Ember.I18n. Set Ember.FEATURES.I18N_TRANSLATE_HELPER_SPAN to
> false to quiet these warnings and maintain older behavior.
> This pollutes the debug/info messages on the console.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
