[jira] [Commented] (HADOOP-15327) Upgrade MR ShuffleHandler to use Netty4
[ https://issues.apache.org/jira/browse/HADOOP-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17783867#comment-17783867 ] Duo Zhang commented on HADOOP-15327: So do we have any plan on backporting this to branch-3.3? Netty 3 has a bunch of CVEs... > Upgrade MR ShuffleHandler to use Netty4 > --- > > Key: HADOOP-15327 > URL: https://issues.apache.org/jira/browse/HADOOP-15327 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Szilard Nemeth >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: HADOOP-15327.001.patch, HADOOP-15327.002.patch, > HADOOP-15327.003.patch, HADOOP-15327.004.patch, HADOOP-15327.005.patch, > HADOOP-15327.005.patch, > getMapOutputInfo_BlockingOperationException_awaitUninterruptibly.log, > hades-results-20221108.zip, testfailure-testMapFileAccess-emptyresponse.zip, > testfailure-testReduceFromPartialMem.zip > > Time Spent: 11.5h > Remaining Estimate: 0h > > This way, we can remove the dependencies on the netty3 (jboss.netty) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18207) Introduce hadoop-logging module
[ https://issues.apache.org/jira/browse/HADOOP-18207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729032#comment-17729032 ] Duo Zhang commented on HADOOP-18207: This is a problem that I've been a branch committer in the past but actually, there is no such thing in the official ASF processes, so I will be listed as a 'member' on github, and others may consider I'm a committer of hadoop... Anyway, I could try to contribute more on hadoop and finally become a committer of hadoop so everyone will be happy... And for this patch, it is not merged by me. [~weichiu] has also skimmed the patch and then merged it, I think we can have a implicit +1 from him? If [~ayushtkn] you think this is not valid, we can revert it and let [~vjasani] open a new PR with the addendum integrated, and then we merge it again with an explicit +1 from hadoop committers. Or we could just merge the addendum quickly so it will not break our jenkins jobs. WDYT? Thanks. > Introduce hadoop-logging module > --- > > Key: HADOOP-18207 > URL: https://issues.apache.org/jira/browse/HADOOP-18207 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > There are several goals here: > # Provide the ability to change log level, get log level, etc. > # Place all the appender implementation(?) > # Hide the real logging implementation. > # Later we could remove all the log4j references in other hadoop module. > # Move as much log4j usage to the module as possible. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HADOOP-16206: -- Assignee: Viraj Jasani (was: Duo Zhang) > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[ https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678119#comment-17678119 ] Duo Zhang commented on HADOOP-18206: There are not too many modules which reference commons-logging IIRC, so let's prepare a single PR first. If others have other opinion when reviewing, we can split it. Thanks. > Cleanup the commons-logging references in the code base > --- > > Key: HADOOP-18206 > URL: https://issues.apache.org/jira/browse/HADOOP-18206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Viraj Jasani >Priority: Major > > Should always use slf4j -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[ https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678047#comment-17678047 ] Duo Zhang commented on HADOOP-18206: This issue is just for removing the commons-logging references. For changing log level at runtime, in hbase I introduced a hbase-logging module, where we added log4j as a provided dependencies, and use reflection to call the actual methods in log4j for changing method, so other modules do not need to depend on log4j. We can do the same for hadoop. > Cleanup the commons-logging references in the code base > --- > > Key: HADOOP-18206 > URL: https://issues.apache.org/jira/browse/HADOOP-18206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Viraj Jasani >Priority: Major > > Should always use slf4j -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17557765#comment-17557765 ] Duo Zhang commented on HADOOP-16206: The work is not easy. I haven't found enough time to finish it yet. For security issues, hadoop 3.3.2 or 3.3.3 has already replaced all the log4j dependencies with reload4j dependencies, which should be enough for fixing security issues. You could try to upgrade to these versions. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18208) Remove all the log4j reference in modules other than hadoop-logging
Duo Zhang created HADOOP-18208: -- Summary: Remove all the log4j reference in modules other than hadoop-logging Key: HADOOP-18208 URL: https://issues.apache.org/jira/browse/HADOOP-18208 Project: Hadoop Common Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18207) Introduce a hadoop-logging module
Duo Zhang created HADOOP-18207: -- Summary: Introduce a hadoop-logging module Key: HADOOP-18207 URL: https://issues.apache.org/jira/browse/HADOOP-18207 Project: Hadoop Common Issue Type: Sub-task Reporter: Duo Zhang There are several goals here: 1. Provide the ability to change log level, get log level, etc. 2. Place all the appender implementation(?) 3. Hide the real logging implementation. 4. Later we could remove all the log4j references in other hadoop module. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18206) Cleanup the commons-logging references in the code base
Duo Zhang created HADOOP-18206: -- Summary: Cleanup the commons-logging references in the code base Key: HADOOP-18206 URL: https://issues.apache.org/jira/browse/HADOOP-18206 Project: Hadoop Common Issue Type: Sub-task Reporter: Duo Zhang Should always use slf4j -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16206: --- Parent: (was: HADOOP-12956) Issue Type: Task (was: Sub-task) > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522832#comment-17522832 ] Duo Zhang commented on HADOOP-16206: Thanks [~ste...@apache.org]. Seems no other response, then let me convert this issue to a top level issue, link it with its current parent, and open sub tasks to land the work piece by piece. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17519391#comment-17519391 ] Duo Zhang commented on HADOOP-16206: I've already modified more than 1k lines of code(do not include the log4j.properties change), but maybe only half done. So finally I can provide a patch but I do not think it is easy to review or test. The related code must be changed are: 1. Dynamically changing appenders(add a async appender in front of all the current appenders usually) and log levels. 2. Adding an appender in UTs to capture the log output. 3. Customized appenders, like Log4Json(is it still used?) or Log4jWarningErrorMetricsAppender. 4. In hadoop-kms we will programmingly load the log4j.properties(is it really necessary?) And we still references commons-logging at many places, which should also be purged. I prefer we first discuss and fix the above problems one by one, before finally moving to log4j2, to make the final move a bit easier. For example, I really doubt the value of modifying appenders at runtime, especially adding async appender. This should be done in log4j.properties, and you can also use the -Dhadoop.root.logger=INFO,AsyncAppender to control the appender to use when starting. And for changing log levels and capture log output in UTs, I think we could introduce a hadoop-logging module, provide some logging framework free interfaces to support these operations, so all other modules can just reference these interfaces to avoid depending on the real logging framework directly, then when moving to log4j2, we just need to modify one place, instead of modifying a bunch of classes. If no big concerns, I will file new sub tasks to land the above proposals first. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17515327#comment-17515327 ] Duo Zhang commented on HADOOP-16206: This was talked before. Please see the above comments. The final decision is to move to log4j2 completely, without using the bridge. Of course we could leave the bridge on classpath for our users if they have their own log4j.properties. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17512740#comment-17512740 ] Duo Zhang commented on HADOOP-16206: Some updates here. I've already cleaned up all the log4j1 dependenies and start to fix the compile errors. There are still lots of commons-logging dependecies, I've cleaned them too. Another problem is async http request log setup, now we have a method in the code base to add async appender, but for log4j2, it is not easy to switch appender at runtime, the suggested way is to configure in the config file, and use system properties to change the appenders. Can be discussed later. Now I'm working on implement Log4jWarningErrorMetricsAppender based on log4j2. The way to customize appenders are changed a lot in log4j2, I need to learn first. Will report back later when I have progresses. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17526) Use Slf4jRequestLog for HttpRequestLog
[ https://issues.apache.org/jira/browse/HADOOP-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-17526: --- Release Note: Use jetty's Slf4jRequestLog for http request log. As a side effect, we remove the dummy HttpRequestLogAppender, just make use of DailyRollingFileAppender. But the DRFA in log4j1 lacks the ability of specifying the max retain files, so there is no retainDays config any more. Will add the above ability back after we switch to log4j2's RollingFileAppender. > Use Slf4jRequestLog for HttpRequestLog > -- > > Key: HADOOP-17526 > URL: https://issues.apache.org/jira/browse/HADOOP-17526 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Now the log file name and retain days in HttpRequestLog is configured from > log4j.properties. It can be configured from Configuration instead to drop the > usage of log4j API. > Update > Use jetty's Slf4jRequestLogWriter, we can remove the code dependency on log > appenders. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17526) Use Slf4jRequestLog for HttpRequestLog
[ https://issues.apache.org/jira/browse/HADOOP-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HADOOP-17526: -- Assignee: Duo Zhang > Use Slf4jRequestLog for HttpRequestLog > -- > > Key: HADOOP-17526 > URL: https://issues.apache.org/jira/browse/HADOOP-17526 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > > Now the log file name and retain days in HttpRequestLog is configured from > log4j.properties. It can be configured from Configuration instead to drop the > usage of log4j API. > Update > Use jetty's Slf4jRequestLogWriter, we can remove the code dependency on log > appenders. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17526) Use Slf4jRequestLog for HttpRequestLog
[ https://issues.apache.org/jira/browse/HADOOP-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-17526: --- Description: Now the log file name and retain days in HttpRequestLog is configured from log4j.properties. It can be configured from Configuration instead to drop the usage of log4j API. Update Use jetty's Slf4jRequestLogWriter, we can remove the code dependency on log appenders. was:Now the log file name and retain days in HttpRequestLog is configured from log4j.properties. It can be configured from Configuration instead to drop the usage of log4j API. > Use Slf4jRequestLog for HttpRequestLog > -- > > Key: HADOOP-17526 > URL: https://issues.apache.org/jira/browse/HADOOP-17526 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Priority: Major > > Now the log file name and retain days in HttpRequestLog is configured from > log4j.properties. It can be configured from Configuration instead to drop the > usage of log4j API. > Update > Use jetty's Slf4jRequestLogWriter, we can remove the code dependency on log > appenders. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17526) Use Slf4jRequestLog for HttpRequestLog
[ https://issues.apache.org/jira/browse/HADOOP-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-17526: --- Summary: Use Slf4jRequestLog for HttpRequestLog (was: HttpRequestLog should be configured via Configuration instead of log4j.properties) > Use Slf4jRequestLog for HttpRequestLog > -- > > Key: HADOOP-17526 > URL: https://issues.apache.org/jira/browse/HADOOP-17526 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Priority: Major > > Now the log file name and retain days in HttpRequestLog is configured from > log4j.properties. It can be configured from Configuration instead to drop the > usage of log4j API. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17526) HttpRequestLog should be configured via Configuration instead of log4j.properties
[ https://issues.apache.org/jira/browse/HADOOP-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17500561#comment-17500561 ] Duo Zhang commented on HADOOP-17526: Will pick up this one first. After reading the code I know how HttpRequestLogAppender works, we will get the appender and also get the Filename and RetainDays field, and then create another logger... Anyway, let's change to use Slf4jRequestLog. > HttpRequestLog should be configured via Configuration instead of > log4j.properties > - > > Key: HADOOP-17526 > URL: https://issues.apache.org/jira/browse/HADOOP-17526 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Priority: Major > > Now the log file name and retain days in HttpRequestLog is configured from > log4j.properties. It can be configured from Configuration instead to drop the > usage of log4j API. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17500545#comment-17500545 ] Duo Zhang commented on HADOOP-16206: Will go with the above approach if no other concerns. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[
https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17498910#comment-17498910
]
Duo Zhang commented on HADOOP-16206:
After some investigating, ContainerLogAppender and ContainerRollingLogAppender
are used for container logging. The goal is to limit the maximum size of logs.
ContainerLogAppender will use a in memory queue to hold the tail of the log
messages after exceeding the size limit, while ContainerRollingLogAppender is
just a rolling file appender, we control the total size by setting the size
limit for one file and the number of files to keep.
So for these two appenders, I think maybe we could just leave the rolling file
based one, as they have almost the same ability. And if we choose the rolling
file based one, we do not need to implement a special appender any more, just
use the RollingFileAppender for log4j2 is enough. We can read system properties
in log4j2.properties so it is easy to control where the log.
For TaskLogAppender, it is almost the same with ContainerLogAppender but I
haven't seen where we plan to reference it. Searching the repo, I can only find
references in 0.21.0's release note.
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/site/markdown/release/0.21.0/RELEASENOTES.0.21.0.md
{noformat}
MAPREDUCE-1086 | Major | hadoop commands in streaming tasks are trying to write
to tasktracker's log
This patch makes TT to set HADOOP_ROOT_LOGGER to INFO,TLA by default in the
environment of taskjvm and its children.
MAPREDUCE-1607 | Major | Task controller may not set permissions for a task
cleanup attempt's log directory
Fixed initialization of a task-cleanup attempt's log directory by setting
correct permissions via task-controller. Added new log4j properties
hadoop.tasklog.iscleanup and log4j.appender.TLA.isCleanup to
conf/log4j.properties. Changed the userlogs for a task-cleanup attempt to go
into its own directory instead of the original attempt directory. This is an
incompatible change as old userlogs of cleanup attempt-dirs before this release
will no longer be visible.
{noformat}
I think it provides the same ability with ContainerLogAppender before we have
yarn. So I prefer we just remove it.
For HttpRequestLogAppender, it is just a dummy appender, see the code here
https://github.com/apache/hadoop/blob/5eab9719cbf6b9bddbdb4454a5f8e1ae12495492/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpRequestLog.java#L86
We will replace it with the actual log writer at runtime. There is a related
issue https://issues.apache.org/jira/browse/HADOOP-17526, where we want to
replace it with Slf4jRequestLog, so we could just a normal log4j2 appender to
configure it. So after HADOOP-17526, we could also remove it.
For Log4jWarningErrorMetricsAppender, I think we need to port it to log4j2.
Thanks.
> Migrate from Log4j1 to Log4j2
> -
>
> Key: HADOOP-16206
> URL: https://issues.apache.org/jira/browse/HADOOP-16206
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HADOOP-16206-wip.001.patch
>
>
> This sub-task is to remove log4j1 dependency and add log4j2 dependency.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17526) HttpRequestLog should be configured via Configuration instead of log4j.properties
[ https://issues.apache.org/jira/browse/HADOOP-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17498613#comment-17498613 ] Duo Zhang commented on HADOOP-17526: I prefer we move to Slf4jRequestLog so we can remove the HttpRequestLogAppender. But could someone explain to me that how does this HttpRequestLogAppender work? https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpRequestLogAppender.java Its append method is just empty... > HttpRequestLog should be configured via Configuration instead of > log4j.properties > - > > Key: HADOOP-17526 > URL: https://issues.apache.org/jira/browse/HADOOP-17526 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Priority: Major > > Now the log file name and retain days in HttpRequestLog is configured from > log4j.properties. It can be configured from Configuration instead to drop the > usage of log4j API. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[
https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17498595#comment-17498595
]
Duo Zhang commented on HADOOP-16206:
The progress till now:
I've implemented a converter to convert the log4j1 properties file to log4j2
properties file.
https://github.com/Apache9/TempCode/blob/master/src/main/java/com/github/apache9/log4j2/HadoopLog4j1ConfigurationConverter.java
It will try its best to convert and if it is impossible, will wrap the config
value with ' ' so we can easily grep them out.
And while converting, I found some typos, and also an invalid configuration...
{noformat}
zhangduo@zhangduo-ubuntu:~/hadoop/code$ grep -R log4j.threshhold .
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/resources/log4j-server.properties:log4j.threshhold=ALL
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/conf/yarnservice-log4j.properties:log4j.threshhold=ALL
./hadoop-cloud-storage-project/hadoop-cos/src/test/resources/log4j.properties:log4j.threshhold=ALL
{noformat}
'threshhold' -> 'threshold'
{noformat}
zhangduo@zhangduo-ubuntu:~/hadoop/code$ grep -R log4j.logger= .
./hadoop-tools/hadoop-sls/src/main/sample-conf/log4j.properties:log4j.logger=NONE,
test
{noformat}
log4j.logger is not a valid config way and NONE is also not a valid level?
And after converting all the files, I greped all the '' in the code base.
{noformat}
zhangduo@zhangduo-ubuntu:~/hadoop/code$ grep -R . | grep type
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j2.properties:appender.CLA.type
= org.apache.hadoop.yarn.ContainerLogAppender
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j2.properties:appender.CRLA.type
= org.apache.hadoop.yarn.ContainerRollingLogAppender
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j2.properties:appender.shuffleCLA.type
= org.apache.hadoop.yarn.ContainerLogAppender
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j2.properties:appender.shuffleCRLA.type
= org.apache.hadoop.yarn.ContainerRollingLogAppender
./hadoop-common-project/hadoop-common/src/main/conf/log4j2.properties:appender.TLA.type
= org.apache.hadoop.mapred.TaskLogAppender
./hadoop-common-project/hadoop-common/src/main/conf/log4j2.properties:#appender.namenoderequestlog.type
= org.apache.hadoop.http.HttpRequestLogAppender
./hadoop-common-project/hadoop-common/src/main/conf/log4j2.properties:#appender.datanoderequestlog.type
= org.apache.hadoop.http.HttpRequestLogAppender
./hadoop-common-project/hadoop-common/src/main/conf/log4j2.properties:#appender.resourcemanagerrequestlog.type
= org.apache.hadoop.http.HttpRequestLogAppender
./hadoop-common-project/hadoop-common/src/main/conf/log4j2.properties:#appender.jobhistoryrequestlog.type
= org.apache.hadoop.http.HttpRequestLogAppender
./hadoop-common-project/hadoop-common/src/main/conf/log4j2.properties:#appender.nodemanagerrequestlog.type
= org.apache.hadoop.http.HttpRequestLogAppender
./hadoop-common-project/hadoop-common/src/main/conf/log4j2.properties:appender.EWMA.type
= org.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppender
{noformat}
All of them are customized appenders. They are
org.apache.hadoop.yarn.ContainerLogAppender
org.apache.hadoop.yarn.ContainerRollingLogAppender
org.apache.hadoop.mapred.TaskLogAppender
org.apache.hadoop.http.HttpRequestLogAppender
org.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppender
Next I will dig more on whether these appenders are still needed, and if so,
will try to port them to log4j2.
Thanks.
> Migrate from Log4j1 to Log4j2
> -
>
> Key: HADOOP-16206
> URL: https://issues.apache.org/jira/browse/HADOOP-16206
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HADOOP-16206-wip.001.patch
>
>
> This sub-task is to remove log4j1 dependency and add log4j2 dependency.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17498538#comment-17498538 ] Duo Zhang commented on HADOOP-16206: This is all the log4j configuration files in our code base. ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/resources/log4j-server.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/conf/yarnservice-log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/main/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/log4j.properties ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/default-log4j.properties ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/conf/httpfs-log4j.properties ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/resources/httpfs-log4j.properties ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/resources/default-log4j.properties ./hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/resources/log4j.properties ./hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/resources/log4j.properties ./hadoop-cloud-storage-project/hadoop-cos/src/test/resources/log4j.properties ./hadoop-cloud-storage-project/hadoop-huaweicloud/src/test/resources/log4j.properties ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/resources/log4j.properties ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/resources/log4j.properties ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/log4j.properties ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/resources/log4j.properties ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/log4j.properties ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/resources/log4j.properties ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/log4j.properties ./hadoop-common-project/hadoop-auth-examples/src/main/resources/log4j.properties ./hadoop-common-project/hadoop-registry/src/test/resources/log4j.properties ./hadoop-common-project/hadoop-nfs/src/test/resources/log4j.properties ./hadoop-common-project/hadoop-common/src/main/conf/log4j.properties ./hadoop-common-project/hadoop-common/src/test/resources/log4j.properties ./hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties ./hadoop-common-project/hadoop-kms/src/test/resources/log4j.properties ./hadoop-common-project/hadoop-kms/src/test/resources/log4j-kmsaudit.properties ./hadoop-common-project/hadoop-minikdc/src/main/resources/log4j.properties ./hadoop-client-modules/hadoop-client-integration-tests/src/test/resources/log4j.properties ./hadoop-tools/hadoop-distcp/src/test/resources/log4j.properties ./hadoop-tools/hadoop-azure-datalake/src/test/resources/log4j.pro
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17498537#comment-17498537 ] Duo Zhang commented on HADOOP-16206: The log4j 2.17.2 is out, with LOG4J2-3341 in place. I think it is time to start the work here again. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17487504#comment-17487504 ] Duo Zhang commented on HADOOP-16206: Some updates. I've already tried the newest snapshot version of log4j2 which contains LOG4J2-3341 in HBASE-26723 but we still have some problems. Will come back here once HBASE-26723 could be successfully landed. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476518#comment-17476518 ] Duo Zhang commented on HADOOP-16206: Discussed with the log4j2 community and they are willing to add support for passing '-Dlog4j.rootLogger'. Filed LOG4J2-3341 and linked it here. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[
https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378479#comment-17378479
]
Duo Zhang commented on HADOOP-16206:
{quote}
Are there any updates?
{quote}
Not really. I googled a lot of how to do some magics in the log4j configuration
to split log4j.rootLogger but no progress. Another thing maybe we do something
in java at the very beginning of a start up and do the split but it seems that
we can not set the environment variables in a simple way after a process is
started...
So here for me, the only choice is to do breaking change, like what I have done
in HBase. If we all agree, I still try to go with this direction.
Maybe I should start a discussion thread on the mailing list?
Thanks.
> Migrate from Log4j1 to Log4j2
> -
>
> Key: HADOOP-16206
> URL: https://issues.apache.org/jira/browse/HADOOP-16206
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HADOOP-16206-wip.001.patch
>
>
> This sub-task is to remove log4j1 dependency and add log4j2 dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350474#comment-17350474 ] Duo Zhang commented on HADOOP-16206: I do not think it is about the difficulty, if we choose to do it then I think we could finally get it done. The problem is about compatibility... I'm sure that some end users will use their own scripts to start a hadoop cluster, this change may break their scripts. For HBase, we clearly mention that for a major release, we are allowed to do breaking changes and we will document this out. But hadoop is another story, it is more widely used than HBase so I'm not sure if this is the best choice... And on testing, maybe a possible way to reduce the complexity, is to introduce a common module with a test-jar contains a log4j2.xml, which will be used for all other moduels to do logging when running UTs, as most of the current log4j.properties files under src/test/resources are almost the same. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16377) Moving logging APIs over to slf4j
[
https://issues.apache.org/jira/browse/HADOOP-16377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17349581#comment-17349581
]
Duo Zhang commented on HADOOP-16377:
I think this issue should be done prior to HADOOP-16206.
It is not easy to purge all the log4j dependencies, as slf4j does not provide
all the necessary abilities. For example, you must touch the actual logging
framework to change the log level, you can not do it through slf4j.
What I have done in HBase is to introduce a hbase-logging module, keep all the
direct reference to log4j only in this module, but set log4j dependencies as
provided in the pom of this module. And then actually pull log4j in when
running UTs, and also in the finally assembly stage.
> Moving logging APIs over to slf4j
> -
>
> Key: HADOOP-16377
> URL: https://issues.apache.org/jira/browse/HADOOP-16377
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: HADOOP-16357-002.patch, HADOOP-16377-001.patch,
> HADOOP-16377-003.patch, HADOOP-16377-004.patch, HADOOP-16377-005.patch,
> HADOOP-16377-006.patch, HADOOP-16377-007.patch, HADOOP-16377-008.patch,
> HADOOP-16377-009.patch
>
>
> As of today, there are still 50 references to log4j1
> {code}
> $ grep -r "import org.apache.commons.logging.Log;" . |wc - l
> 50
> {code}
> To achieve the goal of HADOOP-12956/HADOOP-16206, we should invest time to
> move them to slf4j
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17349575#comment-17349575 ] Duo Zhang commented on HADOOP-16206: So any suggestions here? Should we do the same trick in HBase, or someone could find a more compatible way? Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[
https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17332431#comment-17332431
]
Duo Zhang commented on HADOOP-16206:
The converted log4j2.xml has some problems. The biggest problem is that, there
is no direct mapping for the configuration style
'log4j.rootLogger=INFO,console'.
In log4j2, you have to set level and appender separately.
In HBase, we keep the config in hbase-env.sh the same, but in bin/hbase, we
split the property and pass level and appender with difference system
properties, i.e, two '-Dxxx', to the actual java process.
This is an example:
https://github.com/apache/hbase/blob/a4d954e60645f2c81d53c2081d90ea9c175314dc/conf/log4j2.xml#L76
{code}
{code}
This is where I split the config:
https://github.com/apache/hbase/blob/a4d954e60645f2c81d53c2081d90ea9c175314dc/bin/hbase#L793
{code}
# log4j2 does not support setting log level and appender at once, so we need to
split HBASE_ROOT_LOGGER
HBASE_ROOT_LOGGER=${HBASE_ROOT_LOGGER:-INFO,console}
array=(${HBASE_ROOT_LOGGER//,/ })
HBASE_OPTS="$HBASE_OPTS -Dhbase.root.logger.level=${array[0]}"
HBASE_OPTS="$HBASE_OPTS -Dhbase.root.logger.appender=${array[1]}"
{code}
In hadoop, we also need to find a way to deal with this.
> Migrate from Log4j1 to Log4j2
> -
>
> Key: HADOOP-16206
> URL: https://issues.apache.org/jira/browse/HADOOP-16206
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HADOOP-16206-wip.001.patch
>
>
> This sub-task is to remove log4j1 dependency and add log4j2 dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331570#comment-17331570 ] Duo Zhang commented on HADOOP-16206: Implemented a simple tool to convert log4j1 properties to log4j2 xml, based on Log4j1ConfigurationConverter. https://github.com/Apache9/TempCode/blob/master/src/main/java/com/github/apache9/log4j2/HadoopLog4j1ConfigurationConverter.java > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17327051#comment-17327051 ] Duo Zhang commented on HADOOP-16206: Oh, good. Let me check the tool. Thanks for the pointer! > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326619#comment-17326619 ] Duo Zhang commented on HADOOP-16206: Some updates here, I changed the dependencies, to remove log4j1 dependency and add log4j2 dependency, tried best to follow the old dependency scope(although some of dependency scopes are wrong I believe). Obviously it will cause compile error as we depend on log4j classes directly in our code base. Will fix them later. Another big project is to convert log4j.properties to log4j2.properties, there are a bunch of these config files in our code base, I tried to modify them manually but soon gave up, as it is impossible to make all the copy paste without mistakes... I need to write a simple helper tool to automatically generate the log4j2.properties based on the log4j.properties. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17525) Support log4j2 API in GenericTestUtils.setLogLevel
[ https://issues.apache.org/jira/browse/HADOOP-17525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322265#comment-17322265 ] Duo Zhang commented on HADOOP-17525: I think this shouldbe be part of HADOOP-16206? We need to migrate to log4j2 first before adding support for log4j2 in GenericTestUtils.setLogLevel... > Support log4j2 API in GenericTestUtils.setLogLevel > -- > > Key: HADOOP-17525 > URL: https://issues.apache.org/jira/browse/HADOOP-17525 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > > GenericTestUtils.setLogLevel depends on Log4J 1.x API, should be updated to > use Log4J 2.x API. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17524) Remove EventCounter
[ https://issues.apache.org/jira/browse/HADOOP-17524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17320200#comment-17320200 ] Duo Zhang commented on HADOOP-17524: I'm OK with removing it. > Remove EventCounter > --- > > Key: HADOOP-17524 > URL: https://issues.apache.org/jira/browse/HADOOP-17524 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Priority: Major > > EventCount is using Log4J 1.x API. We need to remove it to drop Log4J 1.x. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17320152#comment-17320152 ] Duo Zhang commented on HADOOP-16206: Sounds good to me. So [~ahussein] you would like to provide a patch? If not I could also give a try. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[
https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17319437#comment-17319437
]
Duo Zhang commented on HADOOP-16206:
{quote}
Is it possible to break this up? I.e. Hadoop common, HDFS, and MR. Easier to
review/commit
{quote}
Maybe we could migrate to log4j-1.2-api first, and then move up to log4j2
directly module by module? What do you think?
> Migrate from Log4j1 to Log4j2
> -
>
> Key: HADOOP-16206
> URL: https://issues.apache.org/jira/browse/HADOOP-16206
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-16206-wip.001.patch
>
>
> This sub-task is to remove log4j1 dependency and add log4j2 dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314385#comment-17314385 ] Duo Zhang commented on HADOOP-16206: [~ahussein] So you suggest we do a benchmark first to get a clear evidence of performance gains in log4j2, and then start our work? Or you mean we could just start the work as you think we'd better fully move to log4j2? Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17526) HttpRequestLog should be configured via Configuration instead of log4j.properties
[ https://issues.apache.org/jira/browse/HADOOP-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17305445#comment-17305445 ] Duo Zhang commented on HADOOP-17526: Since hadoop is still on jetty 9.3, we could make use of Slf4jRequestLog in HttpRequestLog to remove the dependency on log4j directly. For jetty 9.4, the pattern has been changed to make use of CustomRequestLog and Slf4jRequestLogWriter. > HttpRequestLog should be configured via Configuration instead of > log4j.properties > - > > Key: HADOOP-17526 > URL: https://issues.apache.org/jira/browse/HADOOP-17526 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Priority: Major > > Now the log file name and retain days in HttpRequestLog is configured from > log4j.properties. It can be configured from Configuration instead to drop the > usage of log4j API. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17525) Remove GenericTestUtils.setLogLevel
[ https://issues.apache.org/jira/browse/HADOOP-17525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17305442#comment-17305442 ] Duo Zhang commented on HADOOP-17525: Yes, in HBase we also have some tests which depends on log4j2 directly as we want to capture the log output. Just need to make sure that we do not expose these classes through the test utils, otherwise it may mess up the dependencies for downstream users. > Remove GenericTestUtils.setLogLevel > --- > > Key: HADOOP-17525 > URL: https://issues.apache.org/jira/browse/HADOOP-17525 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Priority: Major > > GenericTestUtils.setLogLevel depends on Log4J 1.x API, and it should be > removed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17305441#comment-17305441 ] Duo Zhang commented on HADOOP-16206: HBase has already migrated from log4j1 to log4j2 on master branch, and expect to ship it with 3.0.0 release. The most painful thing when migrating is the log4j1 dependency of hadoop if we want to start mini cluster for testing. Even the log4j-1.2-api can not solve all the problem as we extend the log4j1 FileAppender for implementing special appender for container... I would offer my help here but I'm not sure what is the current direction. Do we want to replace all the old log4j1 config files with log4j2 config files, or we just want to use the log4j-1.2-api to switch the logging framework to log4j2 but still retain the old config files? Or we have other plans here? Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16610) Upgrade to yetus 0.11.1 and use emoji vote on github pre commit
[ https://issues.apache.org/jira/browse/HADOOP-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16610: --- Summary: Upgrade to yetus 0.11.1 and use emoji vote on github pre commit (was: Upgrade to yetus 0.11.0 and use emoji vote on github pre commit) > Upgrade to yetus 0.11.1 and use emoji vote on github pre commit > --- > > Key: HADOOP-16610 > URL: https://issues.apache.org/jira/browse/HADOOP-16610 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16610) Upgrade to yetus 0.11.1 and use emoji vote on github pre commit
[ https://issues.apache.org/jira/browse/HADOOP-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16610: --- Component/s: build > Upgrade to yetus 0.11.1 and use emoji vote on github pre commit > --- > > Key: HADOOP-16610 > URL: https://issues.apache.org/jira/browse/HADOOP-16610 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16664) Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16664: --- Attachment: HADOOP-16664-branch-3.2.patch > Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > - > > Key: HADOOP-16664 > URL: https://issues.apache.org/jira/browse/HADOOP-16664 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16664-branch-2.patch, > HADOOP-16664-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16664) Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16664: --- Attachment: (was: HADOOP-16664-branch-3.2.patch) > Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > - > > Key: HADOOP-16664 > URL: https://issues.apache.org/jira/browse/HADOOP-16664 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16664-branch-2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16664) Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16664: --- Status: Patch Available (was: In Progress) > Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > - > > Key: HADOOP-16664 > URL: https://issues.apache.org/jira/browse/HADOOP-16664 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16664-branch-2.patch, > HADOOP-16664-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16664) Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16664: --- Attachment: HADOOP-16664-branch-2.patch > Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > - > > Key: HADOOP-16664 > URL: https://issues.apache.org/jira/browse/HADOOP-16664 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16664-branch-2.patch, > HADOOP-16664-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16664) Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16664: --- Attachment: HADOOP-16664-branch-3.2.patch > Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > - > > Key: HADOOP-16664 > URL: https://issues.apache.org/jira/browse/HADOOP-16664 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16664-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16663) Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16663: --- Attachment: HADOOP-16663-branch-2.patch > Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf > classes" to all active branches > > > Key: HADOOP-16663 > URL: https://issues.apache.org/jira/browse/HADOOP-16663 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16663-branch-2.patch, > HADOOP-16663-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16663) Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16663: --- Attachment: (was: HADOOP-16663-branch-2.patch) > Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf > classes" to all active branches > > > Key: HADOOP-16663 > URL: https://issues.apache.org/jira/browse/HADOOP-16663 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16663-branch-2.patch, > HADOOP-16663-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[
https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16964636#comment-16964636
]
Duo Zhang commented on HADOOP-16598:
[~stack] sir, seems you pushed wrong patch for branch-2 and branch-2.9? For
branch-2.x we still need to support jdk7 so we have to use a special guava
version for the new plugin, but seems the piece of code get lost in the current
code base.
{noformat}
+
+
+
+ com.google.guava
+ guava
+ 20.0
+
+
{noformat}
> Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate
> protobuf classes" to all active branches
> ---
>
> Key: HADOOP-16598
> URL: https://issues.apache.org/jira/browse/HADOOP-16598
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.4, 2.9.3, 3.1.4, 3.2.2, 2.10.1, 2.11.0
>
> Attachments: HADOOP-16598-branch-2-v1.patch,
> HADOOP-16598-branch-2-v1.patch, HADOOP-16598-branch-2-v2.patch,
> HADOOP-16598-branch-2.9-v1.patch, HADOOP-16598-branch-2.9-v1.patch,
> HADOOP-16598-branch-2.9.patch, HADOOP-16598-branch-2.patch,
> HADOOP-16598-branch-3.1.patch, HADOOP-16598-branch-3.2.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16663) Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16663: --- Attachment: HADOOP-16663-branch-2.patch > Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf > classes" to all active branches > > > Key: HADOOP-16663 > URL: https://issues.apache.org/jira/browse/HADOOP-16663 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16663-branch-2.patch, > HADOOP-16663-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16663) Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16663: --- Attachment: HADOOP-16663-branch-3.2.patch > Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf > classes" to all active branches > > > Key: HADOOP-16663 > URL: https://issues.apache.org/jira/browse/HADOOP-16663 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16663-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16663) Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16663: --- Attachment: (was: HADOOP-16663-branch-3.2.patch) > Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf > classes" to all active branches > > > Key: HADOOP-16663 > URL: https://issues.apache.org/jira/browse/HADOOP-16663 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16663) Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16663: --- Status: Patch Available (was: In Progress) > Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf > classes" to all active branches > > > Key: HADOOP-16663 > URL: https://issues.apache.org/jira/browse/HADOOP-16663 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16663-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16663) Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16663: --- Attachment: HADOOP-16663-branch-3.2.patch > Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf > classes" to all active branches > > > Key: HADOOP-16663 > URL: https://issues.apache.org/jira/browse/HADOOP-16663 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16663-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16664) Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HADOOP-16664: -- Assignee: Duo Zhang > Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > - > > Key: HADOOP-16664 > URL: https://issues.apache.org/jira/browse/HADOOP-16664 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-16664) Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16664 started by Duo Zhang. -- > Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > - > > Key: HADOOP-16664 > URL: https://issues.apache.org/jira/browse/HADOOP-16664 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16663) Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HADOOP-16663: -- Assignee: Duo Zhang > Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf > classes" to all active branches > > > Key: HADOOP-16663 > URL: https://issues.apache.org/jira/browse/HADOOP-16663 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-16663) Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16663 started by Duo Zhang. -- > Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf > classes" to all active branches > > > Key: HADOOP-16663 > URL: https://issues.apache.org/jira/browse/HADOOP-16663 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16664) Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes" to all active branches
Duo Zhang created HADOOP-16664: -- Summary: Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes" to all active branches Key: HADOOP-16664 URL: https://issues.apache.org/jira/browse/HADOOP-16664 Project: Hadoop Common Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16663) Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16663: --- Summary: Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf classes" to all active branches (was: Backport ) > Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf > classes" to all active branches > > > Key: HADOOP-16663 > URL: https://issues.apache.org/jira/browse/HADOOP-16663 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16663) Backport
Duo Zhang created HADOOP-16663: -- Summary: Backport Key: HADOOP-16663 URL: https://issues.apache.org/jira/browse/HADOOP-16663 Project: Hadoop Common Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955131#comment-16955131 ] Duo Zhang commented on HADOOP-16598: This is only for hadoop-common and hadoop-hdfs, hadoop-yarn will be another backport. The reason for this is that, if we modify all the poms at once, the pre commit will always timeout since we have to test too many modules... > Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > --- > > Key: HADOOP-16598 > URL: https://issues.apache.org/jira/browse/HADOOP-16598 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16598-branch-2-v1.patch, > HADOOP-16598-branch-2-v1.patch, HADOOP-16598-branch-2-v2.patch, > HADOOP-16598-branch-2.9-v1.patch, HADOOP-16598-branch-2.9-v1.patch, > HADOOP-16598-branch-2.9.patch, HADOOP-16598-branch-2.patch, > HADOOP-16598-branch-3.1.patch, HADOOP-16598-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16595) [pb-upgrade] Create hadoop-thirdparty artifact to have shaded protobuf
[ https://issues.apache.org/jira/browse/HADOOP-16595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946792#comment-16946792 ] Duo Zhang commented on HADOOP-16595: Any updates here? [~vinayakumarb] Thanks. > [pb-upgrade] Create hadoop-thirdparty artifact to have shaded protobuf > -- > > Key: HADOOP-16595 > URL: https://issues.apache.org/jira/browse/HADOOP-16595 > Project: Hadoop Common > Issue Type: Sub-task > Components: hadoop-thirdparty >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Major > > Create a separate repo "hadoop-thirdparty" to have shaded dependencies. > starting with protobuf-java:3.7.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944253#comment-16944253 ] Duo Zhang commented on HADOOP-16598: Uploaded the rebased patch for branch-2. Patches for branch-3.2, branch-3.1 and branch-2.9 are still fine. > Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > --- > > Key: HADOOP-16598 > URL: https://issues.apache.org/jira/browse/HADOOP-16598 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16598-branch-2-v1.patch, > HADOOP-16598-branch-2-v1.patch, HADOOP-16598-branch-2-v2.patch, > HADOOP-16598-branch-2.9-v1.patch, HADOOP-16598-branch-2.9-v1.patch, > HADOOP-16598-branch-2.9.patch, HADOOP-16598-branch-2.patch, > HADOOP-16598-branch-3.1.patch, HADOOP-16598-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16598: --- Attachment: HADOOP-16598-branch-2-v2.patch > Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > --- > > Key: HADOOP-16598 > URL: https://issues.apache.org/jira/browse/HADOOP-16598 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16598-branch-2-v1.patch, > HADOOP-16598-branch-2-v1.patch, HADOOP-16598-branch-2-v2.patch, > HADOOP-16598-branch-2.9-v1.patch, HADOOP-16598-branch-2.9-v1.patch, > HADOOP-16598-branch-2.9.patch, HADOOP-16598-branch-2.patch, > HADOOP-16598-branch-3.1.patch, HADOOP-16598-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944249#comment-16944249 ] Duo Zhang commented on HADOOP-16598: Let me rebase... > Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > --- > > Key: HADOOP-16598 > URL: https://issues.apache.org/jira/browse/HADOOP-16598 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16598-branch-2-v1.patch, > HADOOP-16598-branch-2-v1.patch, HADOOP-16598-branch-2.9-v1.patch, > HADOOP-16598-branch-2.9-v1.patch, HADOOP-16598-branch-2.9.patch, > HADOOP-16598-branch-2.patch, HADOOP-16598-branch-3.1.patch, > HADOOP-16598-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940801#comment-16940801 ] Duo Zhang commented on HADOOP-13363: But later we will shade the protobuf and then there is no way to just use -Dprotobuf.version to change the version... If there is no strong reason to stay on 3.7.x, I suggest we just upgrade to the newest one when creating hadoop-thirdparty? It is 3.9.2 now... Thanks. > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Anu Engineer >Assignee: Vinayakumar B >Priority: Major > Labels: security > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16600) StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
[
https://issues.apache.org/jira/browse/HADOOP-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940218#comment-16940218
]
Duo Zhang commented on HADOOP-16600:
Thanks [~ste...@apache.org]. Please help committing the patch to branch-3.1?
Thanks.
> StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
> -
>
> Key: HADOOP-16600
> URL: https://issues.apache.org/jira/browse/HADOOP-16600
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1, 3.1.2
>Reporter: Lisheng Sun
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.1.4
>
> Attachments: HADOOP-16600-branch-3.1-v1.patch,
> HADOOP-16600.branch-3.1.v1.patch
>
>
> details see HADOOP-15398
> Problem: hadoop trunk compilation is failing
> Root Cause:
> compilation error is coming from
> org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase. Compilation error is
> "The method getArgumentAt(int, Class) is undefined for the
> type InvocationOnMock".
> StagingTestBase is using getArgumentAt(int, Class) method
> which is not available in mockito-all 1.8.5 version. getArgumentAt(int,
> Class) method is available only from version 2.0.0-beta
> as follow code:
> {code:java}
> InitiateMultipartUploadRequest req = invocation.getArgumentAt(
> 0, InitiateMultipartUploadRequest.class);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16600) StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
[
https://issues.apache.org/jira/browse/HADOOP-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang reassigned HADOOP-16600:
--
Assignee: Duo Zhang (was: Lisheng Sun)
> StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
> -
>
> Key: HADOOP-16600
> URL: https://issues.apache.org/jira/browse/HADOOP-16600
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1, 3.1.2
>Reporter: Lisheng Sun
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.1.4
>
> Attachments: HADOOP-16600-branch-3.1-v1.patch,
> HADOOP-16600.branch-3.1.v1.patch
>
>
> details see HADOOP-15398
> Problem: hadoop trunk compilation is failing
> Root Cause:
> compilation error is coming from
> org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase. Compilation error is
> "The method getArgumentAt(int, Class) is undefined for the
> type InvocationOnMock".
> StagingTestBase is using getArgumentAt(int, Class) method
> which is not available in mockito-all 1.8.5 version. getArgumentAt(int,
> Class) method is available only from version 2.0.0-beta
> as follow code:
> {code:java}
> InitiateMultipartUploadRequest req = invocation.getArgumentAt(
> 0, InitiateMultipartUploadRequest.class);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[
https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940200#comment-16940200
]
Duo Zhang commented on HADOOP-16598:
{quote}
this isn't going to change protobuf versions is it? It's just moving to the
jar-based protoc?
{quote}
No, it will not change the protobuf version, it is still 2.5.0. Just use a new
protobuf maven plugin to resolve the protoc artifact automatically so users do
not need to install protoc manually.
> Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate
> protobuf classes" to all active branches
> ---
>
> Key: HADOOP-16598
> URL: https://issues.apache.org/jira/browse/HADOOP-16598
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HADOOP-16598-branch-2-v1.patch,
> HADOOP-16598-branch-2.9-v1.patch, HADOOP-16598-branch-2.9.patch,
> HADOOP-16598-branch-2.patch, HADOOP-16598-branch-3.1.patch,
> HADOOP-16598-branch-3.2.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16600) StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
[
https://issues.apache.org/jira/browse/HADOOP-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HADOOP-16600:
---
Attachment: HADOOP-16600-branch-3.1-v1.patch
> StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
> -
>
> Key: HADOOP-16600
> URL: https://issues.apache.org/jira/browse/HADOOP-16600
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1, 3.1.2
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 3.1.4
>
> Attachments: HADOOP-16600-branch-3.1-v1.patch,
> HADOOP-16600.branch-3.1.v1.patch
>
>
> details see HADOOP-15398
> Problem: hadoop trunk compilation is failing
> Root Cause:
> compilation error is coming from
> org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase. Compilation error is
> "The method getArgumentAt(int, Class) is undefined for the
> type InvocationOnMock".
> StagingTestBase is using getArgumentAt(int, Class) method
> which is not available in mockito-all 1.8.5 version. getArgumentAt(int,
> Class) method is available only from version 2.0.0-beta
> as follow code:
> {code:java}
> InitiateMultipartUploadRequest req = invocation.getArgumentAt(
> 0, InitiateMultipartUploadRequest.class);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16600) StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
[
https://issues.apache.org/jira/browse/HADOOP-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HADOOP-16600:
---
Fix Version/s: 3.1.4
> StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
> -
>
> Key: HADOOP-16600
> URL: https://issues.apache.org/jira/browse/HADOOP-16600
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1, 3.1.2
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 3.1.4
>
> Attachments: HADOOP-16600.branch-3.1.v1.patch
>
>
> details see HADOOP-15398
> Problem: hadoop trunk compilation is failing
> Root Cause:
> compilation error is coming from
> org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase. Compilation error is
> "The method getArgumentAt(int, Class) is undefined for the
> type InvocationOnMock".
> StagingTestBase is using getArgumentAt(int, Class) method
> which is not available in mockito-all 1.8.5 version. getArgumentAt(int,
> Class) method is available only from version 2.0.0-beta
> as follow code:
> {code:java}
> InitiateMultipartUploadRequest req = invocation.getArgumentAt(
> 0, InitiateMultipartUploadRequest.class);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16598: --- Attachment: (was: HADOOP-16598-branch-2-v1.patch) > Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > --- > > Key: HADOOP-16598 > URL: https://issues.apache.org/jira/browse/HADOOP-16598 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16598-branch-2-v1.patch, > HADOOP-16598-branch-2.9-v1.patch, HADOOP-16598-branch-2.9.patch, > HADOOP-16598-branch-2.patch, HADOOP-16598-branch-3.1.patch, > HADOOP-16598-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16598: --- Attachment: HADOOP-16598-branch-2-v1.patch > Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > --- > > Key: HADOOP-16598 > URL: https://issues.apache.org/jira/browse/HADOOP-16598 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16598-branch-2-v1.patch, > HADOOP-16598-branch-2.9-v1.patch, HADOOP-16598-branch-2.9.patch, > HADOOP-16598-branch-2.patch, HADOOP-16598-branch-3.1.patch, > HADOOP-16598-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16600) StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
[
https://issues.apache.org/jira/browse/HADOOP-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939927#comment-16939927
]
Duo Zhang commented on HADOOP-16600:
As here we only want to solve the compile error, I think we could only backport
the mockito related part of HADOOP-14918 to branch-3.1? Let me prepare a patch.
> StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
> -
>
> Key: HADOOP-16600
> URL: https://issues.apache.org/jira/browse/HADOOP-16600
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1, 3.1.2
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16600.branch-3.1.v1.patch
>
>
> details see HADOOP-15398
> Problem: hadoop trunk compilation is failing
> Root Cause:
> compilation error is coming from
> org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase. Compilation error is
> "The method getArgumentAt(int, Class) is undefined for the
> type InvocationOnMock".
> StagingTestBase is using getArgumentAt(int, Class) method
> which is not available in mockito-all 1.8.5 version. getArgumentAt(int,
> Class) method is available only from version 2.0.0-beta
> as follow code:
> {code:java}
> InitiateMultipartUploadRequest req = invocation.getArgumentAt(
> 0, InitiateMultipartUploadRequest.class);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16598: --- Attachment: HADOOP-16598-branch-2.9-v1.patch > Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > --- > > Key: HADOOP-16598 > URL: https://issues.apache.org/jira/browse/HADOOP-16598 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16598-branch-2-v1.patch, > HADOOP-16598-branch-2.9-v1.patch, HADOOP-16598-branch-2.9.patch, > HADOOP-16598-branch-2.patch, HADOOP-16598-branch-3.1.patch, > HADOOP-16598-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16598: --- Attachment: HADOOP-16598-branch-2-v1.patch > Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > --- > > Key: HADOOP-16598 > URL: https://issues.apache.org/jira/browse/HADOOP-16598 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16598-branch-2-v1.patch, > HADOOP-16598-branch-2.9.patch, HADOOP-16598-branch-2.patch, > HADOOP-16598-branch-3.1.patch, HADOOP-16598-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16598: --- Attachment: (was: HADOO-16598-branch-2-v1.patch) > Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > --- > > Key: HADOOP-16598 > URL: https://issues.apache.org/jira/browse/HADOOP-16598 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16598-branch-2.9.patch, > HADOOP-16598-branch-2.patch, HADOOP-16598-branch-3.1.patch, > HADOOP-16598-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16598: --- Attachment: HADOO-16598-branch-2-v1.patch > Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > --- > > Key: HADOOP-16598 > URL: https://issues.apache.org/jira/browse/HADOOP-16598 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOO-16598-branch-2-v1.patch, > HADOOP-16598-branch-2.9.patch, HADOOP-16598-branch-2.patch, > HADOOP-16598-branch-3.1.patch, HADOOP-16598-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939854#comment-16939854 ] Duo Zhang commented on HADOOP-13363: I've asked this question many times in many issues but never get an answer... So why we use protobuf 3.7.1 instead of 3.9.1? > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Anu Engineer >Assignee: Vinayakumar B >Priority: Major > Labels: security > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[
https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939852#comment-16939852
]
Duo Zhang commented on HADOOP-16598:
{noformat}
[INFO] --- protobuf-maven-plugin:0.5.1:compile (src-compile-protoc) @
hadoop-common ---
[WARNING] Error injecting: org.xolstice.maven.plugin.protobuf.ProtocCompileMojo
java.lang.UnsupportedClassVersionError: com/google/common/collect/ImmutableSet
: Unsupported major.minor version 52.0
{noformat}
Seems the plugin is jdk8 only? Let me check.
> Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate
> protobuf classes" to all active branches
> ---
>
> Key: HADOOP-16598
> URL: https://issues.apache.org/jira/browse/HADOOP-16598
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HADOOP-16598-branch-2.9.patch,
> HADOOP-16598-branch-2.patch, HADOOP-16598-branch-3.1.patch,
> HADOOP-16598-branch-3.2.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16593) [YARN] Polish the protobuf plugin for hadoop-yarn-csi
[ https://issues.apache.org/jira/browse/HADOOP-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16593: --- Component/s: build > [YARN] Polish the protobuf plugin for hadoop-yarn-csi > - > > Key: HADOOP-16593 > URL: https://issues.apache.org/jira/browse/HADOOP-16593 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > As discussed here: > https://github.com/apache/hadoop/pull/1496#discussion_r326931072 > We should align the execution id in the parent pom. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16593) [YARN] Polish the protobuf plugin for hadoop-yarn-csi
[ https://issues.apache.org/jira/browse/HADOOP-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HADOOP-16593: -- Assignee: Duo Zhang > [YARN] Polish the protobuf plugin for hadoop-yarn-csi > - > > Key: HADOOP-16593 > URL: https://issues.apache.org/jira/browse/HADOOP-16593 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > As discussed here: > https://github.com/apache/hadoop/pull/1496#discussion_r326931072 > We should align the execution id in the parent pom. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-16593) [YARN] Polish the protobuf plugin for hadoop-yarn-csi
[ https://issues.apache.org/jira/browse/HADOOP-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16593 started by Duo Zhang. -- > [YARN] Polish the protobuf plugin for hadoop-yarn-csi > - > > Key: HADOOP-16593 > URL: https://issues.apache.org/jira/browse/HADOOP-16593 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > As discussed here: > https://github.com/apache/hadoop/pull/1496#discussion_r326931072 > We should align the execution id in the parent pom. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939080#comment-16939080 ] Duo Zhang commented on HADOOP-16598: I assume branch-3.0 and branch-2.8 will be dead soon? If not, will also upload patches for them. [~djp]. > Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > --- > > Key: HADOOP-16598 > URL: https://issues.apache.org/jira/browse/HADOOP-16598 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16598-branch-2.9.patch, > HADOOP-16598-branch-2.patch, HADOOP-16598-branch-3.1.patch, > HADOOP-16598-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16598: --- Attachment: HADOOP-16598-branch-2.9.patch > Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > --- > > Key: HADOOP-16598 > URL: https://issues.apache.org/jira/browse/HADOOP-16598 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16598-branch-2.9.patch, > HADOOP-16598-branch-2.patch, HADOOP-16598-branch-3.1.patch, > HADOOP-16598-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16598: --- Attachment: HADOOP-16598-branch-2.patch > Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > --- > > Key: HADOOP-16598 > URL: https://issues.apache.org/jira/browse/HADOOP-16598 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16598-branch-2.patch, > HADOOP-16598-branch-3.1.patch, HADOOP-16598-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HADOOP-16598: --- Attachment: HADOOP-16598-branch-3.1.patch > Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > --- > > Key: HADOOP-16598 > URL: https://issues.apache.org/jira/browse/HADOOP-16598 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16598-branch-3.1.patch, > HADOOP-16598-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16600) StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
[
https://issues.apache.org/jira/browse/HADOOP-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939059#comment-16939059
]
Duo Zhang commented on HADOOP-16600:
{quote}
maybe there's been some classpath ordering thing which has hid this for me?
{quote}
Oh, missed the mockito-core dependency... I thought since we explicitly declare
mockito-all 1.8.5 in our own pom, it should supercede all other transitive
dependencies. But we may get newer mockito-core dependency from the transitive
dependencies. So if we load the newer mockito-core first we will be safe,
otherwise there will be a compile error...
Anyway, I think we'd better explicitly depend on 1.10.19 so we can get a stable
result? What do you think [~ste...@apache.org].
Thanks.
> StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
> -
>
> Key: HADOOP-16600
> URL: https://issues.apache.org/jira/browse/HADOOP-16600
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1, 3.1.2
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16600.branch-3.1.v1.patch
>
>
> details see HADOOP-15398
> Problem: hadoop trunk compilation is failing
> Root Cause:
> compilation error is coming from
> org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase. Compilation error is
> "The method getArgumentAt(int, Class) is undefined for the
> type InvocationOnMock".
> StagingTestBase is using getArgumentAt(int, Class) method
> which is not available in mockito-all 1.8.5 version. getArgumentAt(int,
> Class) method is available only from version 2.0.0-beta
> as follow code:
> {code:java}
> InitiateMultipartUploadRequest req = invocation.getArgumentAt(
> 0, InitiateMultipartUploadRequest.class);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16600) StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
[
https://issues.apache.org/jira/browse/HADOOP-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938615#comment-16938615
]
Duo Zhang commented on HADOOP-16600:
And another thing is that in HADOOP-14918, we introduced a getArgumentAt which
just ignores the Class parameter. This patch is only committed to brancn-3.2+,
so maybe this is the reason why only branch-3.1 is effected?
> StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
> -
>
> Key: HADOOP-16600
> URL: https://issues.apache.org/jira/browse/HADOOP-16600
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1, 3.1.2
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16600.branch-3.1.v1.patch
>
>
> details see HADOOP-15398
> Problem: hadoop trunk compilation is failing
> Root Cause:
> compilation error is coming from
> org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase. Compilation error is
> "The method getArgumentAt(int, Class) is undefined for the
> type InvocationOnMock".
> StagingTestBase is using getArgumentAt(int, Class) method
> which is not available in mockito-all 1.8.5 version. getArgumentAt(int,
> Class) method is available only from version 2.0.0-beta
> as follow code:
> {code:java}
> InitiateMultipartUploadRequest req = invocation.getArgumentAt(
> 0, InitiateMultipartUploadRequest.class);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16600) StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
[
https://issues.apache.org/jira/browse/HADOOP-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938606#comment-16938606
]
Duo Zhang commented on HADOOP-16600:
Tried to reset to this commit
commit de8b6ca5ef8614de6d6277b7617e27c788b0555c
Author: Steve Loughran
Date: Wed Nov 22 15:28:12 2017 +
HADOOP-13786 Add S3A committer for zero-rename commits to S3 endpoints.
Contributed by Steve Loughran and Ryan Blue.
But I still can not compile the code, this time it is due to an unresolved
dependency. And I tried 'mvn dependency:tree', th mockito-all dependency is
still 1.8.5. Strange. Why we do not find this compile error in the past?
> StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
> -
>
> Key: HADOOP-16600
> URL: https://issues.apache.org/jira/browse/HADOOP-16600
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1, 3.1.2
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16600.branch-3.1.v1.patch
>
>
> details see HADOOP-15398
> Problem: hadoop trunk compilation is failing
> Root Cause:
> compilation error is coming from
> org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase. Compilation error is
> "The method getArgumentAt(int, Class) is undefined for the
> type InvocationOnMock".
> StagingTestBase is using getArgumentAt(int, Class) method
> which is not available in mockito-all 1.8.5 version. getArgumentAt(int,
> Class) method is available only from version 2.0.0-beta
> as follow code:
> {code:java}
> InitiateMultipartUploadRequest req = invocation.getArgumentAt(
> 0, InitiateMultipartUploadRequest.class);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16600) StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
[
https://issues.apache.org/jira/browse/HADOOP-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938551#comment-16938551
]
Duo Zhang commented on HADOOP-16600:
[~ste...@apache.org] The code is introduced by HADOOP-13786. Maybe the problem
is that, when committing HADOOP-13786, we have a transitive dependency of
mockito which had a higher version and later it was removed?
> StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
> -
>
> Key: HADOOP-16600
> URL: https://issues.apache.org/jira/browse/HADOOP-16600
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1, 3.1.2
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16600.branch-3.1.v1.patch
>
>
> details see HADOOP-15398
> Problem: hadoop trunk compilation is failing
> Root Cause:
> compilation error is coming from
> org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase. Compilation error is
> "The method getArgumentAt(int, Class) is undefined for the
> type InvocationOnMock".
> StagingTestBase is using getArgumentAt(int, Class) method
> which is not available in mockito-all 1.8.5 version. getArgumentAt(int,
> Class) method is available only from version 2.0.0-beta
> as follow code:
> {code:java}
> InitiateMultipartUploadRequest req = invocation.getArgumentAt(
> 0, InitiateMultipartUploadRequest.class);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16611) Make test4tests vote for -0 instead of -1
[ https://issues.apache.org/jira/browse/HADOOP-16611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HADOOP-16611. Resolution: Not A Problem > Make test4tests vote for -0 instead of -1 > - > > Key: HADOOP-16611 > URL: https://issues.apache.org/jira/browse/HADOOP-16611 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
[ https://issues.apache.org/jira/browse/HADOOP-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938534#comment-16938534 ] Duo Zhang commented on HADOOP-16598: Will also upload patches for other active branches here, but at least for branch-3.1, it is blocked by HADOOP-16600... > Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate > protobuf classes" to all active branches > --- > > Key: HADOOP-16598 > URL: https://issues.apache.org/jira/browse/HADOOP-16598 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16598-branch-3.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16611) Make test4tests vote for -0 instead of -1
[ https://issues.apache.org/jira/browse/HADOOP-16611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HADOOP-16611: -- Assignee: Duo Zhang > Make test4tests vote for -0 instead of -1 > - > > Key: HADOOP-16611 > URL: https://issues.apache.org/jira/browse/HADOOP-16611 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16610) Upgrade to yetus 0.11.0 and use emoji vote on github pre commit
[ https://issues.apache.org/jira/browse/HADOOP-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HADOOP-16610: -- Assignee: Duo Zhang > Upgrade to yetus 0.11.0 and use emoji vote on github pre commit > --- > > Key: HADOOP-16610 > URL: https://issues.apache.org/jira/browse/HADOOP-16610 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
