[jira] [Commented] (MAPREDUCE-6909) LocalJobRunner fails when run on a node from multiple users
[
https://issues.apache.org/jira/browse/MAPREDUCE-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070979#comment-16070979
]
Hadoop QA commented on MAPREDUCE-6909:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} branch-2 passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} branch-2 passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} branch-2 passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} branch-2 passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
35s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed
with JDK v1.7.0_131. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:5e40efe |
| JIRA Issue | MAPREDUCE-6909 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875280/MAPREDUCE-6909-branch-2.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux cea20173e8c7 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / 94bc5cd |
| Default Java | 1.7.0_131 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_131
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 |
| JDK v1.7.0_131 Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7013/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Consol
[jira] [Updated] (MAPREDUCE-6246) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2
[
https://issues.apache.org/jira/browse/MAPREDUCE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6246:
--
Labels: (was: BB2015-05-RFC)
Status: Patch Available (was: Open)
Thanks [~GergelyNovak]! submit the patch to kick off Jenkins.
> DBOutputFormat.java appending extra semicolon to query which is incompatible
> with DB2
> -
>
> Key: MAPREDUCE-6246
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6246
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
>Affects Versions: 2.4.1
> Environment: OS: RHEL 5.x, RHEL 6.x, SLES 11.x
> Platform: xSeries, pSeries
> Browser: Firefox, IE
> Security Settings: No Security, Flat file, LDAP, PAM
> File System: HDFS, GPFS FPO
>Reporter: ramtin
>Assignee: ramtin
> Attachments: MAPREDUCE-6246.002.patch, MAPREDUCE-6246.003.patch,
> MAPREDUCE-6246.004.patch, MAPREDUCE-6246.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> DBoutputformat is used for writing output of mapreduce jobs to the database
> and when used with db2 jdbc drivers it fails with following error
> com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-104,
> SQLSTATE=42601, SQLERRMC=;;,COUNT) VALUES (?,?);END-OF-STATEMENT,
> DRIVER=4.16.53 at com.ibm.db2.jcc.am.fd.a(fd.java:739) at
> com.ibm.db2.jcc.am.fd.a(fd.java:60) at com.ibm.db2.jcc.am.fd.a(fd.java:127)
> In DBOutputFormat class there is constructQuery method that generates "INSERT
> INTO" statement with semicolon(";") at the end.
> Semicolon is ANSI SQL-92 standard character for a statement terminator but
> this feature is disabled(OFF) as a default settings in IBM DB2.
> Although by using -t we can turn it ON for db2.
> (http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0010410.html?cp=SSEPGG_9.7.0%2F3-6-2-0-2).
> But there are some products that already built on top of this default
> setting (OFF) so by turning ON this feature make them error prone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6909) LocalJobRunner fails when run on a node from multiple users
[ https://issues.apache.org/jira/browse/MAPREDUCE-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6909: -- Target Version/s: 2.9.0, 2.8.2 (was: 2.8.2) Status: Patch Available (was: Open) This patch should be applied to branch-2 and branch-2.8. I'm assuming we're playing the "it's a major release so we are allowed to break stuff" card for the 3.x releases, so this doesn't need to go into trunk. > LocalJobRunner fails when run on a node from multiple users > --- > > Key: MAPREDUCE-6909 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6909 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.8.1 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Attachments: MAPREDUCE-6909-branch-2.001.patch > > > MAPREDUCE-5762 removed mapreduce.jobtracker.staging.root.dir from > mapred-default.xml but the property is still being used by LocalJobRunner and > the code default value does *not* match the value that was removed from > mapred-default.xml. This broke the use case where multiple users are running > local mode jobs on the same node, since they now default to the same > directory in /tmp. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6909) LocalJobRunner fails when run on a node from multiple users
[ https://issues.apache.org/jira/browse/MAPREDUCE-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6909: -- Attachment: MAPREDUCE-6909-branch-2.001.patch No, MAPREDUCE-6441 is fixing a slightly different problem where multiple jobs are running from the same user. This is addressing a problem when two different users on the same node try to run local jobs, not necessarily at the same time. Even after the latest patch for MAPREDUCE-6441 is applied LocalJobRunner is still referencing JTConfig.JT_STAGING_AREA_ROOT, and that will break for multiple users after MAPREDUCE-5762 removed the default value for that property in mapred-default.xml. I went through the list of properties removed by MAPREDUCE-5762, and I created a patch that adds back any property that is being used by the code where the value in mapred-default.xml didn't match the default values in the code. One exception was mapreduce.cluster.temp.dir, but I suspect users are _still_ referencing that value either directly or in their code. > LocalJobRunner fails when run on a node from multiple users > --- > > Key: MAPREDUCE-6909 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6909 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.8.1 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Attachments: MAPREDUCE-6909-branch-2.001.patch > > > MAPREDUCE-5762 removed mapreduce.jobtracker.staging.root.dir from > mapred-default.xml but the property is still being used by LocalJobRunner and > the code default value does *not* match the value that was removed from > mapred-default.xml. This broke the use case where multiple users are running > local mode jobs on the same node, since they now default to the same > directory in /tmp. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6909) LocalJobRunner fails when run on a node from multiple users
[ https://issues.apache.org/jira/browse/MAPREDUCE-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070329#comment-16070329 ] Ray Chiang commented on MAPREDUCE-6909: --- Could this be a duplicate of MAPREDUCE-6441? > LocalJobRunner fails when run on a node from multiple users > --- > > Key: MAPREDUCE-6909 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6909 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.8.1 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > > MAPREDUCE-5762 removed mapreduce.jobtracker.staging.root.dir from > mapred-default.xml but the property is still being used by LocalJobRunner and > the code default value does *not* match the value that was removed from > mapred-default.xml. This broke the use case where multiple users are running > local mode jobs on the same node, since they now default to the same > directory in /tmp. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6909) LocalJobRunner fails when run on a node from multiple users
[
https://issues.apache.org/jira/browse/MAPREDUCE-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070230#comment-16070230
]
Jason Lowe commented on MAPREDUCE-6909:
---
Because the code default value didn't match the value in mapred-default.xml,
removing it while it was still being used became an incompatible code change.
Here's an example stack trace on 2.8 from a user running a local-mode job when
some other user has already done so:
{noformat}
ERROR 6017: JobID: job_local2035229526_0001 Reason: EACCES: Permission denied
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:767)
at
org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:506)
at
org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:487)
at
org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:509)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:619)
at
org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:94)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:97)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:192)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1936)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
at
org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:205)
at java.lang.Thread.run(Thread.java:745)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:288)
{noformat}
> LocalJobRunner fails when run on a node from multiple users
> ---
>
> Key: MAPREDUCE-6909
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6909
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 2.8.1
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Blocker
>
> MAPREDUCE-5762 removed mapreduce.jobtracker.staging.root.dir from
> mapred-default.xml but the property is still being used by LocalJobRunner and
> the code default value does *not* match the value that was removed from
> mapred-default.xml. This broke the use case where multiple users are running
> local mode jobs on the same node, since they now default to the same
> directory in /tmp.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6909) LocalJobRunner fails when run on a node from multiple users
Jason Lowe created MAPREDUCE-6909: - Summary: LocalJobRunner fails when run on a node from multiple users Key: MAPREDUCE-6909 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6909 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.8.1 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker MAPREDUCE-5762 removed mapreduce.jobtracker.staging.root.dir from mapred-default.xml but the property is still being used by LocalJobRunner and the code default value does *not* match the value that was removed from mapred-default.xml. This broke the use case where multiple users are running local mode jobs on the same node, since they now default to the same directory in /tmp. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6246) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2
[
https://issues.apache.org/jira/browse/MAPREDUCE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070027#comment-16070027
]
Gergely Novák commented on MAPREDUCE-6246:
--
I did the rebase, plus added @Test annotations and did some refactor in the
unit tests.
> DBOutputFormat.java appending extra semicolon to query which is incompatible
> with DB2
> -
>
> Key: MAPREDUCE-6246
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6246
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
>Affects Versions: 2.4.1
> Environment: OS: RHEL 5.x, RHEL 6.x, SLES 11.x
> Platform: xSeries, pSeries
> Browser: Firefox, IE
> Security Settings: No Security, Flat file, LDAP, PAM
> File System: HDFS, GPFS FPO
>Reporter: ramtin
>Assignee: ramtin
> Labels: BB2015-05-RFC
> Attachments: MAPREDUCE-6246.002.patch, MAPREDUCE-6246.003.patch,
> MAPREDUCE-6246.004.patch, MAPREDUCE-6246.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> DBoutputformat is used for writing output of mapreduce jobs to the database
> and when used with db2 jdbc drivers it fails with following error
> com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-104,
> SQLSTATE=42601, SQLERRMC=;;,COUNT) VALUES (?,?);END-OF-STATEMENT,
> DRIVER=4.16.53 at com.ibm.db2.jcc.am.fd.a(fd.java:739) at
> com.ibm.db2.jcc.am.fd.a(fd.java:60) at com.ibm.db2.jcc.am.fd.a(fd.java:127)
> In DBOutputFormat class there is constructQuery method that generates "INSERT
> INTO" statement with semicolon(";") at the end.
> Semicolon is ANSI SQL-92 standard character for a statement terminator but
> this feature is disabled(OFF) as a default settings in IBM DB2.
> Although by using -t we can turn it ON for db2.
> (http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0010410.html?cp=SSEPGG_9.7.0%2F3-6-2-0-2).
> But there are some products that already built on top of this default
> setting (OFF) so by turning ON this feature make them error prone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6246) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2
[
https://issues.apache.org/jira/browse/MAPREDUCE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák updated MAPREDUCE-6246:
-
Attachment: MAPREDUCE-6246.004.patch
> DBOutputFormat.java appending extra semicolon to query which is incompatible
> with DB2
> -
>
> Key: MAPREDUCE-6246
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6246
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
>Affects Versions: 2.4.1
> Environment: OS: RHEL 5.x, RHEL 6.x, SLES 11.x
> Platform: xSeries, pSeries
> Browser: Firefox, IE
> Security Settings: No Security, Flat file, LDAP, PAM
> File System: HDFS, GPFS FPO
>Reporter: ramtin
>Assignee: ramtin
> Labels: BB2015-05-RFC
> Attachments: MAPREDUCE-6246.002.patch, MAPREDUCE-6246.003.patch,
> MAPREDUCE-6246.004.patch, MAPREDUCE-6246.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> DBoutputformat is used for writing output of mapreduce jobs to the database
> and when used with db2 jdbc drivers it fails with following error
> com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-104,
> SQLSTATE=42601, SQLERRMC=;;,COUNT) VALUES (?,?);END-OF-STATEMENT,
> DRIVER=4.16.53 at com.ibm.db2.jcc.am.fd.a(fd.java:739) at
> com.ibm.db2.jcc.am.fd.a(fd.java:60) at com.ibm.db2.jcc.am.fd.a(fd.java:127)
> In DBOutputFormat class there is constructQuery method that generates "INSERT
> INTO" statement with semicolon(";") at the end.
> Semicolon is ANSI SQL-92 standard character for a statement terminator but
> this feature is disabled(OFF) as a default settings in IBM DB2.
> Although by using -t we can turn it ON for db2.
> (http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0010410.html?cp=SSEPGG_9.7.0%2F3-6-2-0-2).
> But there are some products that already built on top of this default
> setting (OFF) so by turning ON this feature make them error prone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
