[jira] [Commented] (MAPREDUCE-6316) Task Attempt List entries should link to the task overview
[ https://issues.apache.org/jira/browse/MAPREDUCE-6316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502259#comment-14502259 ] Gera Shegalov commented on MAPREDUCE-6316: -- [~l201514], Can we make sure that the link is disabled when the request is already routed via the task overview page? > Task Attempt List entries should link to the task overview > -- > > Key: MAPREDUCE-6316 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6316 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Siqi Li >Assignee: Siqi Li > Attachments: MAPREDUCE-6316.v1.patch > > > Typical workflow is to click on the list of failed attempts. Then you want to > look at the counters, or the list of attempts of just one task in general. If > each entry task attempt id linked the task id portion of it back to the task, > we would not have to go through the list of tasks to search for the task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6293) uberized job fails with the job classloader enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502223#comment-14502223
]
Gera Shegalov commented on MAPREDUCE-6293:
--
Thanks for v2, [~sjlee0]!
Can we just use implicit {{toString}} ?
{code}
152 // note the current TCCL
153 LOG.info("Context classloader of thread " + eventHandler +
154 ": " + eventHandler.getContextClassLoader());
{code}
Then we would not need to worry whether
eventHandler.setContextClassLoader(null) was ever called.
> uberized job fails with the job classloader enabled
> ---
>
> Key: MAPREDUCE-6293
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6293
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Attachments: MAPREDUCE-6293.001.patch, MAPREDUCE-6293.002.patch
>
>
> An uberized job fails if the job classloader is enabled and the job needs to
> use the thread context classloader to load a class. Some example error in the
> log:
> {quote}
> 2015-03-23 23:28:34,675 INFO [main\]
> org.apache.hadoop.mapreduce.v2.util.MRApps: Creating job classloader
> ...
> 2015-03-23 23:28:42,096 ERROR [uber-SubtaskRunner\]
> cascading.provider.ServiceLoader: unable to find service class:
> cascading.tuple.hadoop.collect.HadoopTupleMapFactory, with exception:
> java.lang.ClassNotFoundException:
> cascading.tuple.hadoop.collect.HadoopTupleMapFactory
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6246) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2
[
https://issues.apache.org/jira/browse/MAPREDUCE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502111#comment-14502111
]
Hadoop QA commented on MAPREDUCE-6246:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726452/MAPREDUCE-6246.002.patch
against trunk revision 5459b24.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5412//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5412//console
This message is automatically generated.
> DBOutputFormat.java appending extra semicolon to query which is incompatible
> with DB2
> -
>
> Key: MAPREDUCE-6246
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6246
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
>Affects Versions: 2.4.1
> Environment: OS: RHEL 5.x, RHEL 6.x, SLES 11.x
> Platform: xSeries, pSeries
> Browser: Firefox, IE
> Security Settings: No Security, Flat file, LDAP, PAM
> File System: HDFS, GPFS FPO
>Reporter: ramtin
>Assignee: ramtin
> Labels: DB2, mapreduce
> Attachments: MAPREDUCE-6246.002.patch, MAPREDUCE-6246.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> DBoutputformat is used for writing output of mapreduce jobs to the database
> and when used with db2 jdbc drivers it fails with following error
> com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-104,
> SQLSTATE=42601, SQLERRMC=;;,COUNT) VALUES (?,?);END-OF-STATEMENT,
> DRIVER=4.16.53 at com.ibm.db2.jcc.am.fd.a(fd.java:739) at
> com.ibm.db2.jcc.am.fd.a(fd.java:60) at com.ibm.db2.jcc.am.fd.a(fd.java:127)
> In DBOutputFormat class there is constructQuery method that generates "INSERT
> INTO" statement with semicolon(";") at the end.
> Semicolon is ANSI SQL-92 standard character for a statement terminator but
> this feature is disabled(OFF) as a default settings in IBM DB2.
> Although by using -t we can turn it ON for db2.
> (http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0010410.html?cp=SSEPGG_9.7.0%2F3-6-2-0-2).
> But there are some products that already built on top of this default
> setting (OFF) so by turning ON this feature make them error prone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6320) Configuration of retrieved Job via Cluster is not properly set-up
[
https://issues.apache.org/jira/browse/MAPREDUCE-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502077#comment-14502077
]
Hadoop QA commented on MAPREDUCE-6320:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726455/MAPREDUCE-6320.002.patch
against trunk revision 5459b24.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core-tests:
org.apache.hadoop.mapreduce.TestCluster
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5413//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5413//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5413//console
This message is automatically generated.
> Configuration of retrieved Job via Cluster is not properly set-up
> -
>
> Key: MAPREDUCE-6320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jens Rabe
>Assignee: Jens Rabe
> Attachments: MAPREDUCE-6320.001.patch, MAPREDUCE-6320.002.patch
>
>
> When getting a Job via the Cluster API, it is not correctly configured.
> To reproduce this:
> # Submit a MR job, and set some arbitrary parameter to its configuration
> {code:java}
> job.getConfiguration().set("foo", "bar");
> job.setJobName("foo-bug-demo");
> {code}
> # Get the job in a client:
> {code:java}
> final Cluster c = new Cluster(conf);
> final JobStatus[] statuses = c.getAllJobStatuses();
> final JobStatus s = ... // get the status for the job named foo-bug-demo
> final Job j = c.getJob(s.getJobId());
> final Configuration conf = job.getConfiguration();
> {code}
> # Get its "foo" entry
> {code:java}
> final String s = conf.get("foo");
> {code}
> # Expected: s is "bar"; But: s is null.
> The reason is that the job's configuration is stored on HDFS (the
> Configuration has a resource with a *hdfs://* URL) and in the *loadResource*
> it is changed to a path on the local file system
> (hdfs://host.domain:port/tmp/hadoop-yarn/... is changed to
> /tmp/hadoop-yarn/...), which does not exist, and thus the configuration is
> not populated.
> The bug happens in the *Cluster* class, where *JobConfs* are created from
> *status.getJobFile()*. A quick fix would be to copy this job file to a
> temporary file in the local file system and populate the JobConf from this
> file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6320) Configuration of retrieved Job via Cluster is not properly set-up
[
https://issues.apache.org/jira/browse/MAPREDUCE-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jens Rabe updated MAPREDUCE-6320:
-
Status: Patch Available (was: In Progress)
I put the test in an extra Maven module to resolve the cyclic dependencies
which caused the build to fail. See [^MAPREDUCE-6320.002.patch]
> Configuration of retrieved Job via Cluster is not properly set-up
> -
>
> Key: MAPREDUCE-6320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jens Rabe
>Assignee: Jens Rabe
> Attachments: MAPREDUCE-6320.001.patch, MAPREDUCE-6320.002.patch
>
>
> When getting a Job via the Cluster API, it is not correctly configured.
> To reproduce this:
> # Submit a MR job, and set some arbitrary parameter to its configuration
> {code:java}
> job.getConfiguration().set("foo", "bar");
> job.setJobName("foo-bug-demo");
> {code}
> # Get the job in a client:
> {code:java}
> final Cluster c = new Cluster(conf);
> final JobStatus[] statuses = c.getAllJobStatuses();
> final JobStatus s = ... // get the status for the job named foo-bug-demo
> final Job j = c.getJob(s.getJobId());
> final Configuration conf = job.getConfiguration();
> {code}
> # Get its "foo" entry
> {code:java}
> final String s = conf.get("foo");
> {code}
> # Expected: s is "bar"; But: s is null.
> The reason is that the job's configuration is stored on HDFS (the
> Configuration has a resource with a *hdfs://* URL) and in the *loadResource*
> it is changed to a path on the local file system
> (hdfs://host.domain:port/tmp/hadoop-yarn/... is changed to
> /tmp/hadoop-yarn/...), which does not exist, and thus the configuration is
> not populated.
> The bug happens in the *Cluster* class, where *JobConfs* are created from
> *status.getJobFile()*. A quick fix would be to copy this job file to a
> temporary file in the local file system and populate the JobConf from this
> file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6320) Configuration of retrieved Job via Cluster is not properly set-up
[
https://issues.apache.org/jira/browse/MAPREDUCE-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jens Rabe updated MAPREDUCE-6320:
-
Attachment: MAPREDUCE-6320.002.patch
> Configuration of retrieved Job via Cluster is not properly set-up
> -
>
> Key: MAPREDUCE-6320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jens Rabe
>Assignee: Jens Rabe
> Attachments: MAPREDUCE-6320.001.patch, MAPREDUCE-6320.002.patch
>
>
> When getting a Job via the Cluster API, it is not correctly configured.
> To reproduce this:
> # Submit a MR job, and set some arbitrary parameter to its configuration
> {code:java}
> job.getConfiguration().set("foo", "bar");
> job.setJobName("foo-bug-demo");
> {code}
> # Get the job in a client:
> {code:java}
> final Cluster c = new Cluster(conf);
> final JobStatus[] statuses = c.getAllJobStatuses();
> final JobStatus s = ... // get the status for the job named foo-bug-demo
> final Job j = c.getJob(s.getJobId());
> final Configuration conf = job.getConfiguration();
> {code}
> # Get its "foo" entry
> {code:java}
> final String s = conf.get("foo");
> {code}
> # Expected: s is "bar"; But: s is null.
> The reason is that the job's configuration is stored on HDFS (the
> Configuration has a resource with a *hdfs://* URL) and in the *loadResource*
> it is changed to a path on the local file system
> (hdfs://host.domain:port/tmp/hadoop-yarn/... is changed to
> /tmp/hadoop-yarn/...), which does not exist, and thus the configuration is
> not populated.
> The bug happens in the *Cluster* class, where *JobConfs* are created from
> *status.getJobFile()*. A quick fix would be to copy this job file to a
> temporary file in the local file system and populate the JobConf from this
> file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6320) Configuration of retrieved Job via Cluster is not properly set-up
[
https://issues.apache.org/jira/browse/MAPREDUCE-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jens Rabe updated MAPREDUCE-6320:
-
Assignee: Jens Rabe
Status: In Progress (was: Patch Available)
> Configuration of retrieved Job via Cluster is not properly set-up
> -
>
> Key: MAPREDUCE-6320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jens Rabe
>Assignee: Jens Rabe
> Attachments: MAPREDUCE-6320.001.patch, MAPREDUCE-6320.002.patch
>
>
> When getting a Job via the Cluster API, it is not correctly configured.
> To reproduce this:
> # Submit a MR job, and set some arbitrary parameter to its configuration
> {code:java}
> job.getConfiguration().set("foo", "bar");
> job.setJobName("foo-bug-demo");
> {code}
> # Get the job in a client:
> {code:java}
> final Cluster c = new Cluster(conf);
> final JobStatus[] statuses = c.getAllJobStatuses();
> final JobStatus s = ... // get the status for the job named foo-bug-demo
> final Job j = c.getJob(s.getJobId());
> final Configuration conf = job.getConfiguration();
> {code}
> # Get its "foo" entry
> {code:java}
> final String s = conf.get("foo");
> {code}
> # Expected: s is "bar"; But: s is null.
> The reason is that the job's configuration is stored on HDFS (the
> Configuration has a resource with a *hdfs://* URL) and in the *loadResource*
> it is changed to a path on the local file system
> (hdfs://host.domain:port/tmp/hadoop-yarn/... is changed to
> /tmp/hadoop-yarn/...), which does not exist, and thus the configuration is
> not populated.
> The bug happens in the *Cluster* class, where *JobConfs* are created from
> *status.getJobFile()*. A quick fix would be to copy this job file to a
> temporary file in the local file system and populate the JobConf from this
> file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6320) Configuration of retrieved Job via Cluster is not properly set-up
[
https://issues.apache.org/jira/browse/MAPREDUCE-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502030#comment-14502030
]
Hadoop QA commented on MAPREDUCE-6320:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726450/MAPREDUCE-6320.001.patch
against trunk revision 5459b24.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5411//console
This message is automatically generated.
> Configuration of retrieved Job via Cluster is not properly set-up
> -
>
> Key: MAPREDUCE-6320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jens Rabe
> Attachments: MAPREDUCE-6320.001.patch
>
>
> When getting a Job via the Cluster API, it is not correctly configured.
> To reproduce this:
> # Submit a MR job, and set some arbitrary parameter to its configuration
> {code:java}
> job.getConfiguration().set("foo", "bar");
> job.setJobName("foo-bug-demo");
> {code}
> # Get the job in a client:
> {code:java}
> final Cluster c = new Cluster(conf);
> final JobStatus[] statuses = c.getAllJobStatuses();
> final JobStatus s = ... // get the status for the job named foo-bug-demo
> final Job j = c.getJob(s.getJobId());
> final Configuration conf = job.getConfiguration();
> {code}
> # Get its "foo" entry
> {code:java}
> final String s = conf.get("foo");
> {code}
> # Expected: s is "bar"; But: s is null.
> The reason is that the job's configuration is stored on HDFS (the
> Configuration has a resource with a *hdfs://* URL) and in the *loadResource*
> it is changed to a path on the local file system
> (hdfs://host.domain:port/tmp/hadoop-yarn/... is changed to
> /tmp/hadoop-yarn/...), which does not exist, and thus the configuration is
> not populated.
> The bug happens in the *Cluster* class, where *JobConfs* are created from
> *status.getJobFile()*. A quick fix would be to copy this job file to a
> temporary file in the local file system and populate the JobConf from this
> file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6320) Configuration of retrieved Job via Cluster is not properly set-up
[
https://issues.apache.org/jira/browse/MAPREDUCE-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502022#comment-14502022
]
Jens Rabe commented on MAPREDUCE-6320:
--
I was wrong in the first comment, the fix was introduced in commit
e8a31f2e1c34514fba2f480e8db652f6e2ed65d8 for MAPREDUCE-5875.
> Configuration of retrieved Job via Cluster is not properly set-up
> -
>
> Key: MAPREDUCE-6320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jens Rabe
> Attachments: MAPREDUCE-6320.001.patch
>
>
> When getting a Job via the Cluster API, it is not correctly configured.
> To reproduce this:
> # Submit a MR job, and set some arbitrary parameter to its configuration
> {code:java}
> job.getConfiguration().set("foo", "bar");
> job.setJobName("foo-bug-demo");
> {code}
> # Get the job in a client:
> {code:java}
> final Cluster c = new Cluster(conf);
> final JobStatus[] statuses = c.getAllJobStatuses();
> final JobStatus s = ... // get the status for the job named foo-bug-demo
> final Job j = c.getJob(s.getJobId());
> final Configuration conf = job.getConfiguration();
> {code}
> # Get its "foo" entry
> {code:java}
> final String s = conf.get("foo");
> {code}
> # Expected: s is "bar"; But: s is null.
> The reason is that the job's configuration is stored on HDFS (the
> Configuration has a resource with a *hdfs://* URL) and in the *loadResource*
> it is changed to a path on the local file system
> (hdfs://host.domain:port/tmp/hadoop-yarn/... is changed to
> /tmp/hadoop-yarn/...), which does not exist, and thus the configuration is
> not populated.
> The bug happens in the *Cluster* class, where *JobConfs* are created from
> *status.getJobFile()*. A quick fix would be to copy this job file to a
> temporary file in the local file system and populate the JobConf from this
> file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6246) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2
[
https://issues.apache.org/jira/browse/MAPREDUCE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramtin updated MAPREDUCE-6246:
--
Attachment: MAPREDUCE-6246.002.patch
> DBOutputFormat.java appending extra semicolon to query which is incompatible
> with DB2
> -
>
> Key: MAPREDUCE-6246
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6246
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
>Affects Versions: 2.4.1
> Environment: OS: RHEL 5.x, RHEL 6.x, SLES 11.x
> Platform: xSeries, pSeries
> Browser: Firefox, IE
> Security Settings: No Security, Flat file, LDAP, PAM
> File System: HDFS, GPFS FPO
>Reporter: ramtin
>Assignee: ramtin
> Labels: DB2, mapreduce
> Attachments: MAPREDUCE-6246.002.patch, MAPREDUCE-6246.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> DBoutputformat is used for writing output of mapreduce jobs to the database
> and when used with db2 jdbc drivers it fails with following error
> com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-104,
> SQLSTATE=42601, SQLERRMC=;;,COUNT) VALUES (?,?);END-OF-STATEMENT,
> DRIVER=4.16.53 at com.ibm.db2.jcc.am.fd.a(fd.java:739) at
> com.ibm.db2.jcc.am.fd.a(fd.java:60) at com.ibm.db2.jcc.am.fd.a(fd.java:127)
> In DBOutputFormat class there is constructQuery method that generates "INSERT
> INTO" statement with semicolon(";") at the end.
> Semicolon is ANSI SQL-92 standard character for a statement terminator but
> this feature is disabled(OFF) as a default settings in IBM DB2.
> Although by using -t we can turn it ON for db2.
> (http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0010410.html?cp=SSEPGG_9.7.0%2F3-6-2-0-2).
> But there are some products that already built on top of this default
> setting (OFF) so by turning ON this feature make them error prone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6246) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2
[
https://issues.apache.org/jira/browse/MAPREDUCE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramtin updated MAPREDUCE-6246:
--
Target Version/s: (was: 2.5.2)
> DBOutputFormat.java appending extra semicolon to query which is incompatible
> with DB2
> -
>
> Key: MAPREDUCE-6246
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6246
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
>Affects Versions: 2.4.1
> Environment: OS: RHEL 5.x, RHEL 6.x, SLES 11.x
> Platform: xSeries, pSeries
> Browser: Firefox, IE
> Security Settings: No Security, Flat file, LDAP, PAM
> File System: HDFS, GPFS FPO
>Reporter: ramtin
>Assignee: ramtin
> Labels: DB2, mapreduce
> Attachments: MAPREDUCE-6246.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> DBoutputformat is used for writing output of mapreduce jobs to the database
> and when used with db2 jdbc drivers it fails with following error
> com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-104,
> SQLSTATE=42601, SQLERRMC=;;,COUNT) VALUES (?,?);END-OF-STATEMENT,
> DRIVER=4.16.53 at com.ibm.db2.jcc.am.fd.a(fd.java:739) at
> com.ibm.db2.jcc.am.fd.a(fd.java:60) at com.ibm.db2.jcc.am.fd.a(fd.java:127)
> In DBOutputFormat class there is constructQuery method that generates "INSERT
> INTO" statement with semicolon(";") at the end.
> Semicolon is ANSI SQL-92 standard character for a statement terminator but
> this feature is disabled(OFF) as a default settings in IBM DB2.
> Although by using -t we can turn it ON for db2.
> (http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0010410.html?cp=SSEPGG_9.7.0%2F3-6-2-0-2).
> But there are some products that already built on top of this default
> setting (OFF) so by turning ON this feature make them error prone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6320) Configuration of retrieved Job via Cluster is not properly set-up
[
https://issues.apache.org/jira/browse/MAPREDUCE-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jens Rabe updated MAPREDUCE-6320:
-
Affects Version/s: (was: 2.4.1)
Status: Patch Available (was: Open)
See [^MAPREDUCE-6320.001.patch]
> Configuration of retrieved Job via Cluster is not properly set-up
> -
>
> Key: MAPREDUCE-6320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jens Rabe
> Attachments: MAPREDUCE-6320.001.patch
>
>
> When getting a Job via the Cluster API, it is not correctly configured.
> To reproduce this:
> # Submit a MR job, and set some arbitrary parameter to its configuration
> {code:java}
> job.getConfiguration().set("foo", "bar");
> job.setJobName("foo-bug-demo");
> {code}
> # Get the job in a client:
> {code:java}
> final Cluster c = new Cluster(conf);
> final JobStatus[] statuses = c.getAllJobStatuses();
> final JobStatus s = ... // get the status for the job named foo-bug-demo
> final Job j = c.getJob(s.getJobId());
> final Configuration conf = job.getConfiguration();
> {code}
> # Get its "foo" entry
> {code:java}
> final String s = conf.get("foo");
> {code}
> # Expected: s is "bar"; But: s is null.
> The reason is that the job's configuration is stored on HDFS (the
> Configuration has a resource with a *hdfs://* URL) and in the *loadResource*
> it is changed to a path on the local file system
> (hdfs://host.domain:port/tmp/hadoop-yarn/... is changed to
> /tmp/hadoop-yarn/...), which does not exist, and thus the configuration is
> not populated.
> The bug happens in the *Cluster* class, where *JobConfs* are created from
> *status.getJobFile()*. A quick fix would be to copy this job file to a
> temporary file in the local file system and populate the JobConf from this
> file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6320) Configuration of retrieved Job via Cluster is not properly set-up
[
https://issues.apache.org/jira/browse/MAPREDUCE-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jens Rabe updated MAPREDUCE-6320:
-
Attachment: MAPREDUCE-6320.001.patch
This was already fixed in MAPREDUCE-5001 for the getJob(JobID) method. Patch
adds a test checking this, and updated the deprecated getAllJobs() method.
> Configuration of retrieved Job via Cluster is not properly set-up
> -
>
> Key: MAPREDUCE-6320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.4.1
>Reporter: Jens Rabe
> Attachments: MAPREDUCE-6320.001.patch
>
>
> When getting a Job via the Cluster API, it is not correctly configured.
> To reproduce this:
> # Submit a MR job, and set some arbitrary parameter to its configuration
> {code:java}
> job.getConfiguration().set("foo", "bar");
> job.setJobName("foo-bug-demo");
> {code}
> # Get the job in a client:
> {code:java}
> final Cluster c = new Cluster(conf);
> final JobStatus[] statuses = c.getAllJobStatuses();
> final JobStatus s = ... // get the status for the job named foo-bug-demo
> final Job j = c.getJob(s.getJobId());
> final Configuration conf = job.getConfiguration();
> {code}
> # Get its "foo" entry
> {code:java}
> final String s = conf.get("foo");
> {code}
> # Expected: s is "bar"; But: s is null.
> The reason is that the job's configuration is stored on HDFS (the
> Configuration has a resource with a *hdfs://* URL) and in the *loadResource*
> it is changed to a path on the local file system
> (hdfs://host.domain:port/tmp/hadoop-yarn/... is changed to
> /tmp/hadoop-yarn/...), which does not exist, and thus the configuration is
> not populated.
> The bug happens in the *Cluster* class, where *JobConfs* are created from
> *status.getJobFile()*. A quick fix would be to copy this job file to a
> temporary file in the local file system and populate the JobConf from this
> file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
