Build failed in Jenkins: Hadoop-0.20.204-Build #4
See https://builds.apache.org/job/Hadoop-0.20.204-Build/4/changes Changes: [omalley] HADOOP-7459. Remove jdk-1.6.0 dependency check from rpm. (omalley) -- [...truncated 8398 lines...] [exec] checking for ld used by gcc... /usr/bin/ld [exec] checking if the linker (/usr/bin/ld) is GNU ld... yes [exec] checking for /usr/bin/ld option to reload object files... -r [exec] checking for BSD-compatible nm... /usr/bin/nm -B [exec] checking whether ln -s works... yes [exec] checking how to recognise dependent libraries... pass_all [exec] checking dlfcn.h usability... yes [exec] checking dlfcn.h presence... yes [exec] checking for dlfcn.h... yes [exec] checking how to run the C++ preprocessor... g++ -E [exec] checking for g77... no [exec] checking for xlf... no [exec] checking for f77... no [exec] checking for frt... no [exec] checking for pgf77... no [exec] checking for cf77... no [exec] checking for fort77... no [exec] checking for fl32... no [exec] checking for af77... no [exec] checking for xlf90... no [exec] checking for f90... no [exec] checking for pgf90... no [exec] checking for pghpf... no [exec] checking for epcf90... no [exec] checking for gfortran... no [exec] checking for g95... no [exec] checking for xlf95... no [exec] checking for f95... no [exec] checking for fort... no [exec] checking for ifort... no [exec] checking for ifc... no [exec] checking for efc... no [exec] checking for pgf95... no [exec] checking for lf95... no [exec] checking for ftn... no [exec] checking whether we are using the GNU Fortran 77 compiler... no [exec] checking whether accepts -g... no [exec] checking the maximum length of command line arguments... 32768 [exec] checking command to parse /usr/bin/nm -B output from gcc object... ok [exec] checking for objdir... .libs [exec] checking for ar... ar [exec] checking for ranlib... ranlib [exec] checking for strip... strip [exec] checking if gcc static flag works... yes [exec] checking if gcc supports -fno-rtti -fno-exceptions... no [exec] checking for gcc option to produce PIC... -fPIC [exec] checking if gcc PIC flag -fPIC works... yes [exec] checking if gcc supports -c -o file.o... yes [exec] checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes [exec] checking whether -lc should be explicitly linked in... no [exec] checking dynamic linker characteristics... GNU/Linux ld.so [exec] checking how to hardcode library paths into programs... immediate [exec] checking whether stripping libraries is possible... yes [exec] checking if libtool supports shared libraries... yes [exec] checking whether to build shared libraries... yes [exec] checking whether to build static libraries... yes [exec] configure: creating libtool [exec] appending configuration tag CXX to libtool [exec] checking for ld used by g++... /usr/bin/ld -m elf_x86_64 [exec] checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes [exec] checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes [exec] checking for g++ option to produce PIC... -fPIC [exec] checking if g++ PIC flag -fPIC works... yes [exec] checking if g++ supports -c -o file.o... yes [exec] checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes [exec] checking dynamic linker characteristics... GNU/Linux ld.so [exec] checking how to hardcode library paths into programs... immediate [exec] checking whether stripping libraries is possible... yes [exec] appending configuration tag F77 to libtool [exec] checking for unistd.h... (cached) yes [exec] checking for stdbool.h that conforms to C99... yes [exec] checking for _Bool... no [exec] checking for an ANSI C-conforming const... yes [exec] checking for off_t... yes [exec] checking for size_t... yes [exec] checking whether strerror_r is declared... yes [exec] checking for strerror_r... yes [exec] checking whether strerror_r returns char *... yes [exec] checking for mkdir... yes [exec] checking for uname... yes [exec] configure: creating ./config.status [exec] config.status: creating Makefile [exec] config.status: creating impl/config.h [exec] config.status: impl/config.h is unchanged [exec] config.status: executing depfiles commands compile-c++-utils: [exec] make[1]: Entering directory `https://builds.apache.org/job/Hadoop-0.20.204-Build/ws/trunk/build/c++-build/Linux-i386-32/utils' [exec] test -z https://builds.apache.org/job/Hadoop-0.20.204-Build/ws/trunk/build/c++/Linux-i386-32/lib; || mkdir -p --
[jira] [Resolved] (HADOOP-3291) Add StackWritable and QueueWritable classes
[ https://issues.apache.org/jira/browse/HADOOP-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-3291. - Resolution: Not A Problem I feel these are better suited as 3rd party packages, or under projects like Mahout where they may be utilized as a utility class. However, feel free to reopen if you feel they add good value to Hadoop common itself. Add StackWritable and QueueWritable classes --- Key: HADOOP-3291 URL: https://issues.apache.org/jira/browse/HADOOP-3291 Project: Hadoop Common Issue Type: New Feature Components: io Environment: All Reporter: Dennis Kubes Assignee: Dennis Kubes Attachments: HADOOP-3291-1-20080421.patch Adds Writable classes for FIFO Queu and LIFO Stack data structures. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-6342) Create a script to squash a common, hdfs, and mapreduce tarball into a single hadoop tarball
[ https://issues.apache.org/jira/browse/HADOOP-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-6342. - Resolution: Not A Problem Fix Version/s: (was: 0.21.1) 0.22.0 Looks like this was not marked as resolved post Tom's comment earlier, of https://issues.apache.org/jira/browse/HADOOP-6846 having fixed it in 0.22 Create a script to squash a common, hdfs, and mapreduce tarball into a single hadoop tarball Key: HADOOP-6342 URL: https://issues.apache.org/jira/browse/HADOOP-6342 Project: Hadoop Common Issue Type: New Feature Components: build Reporter: Owen O'Malley Assignee: Owen O'Malley Priority: Blocker Fix For: 0.22.0 Attachments: HADOOP-6342.2.patch, HADOOP-6342.patch, h-6342.patch, tar-munge, tar-munge It would be convenient for the transition if we had a script to take a set of common, hdfs, and mapreduce tarballs and merge them into a single tarball. This is intended just to help users who don't want to transition to split projects for deployment immediately. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-516) Eclipse-based GUI: DFS explorer and basic Map/Reduce job launcher
[ https://issues.apache.org/jira/browse/HADOOP-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-516. Resolution: Fixed The plugin seems to have integrated this feature already. Should this still be open, please reopen it. Eclipse-based GUI: DFS explorer and basic Map/Reduce job launcher - Key: HADOOP-516 URL: https://issues.apache.org/jira/browse/HADOOP-516 Project: Hadoop Common Issue Type: New Feature Environment: Eclipse 3.2 JDK 1.5 Reporter: Frédéric Bertin Attachments: hdfsExplorer.zip, hdfsExplorer2.zip to increase productivity in our current project (which makes a heavy use of Hadoop), we wrote a small Eclipse-based GUI application which basically consists in 2 views: * a HDFS explorer adapted from Eclipse filesystem explorer example. For now, it includes the following features: o classical tree-based browsing interface, with directory content being detailed in a 3 columns table (file name, file size, file type) o refresh button o delete file or directory (with confirm dialog): select files in the tree or table and click the Delete button o rename file or directory: simple click on the file in the table, type the new name and validate o open file with system editor: select the file in the table and click Open button (works on Windows, not on Linux) o internal drag drop o external drag drop from the local filesystem to the HDFS (the opposite doesn't work) * a MapReduce *very* simple job launcher: o select the job XML configuration file o run the job o kill the job o visualize map and reduce progress with progress bars o open a browser on the Hadoop job tracker web interface INSTALLATION NOTES: - Eclipse 3.2 - JDK 1.5 - import the archive in Eclipse - copy your hadoop conf file (hadoop-default.xml in src folder) - this step should be moved in the GUI later - right-click on the project and Run As - Eclipse Application - enjoy... -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-323) IO Exception at LocalFileSystem.renameRaw, when running Nutch nightly builds (0.8-dev).
[ https://issues.apache.org/jira/browse/HADOOP-323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-323. Resolution: Invalid Was fixed long time ago, but wasn't closed. Doesn't apply today. I run LJRunner and it never really complains in any run about things like these. Closing as Invalid (now). IO Exception at LocalFileSystem.renameRaw, when running Nutch nightly builds (0.8-dev). --- Key: HADOOP-323 URL: https://issues.apache.org/jira/browse/HADOOP-323 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.3.2 Environment: Windows XP + CygWin Reporter: KuroSaka TeruHiko IO Exception at LocalFileSystem.renameRaw, when running Nutch nightly builds (0.8-dev). Please see the deatil descriptions in: http://issues.apache.org/jira/browse/NUTCH-266 Not knowing how to reclassify an existing bug, I am opening this new bug under Hadoop. The version number is 0.3.3 but because I don't see it in the jira list, I chose the closest matching version. The Nutch-with-GUI build was running with hadoop-0.2 but stopped running, exhibiting the same symptom with other nightly builds, when switched to use hadoop-0.3.3. I checked fs as component but this bug could also be caused by the order in which jobs are scheduled, I suspect. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-466) Startup scripts will not start instances of Hadoop daemons w/different configs w/o setting separate PID directories
[
https://issues.apache.org/jira/browse/HADOOP-466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved HADOOP-466.
Resolution: Fixed
Fix Version/s: 0.20.0
This problem indeed exists if one doesn't use the HADOOP_IDENT_STRING, but
that's a valid workaround than adding a dependency on md5sum and the like (or
do we already use it?).
I think this may be resolved as fixed with the availability of
HADOOP_IDENT_STRING to workaround with.
Workaround (tested to work in 0.20.2):
{code}
# To start a second DN on same machine, with separated config.
HADOOP_IDENT_STRING=$USER-DN2 hadoop-daemon.sh --config /conf/dn2 start datanode
HADOOP_IDENT_STRING=$USER-DN2 hadoop-daemon.sh --config /conf/dn2 stop datanode
# These manage the PIDs as well, and will not complain that stuff is already
running.
{code}
Startup scripts will not start instances of Hadoop daemons w/different
configs w/o setting separate PID directories
---
Key: HADOOP-466
URL: https://issues.apache.org/jira/browse/HADOOP-466
Project: Hadoop Common
Issue Type: Improvement
Components: conf
Affects Versions: 0.5.0
Reporter: Vetle Roeim
Fix For: 0.20.0
Attachments: hadoop-466.diff
Configuration directories can be specified by either setting HADOOP_CONF_DIR
or using the --config command line option. However, the hadoop-daemon.sh
script will not start the daemons unless the PID directory is separate for
each configuration.
The issue is that the code for generating PID filenames is not dependent on
the configuration directory. While the PID directory can be changed in
hadoop-env.sh, it seems a little unnecessary to have this restriction.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-378) too many files are distributed to slaves nodes
[ https://issues.apache.org/jira/browse/HADOOP-378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-378. Resolution: Incomplete (This one might have gone stale over time, and perhaps does not apply anymore) This one needs to explain more about what its talking about - deployment of Hadoop via HADOOP_MASTER which uses rsync (and thereby does not pass out incrementals)? This would be resolved by better packaging, which is gonna be addressed in future via Bigtop and the like, hopefully. A package for examples, a package for test, a package for clients. With mavenized builds, this is very easy to do and is on its way. Resolving as Incomplete, needs more desc. too many files are distributed to slaves nodes -- Key: HADOOP-378 URL: https://issues.apache.org/jira/browse/HADOOP-378 Project: Hadoop Common Issue Type: Bug Components: conf Reporter: Yoram Arnon Priority: Trivial currently all the svn tree is sync'ed to the slaves on startup, excluding only .svn files. that includes the example jar file, sources, and other files that are not required. It would be better to pick and choose the files that get sync'ed. It will improve sync times on startup, and can prevent some name conflicts, since all the jar files are also included in the classpath by the hadoop script. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-684) it would be nice to be able to view log and output files w/in the browser
[ https://issues.apache.org/jira/browse/HADOOP-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-684. Resolution: Invalid All log files are plain text files openable by a ton of tools out there, even browsers today, and the DFS does carry a browser along in the web UI for the DFS files. Logger levels can also be set and logs can be queried via the /logs and /logLevel areas of any web UI under Hadoop. Resolving as Invalid (now). it would be nice to be able to view log and output files w/in the browser - Key: HADOOP-684 URL: https://issues.apache.org/jira/browse/HADOOP-684 Project: Hadoop Common Issue Type: Improvement Components: scripts Environment: osx, ubuntu 6.10b Reporter: James Todd Priority: Minor it would be nice to be able to view the node logs and output files from w/in the browser vs saving the file and opening an external application. one way to achieve this is to simply add .txt as a suffix to the relevant files, eg: Index: bin/hadoop-daemon.sh === --- bin/hadoop-daemon.sh(revision 468719) +++ bin/hadoop-daemon.sh(working copy) @@ -50,9 +50,9 @@ fi # some variables -export HADOOP_LOGFILE=hadoop-$HADOOP_IDENT_STRING-$command-`hostname`.log +export HADOOP_LOGFILE=hadoop-$HADOOP_IDENT_STRING-$command-`hostname`.log.txt export HADOOP_ROOT_LOGGER=INFO,DRFA -log=$HADOOP_LOG_DIR/hadoop-$HADOOP_IDENT_STRING-$command-`hostname`.out +log=$HADOOP_LOG_DIR/hadoop-$HADOOP_IDENT_STRING-$command-`hostname`.out.txt pid=$HADOOP_PID_DIR/hadoop-$HADOOP_IDENT_STRING-$command.pid # Set default scheduling priority it has been suggested that perhaps the content type of the log files could be specified, thereby allowing for wider/richer support. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-888) SequenceFile constructors should not accept a FileSystem parameter
[ https://issues.apache.org/jira/browse/HADOOP-888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-888. Resolution: Duplicate This has been done by HADOOP-6856 (FS addition is now a deprecated addition with the revamp, Options-style builder). Resolving as duplicate. SequenceFile constructors should not accept a FileSystem parameter -- Key: HADOOP-888 URL: https://issues.apache.org/jira/browse/HADOOP-888 Project: Hadoop Common Issue Type: Improvement Components: io Affects Versions: 0.10.1 Reporter: Doug Cutting Priority: Minor The SequenceFile constructors accept a FileSystem, Path and Configuration. The FileSystem is redundant, as the combination of the Configuration and the Path determine the FileSystem. These methods should thus be deprecated and replaced by FileSystem-less versions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-960) Incorrect number of map tasks when there are multiple input files
[ https://issues.apache.org/jira/browse/HADOOP-960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-960. Resolution: Invalid The ability to specify mapred.map.tasks is going away with the new API of MR. The only _right_ way to control splits is to have your own InputFormat that does it the way you need it to. The default way has worked for many (being local-data sensitive, as long as such information is available, but also split size tunable), and can also be asked to process whole files with a very simple subclass/configuration. Resolving as invalid (now, and onwards) since InputFormat#getSplits(…) is not going anywhere, and can do what you want it to. Regd. record num splits, MR now has NLineInputFormat as well, which indeed opens and reads through the file. Incorrect number of map tasks when there are multiple input files - Key: HADOOP-960 URL: https://issues.apache.org/jira/browse/HADOOP-960 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: 0.10.1 Reporter: Andrew McNabb Priority: Minor This problem happens with hadoop-streaming and possibly elsewhere. If there are 5 input files, it will create 130 map tasks, even if mapred.map.tasks=128. The number of map tasks is incorrectly set to a multiple of the number of files. (I wrote a much more complete bug report, but Jira lost it when it had an error, so I'm not in the mood to write it all again) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-1295) hadoop-config.sh resolving symlinks leads to errors
[ https://issues.apache.org/jira/browse/HADOOP-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-1295. - Resolution: Not A Problem As described by Doug in the last comment here, this can be avoided if HADOOP_HOME is explicitly set, which ought to be easy to do when you are managing your versions. Feel free to reopen if there was something else, of course. hadoop-config.sh resolving symlinks leads to errors Key: HADOOP-1295 URL: https://issues.apache.org/jira/browse/HADOOP-1295 Project: Hadoop Common Issue Type: Improvement Components: scripts Environment: RHEL3 Reporter: Lee Faris Priority: Minor My company uses a versioned deployment system where the final results are symlinked. For example:The hadoop package would be located at this location on all boxes /apollo/env/Hadoop/ This is a symlink generated by the system. The hardlink can look like this: box1: /apollo/env/Hadoop - /apollo/version/Hadoop-4456 box2: /apollo/env/Hadoop - /apollo/version/Hadoop-10039445 This piece of script from hadoop-config.sh resolves symlinks into hard links: while [ -h $this ]; do ls=`ls -ld $this` link=`expr $ls : '.*- \(.*\)$'` if expr $link : '.*/.*' /dev/null; then this=$link else this=`dirname $this`/$link fi done I am not sure why this is done. Commenting out the code makes things work for our system. I assume that was put in for a reason. Is there a solution for the original need for this code to that can work with our use case? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-1713) Create a utility to convert binary (sequence and compressed) files to strings.
[
https://issues.apache.org/jira/browse/HADOOP-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved HADOOP-1713.
-
Resolution: Fixed
The {{hadoop fs -text}} shell utility does just this, with detection.
Looks like it was added after '07, and this appears to be a stale ticket now.
Marking this as 'fixed'.
Create a utility to convert binary (sequence and compressed) files to strings.
--
Key: HADOOP-1713
URL: https://issues.apache.org/jira/browse/HADOOP-1713
Project: Hadoop Common
Issue Type: New Feature
Components: io
Reporter: Owen O'Malley
It would be nice to have a utility that looked at the first N bytes of a file
and picked a decoder for it into Strings. It could then be hooked up to
bin/hadoop fs cat and the web/ui to textify sequence and compressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-907) Unable to set Replication factor on SequenceFile
[ https://issues.apache.org/jira/browse/HADOOP-907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-907. Resolution: Fixed Resolved by https://issues.apache.org/jira/browse/HADOOP-6856 by using the ReplicationOption Unable to set Replication factor on SequenceFile Key: HADOOP-907 URL: https://issues.apache.org/jira/browse/HADOOP-907 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 0.10.1 Environment: all environments Reporter: Doug Judd Priority: Minor There does not seem to be a way to set the replication factor for a SequenceFile when creating one. The API doesn't provide a way to pass it in and a call to FileSystem.setReplication() does not have any effect if called immediately after a call to SequenceFile.createWriter() -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-1555) Fix a few more FindBugs issues
[
https://issues.apache.org/jira/browse/HADOOP-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved HADOOP-1555.
-
Resolution: Cannot Reproduce
Running:
{code}
ant findbugs -Dfindbugs.home=/path/to/findbugs
{code}
Tells me that there does not seem to be any findbugs on common trunk. Marking
as cannot reproduce (now).
This was opened back in '07. A new JIRA can be filed for current findbugs.
Fix a few more FindBugs issues
--
Key: HADOOP-1555
URL: https://issues.apache.org/jira/browse/HADOOP-1555
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 0.13.0
Reporter: Albert Strasheim
Attachments: findbugsv0.patch
I'm attaching a patch to fix a few more FindBugs issues. Most of these fixes
are relatively minor. I also went ahead and fixed some issues in the tests.
This patch includes some fixes for inconsistent synchronization reported by
FindBugs. However, FindBugs still reports some cases of inconsistent
synchronization that I'm not quite sure how to fix.
Also, in ChecksumFileSystem.copyToLocalFile, there is an instanceof
ChecksumFileSystem check that FindBugs says will always return true. I'm not
quite sure what was intended here.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-1371) Remove phased file system
[ https://issues.apache.org/jira/browse/HADOOP-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-1371. - Resolution: Not A Problem PhasedFileSystem was phased-out in 0.17 apparently. JIRA is a no-issue now. Marking as 'Not a problem' (anymore). Remove phased file system - Key: HADOOP-1371 URL: https://issues.apache.org/jira/browse/HADOOP-1371 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Owen O'Malley Assignee: Arun C Murthy It was deprecated in version 13 and should be removed in 14. There is a little bit of cleanup in streaming. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-1684) Release notes point to browsable JIRA
[ https://issues.apache.org/jira/browse/HADOOP-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-1684. - Resolution: Not A Problem Release notes are now being compiled and an editorial pass is made over them for every release as well. I believe these are at least 70% user friendly, having once been a beginner and read the release notes from 0.19 - 0.20. The process is still done in the same way since, I believe. I do not think this JIRA is valid anymore (may be cause of time, it was opened in '07 when nutch hosted hadoop's links). Thus, Not-a-problem (anymore). Release notes point to browsable JIRA - Key: HADOOP-1684 URL: https://issues.apache.org/jira/browse/HADOOP-1684 Project: Hadoop Common Issue Type: Improvement Components: documentation Reporter: Marco Nicosia Currently, the web pages on lucene.apache.org/hadoop link release notes as a pointer to JIRA. http://issues.apache.org/jira/browse/HADOOP?report=com.atlassian.jira.plugin.system.project:changelog-panel The problem with this is that JIRA's really not very user-friendly. Most of the Hadoop JIRA tickets describe the underlying cause, not the user exposed symptoms, so users who are nsidering a new release (say, looking for API changes) can't make head or tails of half of the tickets without being familiar with Hadoop internals. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-397) c/c++ record io library does not use autoconf
[ https://issues.apache.org/jira/browse/HADOOP-397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-397. Resolution: Won't Fix Resolving as Won't Fix, since the whole recordio component is now deprecated in favor of Avro (and technically ought to be removed in 0.22/0.23). Please see https://issues.apache.org/jira/browse/HADOOP-6155 c/c++ record io library does not use autoconf - Key: HADOOP-397 URL: https://issues.apache.org/jira/browse/HADOOP-397 Project: Hadoop Common Issue Type: Improvement Components: record Reporter: Owen O'Malley Assignee: Vivek Ratan It would be more convenient, if the C/C++ portion of record io used autoconf to configure the location of the required libraries and generate make files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-1222) Record IO C++ binding: buffer type not handled correctly
[
https://issues.apache.org/jira/browse/HADOOP-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved HADOOP-1222.
-
Resolution: Won't Fix
Resolving as Won't Fix, since the whole recordio component is now deprecated in
favor of Avro (and technically ought to be removed in 0.22/0.23).
Please see https://issues.apache.org/jira/browse/HADOOP-6155
Record IO C++ binding: buffer type not handled correctly
Key: HADOOP-1222
URL: https://issues.apache.org/jira/browse/HADOOP-1222
Project: Hadoop Common
Issue Type: Bug
Components: record
Reporter: David Bowen
Attachments: test.cc
I added this code to the test, which currently only tests
serialization/deserialization of an empty buffer.
std::string b = r1.getBufferVal();
static char buffer[] = {0, 1, 2, 3, 4, 5};
for (int i = 0; i 6; i++) {
b.push_back(buffer[i]);
}
The csv test fails. The generated file looks like this.
T,102,4567,99344109427290,3.145000,1.523400,',# 0 1 2 3 4 5 0 1 2 3 4
5,v{},m{}
The xml test passes, but the data in the xml file is wrong:
valuestring000102030405000102030405000102030405/string/value
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-1223) Record IO C++ binding: non-empty vector of strings does not work
[ https://issues.apache.org/jira/browse/HADOOP-1223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-1223. - Resolution: Won't Fix Resolving as Won't Fix, since the whole recordio component is now deprecated in favor of Avro (and technically ought to be removed in 0.22/0.23). Please see https://issues.apache.org/jira/browse/HADOOP-6155 Record IO C++ binding: non-empty vector of strings does not work Key: HADOOP-1223 URL: https://issues.apache.org/jira/browse/HADOOP-1223 Project: Hadoop Common Issue Type: Bug Components: record Reporter: David Bowen Attachments: test.cc It works in the binary case, but not in CSV or XML. Here is the code to put some strings in the vector. std::vectorstd::string v = r1.getVectorVal(); v.push_back(hello); v.push_back(world); In the CSV file, the strings appear twice, for some reason. In the XML file they appear three times. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-1277) The class generated by Hadoop Record rcc should provide a static method to return the DDL string
[ https://issues.apache.org/jira/browse/HADOOP-1277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-1277. - Resolution: Won't Fix Resolving as Won't Fix, since the whole recordio component is now deprecated in favor of Avro (and technically ought to be removed in 0.22/0.23). Please see https://issues.apache.org/jira/browse/HADOOP-6155 The class generated by Hadoop Record rcc should provide a static method to return the DDL string Key: HADOOP-1277 URL: https://issues.apache.org/jira/browse/HADOOP-1277 Project: Hadoop Common Issue Type: New Feature Components: record Reporter: Runping Qi The method will look like: public static string getDDL(); With this class, when a map/reduce job write out sequence file swith such a generated class as its value class, the job can also save the DDL of the class into a file. With such a file around, we can implement a record reader that can generate the required class on demand, thus, can read a sequence file of Hadoop Records without having the class a priori. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-1225) Record IO class should provide a toString(String charset) method
[ https://issues.apache.org/jira/browse/HADOOP-1225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-1225. - Resolution: Won't Fix Resolving as Won't Fix, since the whole recordio component is now deprecated in favor of Avro (and technically ought to be removed in 0.22/0.23). Please see https://issues.apache.org/jira/browse/HADOOP-6155 Record IO class should provide a toString(String charset) method Key: HADOOP-1225 URL: https://issues.apache.org/jira/browse/HADOOP-1225 Project: Hadoop Common Issue Type: Improvement Components: record Reporter: Runping Qi Assignee: Sameer Paranjpye Currently, the toString() function returns the csv format serialized form of the record object. Unfortunately, all the fields of Buffer type are serialized into hex string. Although this is a loss less conversion, it is not the most convenient form, when perople use Buffer to store international texts. With a new function toString(String charset) , the user can pass a charset to indicate the desired way to convert a Buffer to a String. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-1095) Provide ByteStreams in C++ version of record I/O
[ https://issues.apache.org/jira/browse/HADOOP-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-1095. - Resolution: Won't Fix Resolving as Won't Fix, since the whole recordio component is now deprecated in favor of Avro (and technically ought to be removed in 0.22/0.23). Please see https://issues.apache.org/jira/browse/HADOOP-6155 Provide ByteStreams in C++ version of record I/O Key: HADOOP-1095 URL: https://issues.apache.org/jira/browse/HADOOP-1095 Project: Hadoop Common Issue Type: Improvement Components: record Affects Versions: 0.12.0 Environment: All Reporter: Milind Bhandarkar Assignee: Vivek Ratan Implement ByteInStream and ByteOutStream for C++ runtime, as they will be needed for using Hadoop Record I/O with forthcoming C++ MapReduce framework (currently, only FileStreams are provided.) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-1227) Record IO C++ binding: cannot write more than one record to an XML stream and read them back
[ https://issues.apache.org/jira/browse/HADOOP-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-1227. - Resolution: Won't Fix Resolving as Won't Fix, since the whole recordio component is now deprecated in favor of Avro (and technically ought to be removed in 0.22/0.23). Please see https://issues.apache.org/jira/browse/HADOOP-6155 Record IO C++ binding: cannot write more than one record to an XML stream and read them back Key: HADOOP-1227 URL: https://issues.apache.org/jira/browse/HADOOP-1227 Project: Hadoop Common Issue Type: Bug Components: record Reporter: David Bowen I tried just writing the same record twice and then reading it back twice, and got a segmentation fault. This works fine in the binary and csv cases. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-6712) librecordio support for xerces 3, eliminate compiler warnings and the (optional) ability to compile in the source directory
[ https://issues.apache.org/jira/browse/HADOOP-6712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-6712. - Resolution: Won't Fix Resolving as Won't Fix, since the whole recordio component is now deprecated in favor of Avro (and technically ought to be removed in 0.22/0.23). Please see https://issues.apache.org/jira/browse/HADOOP-6155 Do reopen if you wish to maintain recordio over 0.20.x branches. Although Avro works with 0.20 as well. librecordio support for xerces 3, eliminate compiler warnings and the (optional) ability to compile in the source directory --- Key: HADOOP-6712 URL: https://issues.apache.org/jira/browse/HADOOP-6712 Project: Hadoop Common Issue Type: Bug Components: record Environment: 64-bit linux w/gcc 4.4.3 w/xerces 3 Reporter: John Plevyak Attachments: librecordio-jp-v1.patch I don't know if this code is current supported, but since it is in the tree here are some fixes: 1. support for xerces 3.X as well as 2.X the patch checks XERCES_VERSION_MAJOR and I have tested on 3.X but before committing, someone should retest on 2.X 2. gcc 4.4.3 on 64-bit complains about using %lld with int64_t. Casting to 'long long int' solves the issue 3. since there is currently no ant target, check if LIBRECORDIO_BUILD_DIR is undefined and if so assume '.' to support compiling in the source directory This should not effect normal compilation if/when an ant target is created. patch attached -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-1916) FSShell put or CopyFromLocal incorrectly treats .
[
https://issues.apache.org/jira/browse/HADOOP-1916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved HADOOP-1916.
-
Resolution: Fixed
Fixed by a redesign on trunk.
Please https://issues.apache.org/jira/browse/HADOOP-7176 for the umbrella of
changes leading to.
FSShell put or CopyFromLocal incorrectly treats .
---
Key: HADOOP-1916
URL: https://issues.apache.org/jira/browse/HADOOP-1916
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 0.14.1
Reporter: Konstantin Shvachko
Assignee: Chris Douglas
Attachments: 1916.patch
The following dfs shell command
{code}
bin/hadoop dfs -put README.txt .
{code}
results in creating a file /user/user name with the contents of README.txt.
A correct behavior would be creating a directory and a file in it:
/user/user name/README.txt
The put command works correctly if /user/user name already exists.
So the following sequence of command leads to the desired result:
{code}
bin/hadoop dfs -mkdir .
bin/hadoop dfs -put README.txt .
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-1919) Add option to allow Binding Jetty to localhost
[ https://issues.apache.org/jira/browse/HADOOP-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-1919. - Resolution: Not A Problem Not a problem. Can be doable by the previous comment, with logical repercussions. Add option to allow Binding Jetty to localhost -- Key: HADOOP-1919 URL: https://issues.apache.org/jira/browse/HADOOP-1919 Project: Hadoop Common Issue Type: New Feature Affects Versions: 0.14.0 Reporter: Thurman Turner Priority: Minor We would like a configurable option to have Jetty bound to the loopback address of the machine so that the dfs-browser is not accessible from outside the host. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-1994) Variable names generated by Record I/O should not clash with user fields
[
https://issues.apache.org/jira/browse/HADOOP-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved HADOOP-1994.
-
Resolution: Won't Fix
Record IO has been deprecated with the advent of Avro.
Please see HADOOP-6155
Resolving as Won't Fix.
Variable names generated by Record I/O should not clash with user fields
Key: HADOOP-1994
URL: https://issues.apache.org/jira/browse/HADOOP-1994
Project: Hadoop Common
Issue Type: Bug
Reporter: Vivek Ratan
Assignee: Vivek Ratan
The code (Java and C++) spit out by the Record I/O compiler contains
variables. We need to make sure these variable names don't clash with names
used by users in the DDL, otherwise the generated code will not compile.
Variable names such as 'a', 'peer', etc, are used. We need better names. For
example, if I have a DDL of the form
{code}
class s1 {
int a;
boolean peer;
int a_;
}
{code}
Both the Java and C++ code will not compile.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
