Fwd: Compatibility in Apache Hadoop
On Mon, Apr 22, 2013 at 5:42 PM, Steve Loughran wrote: > > On 22 April 2013 14:00, Karthik Kambatla wrote: > > > Hadoop devs, > > > > > > This doc does not intend to propose new policies. The idea is to have one > > document that outlines the various compatibility concerns (lots of areas > > beyond API compatibility), captures the respective policies that exist, and > > if we want to define policies for the items where it’s not clear we have > > something to iterate on. > > > > The first draft just lists the types of compatibility. In the next step, we > > can add existing policies and subsequently work towards policies for > > others. > > > > > I don't see -yet- a definition of compatible at the API signature level vs > semantics level. > > The @ interface attributes say "these methods are > internal/external/stable/unstable (there's also @VisibleForTesting,that > comes out of guava (yes?). > > There's a separate issue that says "we make some guarantee that the > behaviour of a interface remains consistent over versions", which is hard > to do without some rigorous definition of what the expected behaviour of an > implementation should be. Good point, Steve. I've assumed the semantics of the API had to respect the attribute (eg changing the semantics of FileSystem#close would be an incompatible change, since this is a public/stable API, even if the new semantics are arguably better). But you're right, unless we've actually defined what the semantics of the APIs are it's hard to say if we've materially changed them. How about adding a new section on the page and calling that out explicitly? In practice I think we'll have to take semantics case by case, clearly define the semantics we care about better in the javadocs (for the major end user-facing classes at least, calling out both intended behavior and behavior that's meant to be undefined) and using individual judgement elsewhere. For example, HDFS-4156 changed DataInputStream#seek to throw an IOE if you seek to a negative offset, instead of succeeding then resulting in an NPE on the next access. That's an incompatible change in terms of semantics, but not semantics intended by the author, or likely semantics programs depend on. However if a change made FileSystem#close three times slower, this perhaps a smaller semantic change (eg doesn't change what exceptions get thrown) but probably much less tolerable for end users. In any case, even if we get an 80% solution to the semantics issue we'll probably be in good shape for v2 GA if we can sort out the remaining topics. See any other topics missing? Once the overall outline is in shape it make sense to annotate the page with the current policy (if there's already consensus on one), and identifying areas where we need to come up with a policy or are leaving TBD. Currently this is a source of confusion for new developers, some downstream projects and users. Thanks, Eli
Re: Compatibility in Apache Hadoop
On 22 April 2013 14:00, Karthik Kambatla wrote:
> Hadoop devs,
>
>
> This doc does not intend to propose new policies. The idea is to have one
> document that outlines the various compatibility concerns (lots of areas
> beyond API compatibility), captures the respective policies that exist, and
> if we want to define policies for the items where it’s not clear we have
> something to iterate on.
>
> The first draft just lists the types of compatibility. In the next step, we
> can add existing policies and subsequently work towards policies for
> others.
>
>
I don't see -yet- a definition of compatible at the API signature level vs
semantics level.
The @ interface attributes say "these methods are
internal/external/stable/unstable (there's also @VisibleForTesting,that
comes out of guava (yes?).
There's a separate issue that says "we make some guarantee that the
behaviour of a interface remains consistent over versions", which is hard
to do without some rigorous definition of what the expected behaviour of an
implementation should be. Even then, there are epiphenomena whose behaviour
isn't a direct part of the specification but which people ended up relying
on. As an example, look how many junit tests have broken in Java7 after
they changed the order in which methods were enumerated. The new ordering
fulfils the java lan specification "no order is guaranteed", but the
pre-java6 implementation offered "same order as you wrote them in the
file". In Hadoop, there's a lot of code that assumes that close() of an
output stream happens within a short period of time, and if you change that
some things break, but nowhere does {{OutputStream.close()}} ever say
"completes within 30s".
Interface semantics are far more important than simple signatures, and they
are the hardest to guarantee -especially when we don't have rigorous enough
definitions of what implementations should really do.
As an example what does the method Seekable.seek() do?
It's header says
/**
* Seek to the given offset from the start of the file.
* The next read() will be from that location. Can't
* seek past the end of the file.
*/
void seek(long pos) throws IOException;
Now, without looking at the source, what do you think the reference
implementation - BufferedFSInputStream - does on any of the sequences:
seek(-1); read()
seek(0); seek(0)
close(); seek(0)
The javadocs don't specify what happens, the implementations don't behave
in ways you'd expect, and that leaves a problem: do you now define the
semantics of seek more rigorously -and fix our implementations -or do you
conclude that some people may expect the current behaviour on negative
seeks (they are ignored, for the curious), and so we can't actually change
them.
Compatibility in Apache Hadoop
Hadoop devs, Eli and I have been working on a writeup on compatibility for developers, downstream projects and end users. The wiki page - http://wiki.apache.org/hadoop/Compatibility - is a strawman effort towards consolidating prior discussions and decisions; including the API classification effort on HADOOP-5073, Sanjay’s work on Hadoop 1.0 compatibility (HADOOP-5071), and the roadmap page on the Hadoop wiki. The goal is to identify the the relevant interfaces and concerns for downstream projects and users. In cases where the community has explicitly agreed upon a policy (e.g. HADOOP-5073: API classification), call out that policy. This doc does not intend to propose new policies. The idea is to have one document that outlines the various compatibility concerns (lots of areas beyond API compatibility), captures the respective policies that exist, and if we want to define policies for the items where it’s not clear we have something to iterate on. The first draft just lists the types of compatibility. In the next step, we can add existing policies and subsequently work towards policies for others. Would love to incorporate your feedback on the first draft. Thanks Karthik
[jira] [Created] (HADOOP-9492) Fix the typo in testConf.xml to make it consistent with FileUtil#copy()
Jing Zhao created HADOOP-9492: - Summary: Fix the typo in testConf.xml to make it consistent with FileUtil#copy() Key: HADOOP-9492 URL: https://issues.apache.org/jira/browse/HADOOP-9492 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.3.0 Reporter: Jing Zhao Assignee: Jing Zhao Priority: Trivial HADOOP-9473 fixed a typo in FileUtil#copy(). We need to fix the same typo in testConf.xml accordingly. Otherwise TestCLI will fail in branch-1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9473) typo in FileUtil copy() method
[
https://issues.apache.org/jira/browse/HADOOP-9473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao resolved HADOOP-9473.
---
Resolution: Fixed
> typo in FileUtil copy() method
> --
>
> Key: HADOOP-9473
> URL: https://issues.apache.org/jira/browse/HADOOP-9473
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.0.0-alpha, 1.1.2
>Reporter: Glen Mazza
>Priority: Trivial
> Fix For: 1.2.0, 2.0.5-beta
>
> Attachments: HADOOP-9473.branch-1.patch, HADOOP-9473.patch,
> HADOOP-9473-update-testConf-b1.2.patch
>
>
> typo:
> {code}
> Index: src/core/org/apache/hadoop/fs/FileUtil.java
> ===
> --- src/core/org/apache/hadoop/fs/FileUtil.java (revision 1467295)
> +++ src/core/org/apache/hadoop/fs/FileUtil.java (working copy)
> @@ -178,7 +178,7 @@
> // Check if dest is directory
> if (!dstFS.exists(dst)) {
>throw new IOException("`" + dst +"': specified destination directory "
> +
> -"doest not exist");
> +"does not exist");
> } else {
>FileStatus sdst = dstFS.getFileStatus(dst);
>if (!sdst.isDir())
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: How to understand Hadoop source code ?
On 19 April 2013 23:08, Noelle Jakusz (c) wrote: > > > I have created an account (noellejakusz) and would like write access to > help with this... > > OK, you have write access
Re: mrv1 vs YARN
Like with most major releases of Hadoop the releases are API compatible, but not necessarily binary compatible. That means a job for 1.0 can be recompiled against 2.0 and it should compile and run similarly to 1.0. If it does not feel free to file a JAIR on the incompatibility. There have been a few and we have worked to make them backwards compatible. As far a binary compatibility is concerned. For the most part you should be able to run your jobs without recompiling. There are some people trying to make it as binary compatible as possible, but it is not a guarantee. I would expect to have a non-alpha semi-stable release of 2.0 by late June or early July. I am not an expert on this and there are lots of things that could show up and cause those dates to slip. --Bobby On 4/21/13 6:45 PM, "Shekhar Gupta" wrote: >I am sorry Amir. I don't have answers for these questions. Because I don't >use Hadoop for any real production jobs. >Mainly I play with the Scheduler and ResourceManager of YARN as part of my >thesis. So I just run some simple jobs to test the performance of the >scheduler. > > >On Sun, Apr 21, 2013 at 4:18 PM, Amir Sanjar wrote: > >> thanks Shekhar, do you know when we will have an stable release for >>hadoop >> 2.0 (not alpha)? >> Also from you experience, what component of hadoop 2.0.3 are most >>unstable >> and are more likely need more attention? >> >> Best Regards >> Amir Sanjar >> >> System Management Architect >> PowerLinux Open Source Hadoop development lead >> IBM Senior Software Engineer >> Phone# 512-286-8393 >> Fax# 512-838-8858 >> >> >> >> [image: Inactive hide details for Shekhar Gupta ---04/21/2013 06:06:33 >> PM---As par my experience the MapReduce API is same for both YAR]Shekhar >> Gupta ---04/21/2013 06:06:33 PM---As par my experience the MapReduce >>API is >> same for both YARN and MRv2. Applications compiled again Y >> >> From: Shekhar Gupta >> To: common-dev@hadoop.apache.org, >> Date: 04/21/2013 06:06 PM >> Subject: Re: mrv1 vs YARN >> -- >> >> >> >> As par my experience the MapReduce API is same for both YARN and MRv2. >> Applications compiled again YARN should run smoothly on MRv1. Ans the >>vice >> versa is also true. >> And in general YARN is pretty stable now. >> >> Regards, >> Shekhar >> >> >> >> On Sun, Apr 21, 2013 at 3:45 PM, Amir Sanjar >>wrote: >> >> > Would an application compiled against YARN/MRv2 run transparently on >> MRv1? >> > Are there any API differences ? >> > How stable is YARN/MRV2? >> > >> > Best Regards >> > Amir Sanjar >> > >> > System Management Architect >> > PowerLinux Open Source Hadoop development lead >> > IBM Senior Software Engineer >> > Phone# 512-286-8393 >> > Fax# 512-838-8858 >> > >> >>
[jira] [Created] (HADOOP-9491) HOD fails to run under python 2.6.1
Jonathan Barber created HADOOP-9491: --- Summary: HOD fails to run under python 2.6.1 Key: HADOOP-9491 URL: https://issues.apache.org/jira/browse/HADOOP-9491 Project: Hadoop Common Issue Type: Bug Components: contrib/hod Affects Versions: 1.1.2 Environment: RHEL6.4, python-2.6.6-36.el6.x86_64 Reporter: Jonathan Barber Priority: Critical Fix For: 1.1.2 Running HOD from the hadoop-1.1.2-bin.tar.gz distribution with python 2.6.6 on RHEL6.4 fails with the output: $ ./contrib/hod/bin/hod -c contrib/hod/conf/hodrc list -d /tmp/hod-jbarber /homes/jbarber/hadoop-1.1.2/contrib/hod/hodlib/Common/threads.py:19: DeprecationWarning: The popen2 module is deprecated. Use the subprocess module. from popen2 import Popen4, Popen3, MAXFD Traceback (most recent call last): File "hod", line 411, in options=hodOptions) File "/homes/jbarber/hadoop-1.1.2/contrib/hod/hodlib/Common/setup.py", line 449, in __init__ self.read(configFile) File "/usr/lib64/python2.6/ConfigParser.py", line 286, in read self._read(fp, filename) File "/usr/lib64/python2.6/ConfigParser.py", line 475, in _read cursect = self._dict() TypeError: 'dict' object is not callable This error appears because the Hadoop "config" class is an instance of the python ConfigParser class with a class variable called ._dict. However, the internals of the ConfigParser class appears to have been changed with later versions of python to have a method called ._dict() which now conflicts with the "config" classes usage. This can be fixed by changing all references in contrib/hod/hodlib/Common/setup.py from ._dict to ._hod_dict, e.g.: sed -i 's/\._dict/._hod_dict/' contrib/hod/hodlib/Common/setup.py I tested this change by repeating the hod "list" command successfully. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
