Hi Andrew,
Have you seen the new MiniMRClientCluster class? It's meant to be what
you describe - a minicluster which only exposes "external" APIs --
most importantly a way of getting at a JobClient to submit jobs. We
have it implemented in both 1.x and 2.x at this point, though I don't
recall if it's in the 1.0.x releases or if it's only slated for 1.1+
-Todd
On Wed, May 9, 2012 at 6:05 PM, Andrew Purtell <andrew.purt...@gmail.com> wrote:
> Hi Suresh,
>
> The unstable designation makes sense. As would one for MiniMRCluster.
>
> I was over the top initially to surprise. I'm sure the MR minicluster seems a
> minor detail.
>
> Maybe it's worth thinking about the miniclusters differently? Please pardon
> if I am rehashing an old discussion.
>
> Things like MRUnit for applications and BigTop for full cluster tests can
> help, but for as mentioned in the below annotation Pig, Hive, HBase, and
> other parts of the stack use miniclusters for local end to end testing in
> unit tests. As the complexity of the stack increases and we consider cross
> version support, unit tests on miniclusters I think will have no substitute.
>
> As Hadoop 2 has been evolving there has been some difficulty keeping up with
> minicluster changes. This makes sense. The attention to stability to client
> APIs and such, and the lack thereof to the minicluster, I think is self
> evident. But the need to fix up tests unpredictably introduces some friction
> that perhaps need not be there.
>
> Would a JIRA to discuss defining a subset of the minicluster interfaces as
> more stable be worthwhile?
>
> Best regards,
>
> - Andy
>
>
> On May 9, 2012, at 1:45 PM, Suresh Srinivas <sur...@hortonworks.com> wrote:
>
>> For this reason, in HDFS, we change MiniDFSCluster to LimitedPrivate and
>> not treat it as such:
>>
>> @InterfaceAudience.LimitedPrivate({"HBase", "HDFS", "Hive", "MapReduce",
>> "Pig"})
>> @InterfaceStability.Unstable
>> public class MiniDFSCluster { ...}
>>
>> On Wed, May 9, 2012 at 11:33 AM, Andrew Purtell <apurt...@apache.org> wrote:
>>
>>> Sounds good Arun.
>>>
>>> How should we consider the suitability and stability of MiniMRCluster
>>> for downstream projects?
>>>
>>> On Wed, May 9, 2012 at 11:30 AM, Arun C Murthy <a...@hortonworks.com>
>>> wrote:
>>>> No worries Andy. I can spin an rc1 once we can pin-point the bug.
>>>>
>>>> thanks,
>>>> Arun
>>>>
>>>> On May 9, 2012, at 10:17 AM, Andrew Purtell wrote:
>>>>
>>>>> -1 (nonbinding), we are currently facing a minicluster semantic change
>>>>> of some kind, or more than one:
>>>>>
>>>>> https://issues.apache.org/jira/browse/HBASE-5966
>>>>>
>>>>> There are other HBase JIRAs related to 2.0.0-alpha that we are working
>>>>> on, but I'd claim those are all our fault for breaking abstractions to
>>>>> solve issues. In one case there's a new helpful 2.x API
>>>>> (ShutdownHookManager, thank you!) that we can eventually move to.
>>>>>
>>>>> However, the minicluster changes are causing us some repeated
>>>>> discomfort. It will break, we'll get some help fixing up our tests for
>>>>> that, then some time later it will break again, repeat. Perhaps we
>>>>> have no right to complain, the minicluster isn't meant to be used by
>>>>> downstream projects. If so then please disregard the complaint, but
>>>>> your assistance in helping to fix the breakage again would be much
>>>>> appreciated. And, if so, perhaps we can discuss what makes sense in
>>>>> terms of a stable minicluster consumable for downstream projects?
>>>>>
>>>>> Best regards,
>>>>>
>>>>> - Andy
>>>>>
>>>>> On Wed, May 9, 2012 at 9:58 AM, Arun C Murthy <a...@hortonworks.com>
>>> wrote:
>>>>>> I've created a release candidate for hadoop-2.0.0-alpha that I would
>>> like to release.
>>>>>>
>>>>>> It is available at:
>>> http://people.apache.org/~acmurthy/hadoop-2.0.0-alpha-rc0/
>>>>>>
>>>>>> The maven artifacts are available via repository.apache.org.
>>>>>>
>>>>>> Please try the release and vote; the vote will run for the usual 7
>>> days.
>>>>>>
>>>>>> This is a big milestone for the Apache Hadoop community -
>>> congratulations and thanks for all the contributions!
>>>>>>
>>>>>> thanks,
>>>>>> Arun
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Arun C. Murthy
>>>>>> Hortonworks Inc.
>>>>>> http://hortonworks.com/
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> - Andy
>>>>>
>>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>>>> Hein (via Tom White)
>>>>
>>>> --
>>>> Arun C. Murthy
>>>> Hortonworks Inc.
>>>> http://hortonworks.com/
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>> Hein (via Tom White)
>>>
--
Todd Lipcon
Software Engineer, Cloudera
