Re: Review Request 28873: ACCUMULO-3393 Follow-on work for per-table volume chooser.

[Sean Busbey]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28873/
---

(Updated Dec. 11, 2014, 8:14 a.m.)


Review request for accumulo and Jenna Huston.


Changes
---

updated for feedback.

* remove object pool and set set intersection optimizations
* disambiguate variable names
* clarified docs on PreferredVolumeChooser config property
* fixed javadoc warnings
* switch to log + continue in the case of misbehaving chooser.


Bugs: ACCUMULO-3393
https://issues.apache.org/jira/browse/ACCUMULO-3393


Repository: accumulo


Description (updated)
---

ACCUMULO-3393 Follow-on work for per-table volume chooser.

Still has TODOs for additional tickets I need to file. I'll update the review 
to remove once I have them all filed. I think everything marked TODO can wait 
for a later ticket. Please comment on relevant section if you think something 
needs to be done now.

* docs clean up + code guideline compliance.
* ensure RandomVolumeChoosers are independent when used per-table.
* make sure that per-table choosers can keep state the way that global choosers 
can
* make sure that a chooser can only pick from the options it is presented.
* reduce object creation in critical path for Tablet.split.


Diffs (updated)
-

  core/src/main/java/org/apache/accumulo/core/conf/Property.java 
4c2d0b41b3bce32449861da3ac42fa27deb2b182 
  
server/base/src/main/java/org/apache/accumulo/server/fs/PerTableVolumeChooser.java
 7a825c796eb5a25de8c748e2aba642f483697b9a 
  
server/base/src/main/java/org/apache/accumulo/server/fs/PreferredVolumeChooser.java
 7ed7bba809a4e5e4b2d472c3327b15adb37251a7 
  
server/base/src/main/java/org/apache/accumulo/server/fs/RandomVolumeChooser.java
 f2eb2113cb848ed58ac5f41573c6ff2cde9b0a77 
  server/base/src/main/java/org/apache/accumulo/server/fs/VolumeChooser.java 
f523057b11a2dc42e82010675bb1ac8e3802f96d 
  server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManager.java 
890651e92f4c34514cb839b7b9ee9d23ad55070a 
  
server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManagerImpl.java 
dc1be739b634d91992894f8f27c2d9c184bd98cd 
  server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java 
877d01c2233cca086c1ac1539eb81cc282a7ceb4 
  
server/base/src/main/java/org/apache/accumulo/server/util/TabletOperations.java 
c0e1a9b991d61a4dbb127c74ae297f171434e7d5 
  
server/base/src/test/java/org/apache/accumulo/server/fs/VolumeManagerImplTest.java
 582822a8ccbc398925a2184eed0b9d7fa853f9b4 

Diff: https://reviews.apache.org/r/28873/diff/


Testing
---

Ran through VolumeManagerImplTest and VolumeChooserIT 


Thanks,

Sean Busbey

Re: accumulo Scanner

[Ryan Leary]

One option is to implement a class extending the RowEncodingIterator: 
http://accumulo.apache.org/1.6/apidocs/org/apache/accumulo/core/iterators/user/RowEncodingIterator.html
 


I personally have implementations that will automatically return a serialized 
protocol buffer or thrift object based on keys in the data. I think it should 
be possible to implement an iterator with rowEncoder and rowDecoder methods to 
convert to a JavaBean.


> On Dec 10, 2014, at 9:55 PM, Russ Weeks  wrote:
> 
> You can use the WholeRowIterator to get a map of all key/value pairs for a
> single row. It works as long as your row is small enough to fit in memory.
> 
> I don't know of any libraries convert between a row and a JavaBean. Typo[2]
> is the closest project I know of.
> 
> -Russ
> 
> 1:
> http://accumulo.apache.org/1.6/apidocs/org/apache/accumulo/core/iterators/user/WholeRowIterator.html
> 2:https://github.com/keith-turner/typo
> 
> On Wed, Dec 10, 2014 at 6:14 PM, panqing...@163.com 
> wrote:
> 
>> HI
>> Scanner full table, but need the data in each row, is the rowkey of the
>> same
>> data, save data to JavaBean.
>> 
>> 
>> 
>> 
>> 
>> --
>> View this message in context:
>> http://apache-accumulo.1065345.n5.nabble.com/accumulo-Scanner-tp12506.html
>> Sent from the Developers mailing list archive at Nabble.com.
>> 

Re: Planning for (eventual) removal of instance.dfs.{uri,dir}

[dlmarion]

How so? If someone upgrades from another version and is using a different dir, 
doesn't specify it in the configuration, and we assume /accumulo, then their 
database won't start. The other option, which may be safer than making any 
assumptions, is to make instance.volumes a required parameter with no defaults. 

- Original Message -

From: "Christopher"  
To: "Accumulo Dev List"  
Sent: Wednesday, December 10, 2014 11:51:39 PM 
Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir} 

The URI is probably reasonable, but the dir is potentially problematic if 
you weren't using the default. 


-- 
Christopher L Tubbs II 
http://gravatar.com/ctubbsii 

On Wed, Dec 10, 2014 at 10:03 PM, dlmarion  wrote: 

> Looks like VolumeConfiguration falls back to fs.defaultFS for the uri and 
> /accumulo for the dir. You could remove both properties and still keep this 
> as the documented default behavior if instance.volumes is not specified. 
> 
> 
> 
>  Original message From: Christopher < 
> ctubb...@apache.org> Date:12/10/2014 9:13 PM (GMT-05:00) 
> To: Accumulo Dev List  
> Cc: Subject: Re: Planning for (eventual) removal of 
> instance.dfs.{uri,dir}  
> My ACCUMULO-2589 branch in github ( 
> https://github.com/ctubbsii/accumulo/tree/ACCUMULO-2589) does have a 
> commit 
> that drops a bunch of stuff (which may or may not be accepted as is for 
> 2.0). The instance.dfs.{uri,dir} properties aren't so easy and require more 
> planning, because it's not just removing a property... it's also dealing 
> with updating internal data by making relative paths absolute. 
> 
> For what it's worth, I'm also looking at what changes if we drop Hadoop 1 
> support. 
> 
> As for the validation of config, I think we do a sanity check on startup 
> already, which we can always extend. Doesn't solve this issue, though. 
> 
> 
> -- 
> Christopher L Tubbs II 
> http://gravatar.com/ctubbsii 
> 
> On Wed, Dec 10, 2014 at 8:59 PM, dlmarion  wrote: 
> 
> > We should schedule a bunch of deprecated things for removal in 2.0 to 
> ease 
> > maintenance. Do we have a way to validate the site.xml and zookeeper 
> > settings before startup and fail with appropriate error message. 
> > 
> > 
> > 
> >  Original message From: Christopher < 
> > ctubb...@apache.org> Date:12/10/2014 8:44 PM (GMT-05:00) 
> > To: Accumulo Dev List  
> > Cc: Subject: Planning for (eventual) removal of 
> > instance.dfs.{uri,dir}  
> > So, 
> > 
> > instance.volumes replaces instance.dfs.uri and instance.dfs.dir in 1.6. 
> > But, what's our long-term plan for these? I ask, because we still have 
> > internal code that uses instance.dfs.uri to resolve relative paths. 
> > 
> > Should we force these to be re-written at some point (maybe on upgrade to 
> > 1.7)? Should we continue to support the deprecated properties 
> indefinitely 
> > and continue the lazy re-write-on-compact? Do we transition by requiring 
> > instance.volumes to specify the volumes, and only use the old properties 
> to 
> > resolve relative paths? 
> > 
> > My personal view is that we should provide an upgrade-prep/check tool 
> prior 
> > to upgrade to 2.0, which checks and/or re-writes paths and verifies that 
> > instance.volumes is set. 
> > 
> > Does anybody have a different opinion on this? 
> > 
> > -- 
> > Christopher L Tubbs II 
> > http://gravatar.com/ctubbsii 
> > 
> 

Re: Planning for (eventual) removal of instance.dfs.{uri,dir}

[Christopher]

Well, no, the database will start if we rely on instance.volumes, but we
may be unable to find files that have relative paths, if we incorrectly
assume /accumulo. Making it required is annoying if users don't have
relative paths.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Thu, Dec 11, 2014 at 8:15 AM,  wrote:

> How so? If someone upgrades from another version and is using a different
> dir, doesn't specify it in the configuration, and we assume /accumulo, then
> their database won't start. The other option, which may be safer than
> making any assumptions, is to make instance.volumes a required parameter
> with no defaults.
>
> - Original Message -
>
> From: "Christopher" 
> To: "Accumulo Dev List" 
> Sent: Wednesday, December 10, 2014 11:51:39 PM
> Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir}
>
> The URI is probably reasonable, but the dir is potentially problematic if
> you weren't using the default.
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
> On Wed, Dec 10, 2014 at 10:03 PM, dlmarion  wrote:
>
> > Looks like VolumeConfiguration falls back to fs.defaultFS for the uri and
> > /accumulo for the dir. You could remove both properties and still keep
> this
> > as the documented default behavior if instance.volumes is not specified.
> >
> >
> >
> >  Original message From: Christopher <
> > ctubb...@apache.org> Date:12/10/2014 9:13 PM (GMT-05:00)
> > To: Accumulo Dev List 
> > Cc: Subject: Re: Planning for (eventual) removal of
> > instance.dfs.{uri,dir} 
> > My ACCUMULO-2589 branch in github (
> > https://github.com/ctubbsii/accumulo/tree/ACCUMULO-2589) does have a
> > commit
> > that drops a bunch of stuff (which may or may not be accepted as is for
> > 2.0). The instance.dfs.{uri,dir} properties aren't so easy and require
> more
> > planning, because it's not just removing a property... it's also dealing
> > with updating internal data by making relative paths absolute.
> >
> > For what it's worth, I'm also looking at what changes if we drop Hadoop 1
> > support.
> >
> > As for the validation of config, I think we do a sanity check on startup
> > already, which we can always extend. Doesn't solve this issue, though.
> >
> >
> > --
> > Christopher L Tubbs II
> > http://gravatar.com/ctubbsii
> >
> > On Wed, Dec 10, 2014 at 8:59 PM, dlmarion  wrote:
> >
> > > We should schedule a bunch of deprecated things for removal in 2.0 to
> > ease
> > > maintenance. Do we have a way to validate the site.xml and zookeeper
> > > settings before startup and fail with appropriate error message.
> > >
> > >
> > >
> > >  Original message From: Christopher <
> > > ctubb...@apache.org> Date:12/10/2014 8:44 PM (GMT-05:00)
> > > To: Accumulo Dev List 
> > > Cc: Subject: Planning for (eventual) removal of
> > > instance.dfs.{uri,dir} 
> > > So,
> > >
> > > instance.volumes replaces instance.dfs.uri and instance.dfs.dir in 1.6.
> > > But, what's our long-term plan for these? I ask, because we still have
> > > internal code that uses instance.dfs.uri to resolve relative paths.
> > >
> > > Should we force these to be re-written at some point (maybe on upgrade
> to
> > > 1.7)? Should we continue to support the deprecated properties
> > indefinitely
> > > and continue the lazy re-write-on-compact? Do we transition by
> requiring
> > > instance.volumes to specify the volumes, and only use the old
> properties
> > to
> > > resolve relative paths?
> > >
> > > My personal view is that we should provide an upgrade-prep/check tool
> > prior
> > > to upgrade to 2.0, which checks and/or re-writes paths and verifies
> that
> > > instance.volumes is set.
> > >
> > > Does anybody have a different opinion on this?
> > >
> > > --
> > > Christopher L Tubbs II
> > > http://gravatar.com/ctubbsii
> > >
> >
>
>

Re: Planning for (eventual) removal of instance.dfs.{uri,dir}

[dlmarion]

"Making it required is annoying if users don't have relative paths." 

But this property is used to determine the location of new files. 

- Original Message -

From: "Christopher"  
To: "Accumulo Dev List"  
Sent: Thursday, December 11, 2014 10:03:56 AM 
Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir} 

Well, no, the database will start if we rely on instance.volumes, but we 
may be unable to find files that have relative paths, if we incorrectly 
assume /accumulo. Making it required is annoying if users don't have 
relative paths. 


-- 
Christopher L Tubbs II 
http://gravatar.com/ctubbsii 

On Thu, Dec 11, 2014 at 8:15 AM,  wrote: 

> How so? If someone upgrades from another version and is using a different 
> dir, doesn't specify it in the configuration, and we assume /accumulo, then 
> their database won't start. The other option, which may be safer than 
> making any assumptions, is to make instance.volumes a required parameter 
> with no defaults. 
> 
> - Original Message - 
> 
> From: "Christopher"  
> To: "Accumulo Dev List"  
> Sent: Wednesday, December 10, 2014 11:51:39 PM 
> Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir} 
> 
> The URI is probably reasonable, but the dir is potentially problematic if 
> you weren't using the default. 
> 
> 
> -- 
> Christopher L Tubbs II 
> http://gravatar.com/ctubbsii 
> 
> On Wed, Dec 10, 2014 at 10:03 PM, dlmarion  wrote: 
> 
> > Looks like VolumeConfiguration falls back to fs.defaultFS for the uri and 
> > /accumulo for the dir. You could remove both properties and still keep 
> this 
> > as the documented default behavior if instance.volumes is not specified. 
> > 
> > 
> > 
> >  Original message From: Christopher < 
> > ctubb...@apache.org> Date:12/10/2014 9:13 PM (GMT-05:00) 
> > To: Accumulo Dev List  
> > Cc: Subject: Re: Planning for (eventual) removal of 
> > instance.dfs.{uri,dir}  
> > My ACCUMULO-2589 branch in github ( 
> > https://github.com/ctubbsii/accumulo/tree/ACCUMULO-2589) does have a 
> > commit 
> > that drops a bunch of stuff (which may or may not be accepted as is for 
> > 2.0). The instance.dfs.{uri,dir} properties aren't so easy and require 
> more 
> > planning, because it's not just removing a property... it's also dealing 
> > with updating internal data by making relative paths absolute. 
> > 
> > For what it's worth, I'm also looking at what changes if we drop Hadoop 1 
> > support. 
> > 
> > As for the validation of config, I think we do a sanity check on startup 
> > already, which we can always extend. Doesn't solve this issue, though. 
> > 
> > 
> > -- 
> > Christopher L Tubbs II 
> > http://gravatar.com/ctubbsii 
> > 
> > On Wed, Dec 10, 2014 at 8:59 PM, dlmarion  wrote: 
> > 
> > > We should schedule a bunch of deprecated things for removal in 2.0 to 
> > ease 
> > > maintenance. Do we have a way to validate the site.xml and zookeeper 
> > > settings before startup and fail with appropriate error message. 
> > > 
> > > 
> > > 
> > >  Original message From: Christopher < 
> > > ctubb...@apache.org> Date:12/10/2014 8:44 PM (GMT-05:00) 
> > > To: Accumulo Dev List  
> > > Cc: Subject: Planning for (eventual) removal of 
> > > instance.dfs.{uri,dir}  
> > > So, 
> > > 
> > > instance.volumes replaces instance.dfs.uri and instance.dfs.dir in 1.6. 
> > > But, what's our long-term plan for these? I ask, because we still have 
> > > internal code that uses instance.dfs.uri to resolve relative paths. 
> > > 
> > > Should we force these to be re-written at some point (maybe on upgrade 
> to 
> > > 1.7)? Should we continue to support the deprecated properties 
> > indefinitely 
> > > and continue the lazy re-write-on-compact? Do we transition by 
> requiring 
> > > instance.volumes to specify the volumes, and only use the old 
> properties 
> > to 
> > > resolve relative paths? 
> > > 
> > > My personal view is that we should provide an upgrade-prep/check tool 
> > prior 
> > > to upgrade to 2.0, which checks and/or re-writes paths and verifies 
> that 
> > > instance.volumes is set. 
> > > 
> > > Does anybody have a different opinion on this? 
> > > 
> > > -- 
> > > Christopher L Tubbs II 
> > > http://gravatar.com/ctubbsii 
> > > 
> > 
> 
> 

Re: Planning for (eventual) removal of instance.dfs.{uri,dir}

[Jeremy Kepner]

When we remove functions, do we have any official guidance to our users who may
have built applications that use those functions?

Right now, the official position is that the Accumulo developers can remove
as they see fit.  However, this provides no guidance to users as to what
they are suppose to do?  As it stands, our guidance is that they have the 
following choices:

(0) Diligently watch the Accumulo e-mail list and aggressively weigh in on
any vote to remove functions that may impact them.

(1) Find someone to modify the original source code of their applications,
build it, and *re-verify* the application.  I emphasise the re-verify because
that is usually the most costly part of the process that often won't get 
approved
by management.

(2) Don't upgrade Accumulo.





On Thu, Dec 11, 2014 at 10:03:56AM -0500, Christopher wrote:
> Well, no, the database will start if we rely on instance.volumes, but we
> may be unable to find files that have relative paths, if we incorrectly
> assume /accumulo. Making it required is annoying if users don't have
> relative paths.
> 
> 
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
> 
> On Thu, Dec 11, 2014 at 8:15 AM,  wrote:
> 
> > How so? If someone upgrades from another version and is using a different
> > dir, doesn't specify it in the configuration, and we assume /accumulo, then
> > their database won't start. The other option, which may be safer than
> > making any assumptions, is to make instance.volumes a required parameter
> > with no defaults.
> >
> > - Original Message -
> >
> > From: "Christopher" 
> > To: "Accumulo Dev List" 
> > Sent: Wednesday, December 10, 2014 11:51:39 PM
> > Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir}
> >
> > The URI is probably reasonable, but the dir is potentially problematic if
> > you weren't using the default.
> >
> >
> > --
> > Christopher L Tubbs II
> > http://gravatar.com/ctubbsii
> >
> > On Wed, Dec 10, 2014 at 10:03 PM, dlmarion  wrote:
> >
> > > Looks like VolumeConfiguration falls back to fs.defaultFS for the uri and
> > > /accumulo for the dir. You could remove both properties and still keep
> > this
> > > as the documented default behavior if instance.volumes is not specified.
> > >
> > >
> > >
> > >  Original message From: Christopher <
> > > ctubb...@apache.org> Date:12/10/2014 9:13 PM (GMT-05:00)
> > > To: Accumulo Dev List 
> > > Cc: Subject: Re: Planning for (eventual) removal of
> > > instance.dfs.{uri,dir} 
> > > My ACCUMULO-2589 branch in github (
> > > https://github.com/ctubbsii/accumulo/tree/ACCUMULO-2589) does have a
> > > commit
> > > that drops a bunch of stuff (which may or may not be accepted as is for
> > > 2.0). The instance.dfs.{uri,dir} properties aren't so easy and require
> > more
> > > planning, because it's not just removing a property... it's also dealing
> > > with updating internal data by making relative paths absolute.
> > >
> > > For what it's worth, I'm also looking at what changes if we drop Hadoop 1
> > > support.
> > >
> > > As for the validation of config, I think we do a sanity check on startup
> > > already, which we can always extend. Doesn't solve this issue, though.
> > >
> > >
> > > --
> > > Christopher L Tubbs II
> > > http://gravatar.com/ctubbsii
> > >
> > > On Wed, Dec 10, 2014 at 8:59 PM, dlmarion  wrote:
> > >
> > > > We should schedule a bunch of deprecated things for removal in 2.0 to
> > > ease
> > > > maintenance. Do we have a way to validate the site.xml and zookeeper
> > > > settings before startup and fail with appropriate error message.
> > > >
> > > >
> > > >
> > > >  Original message From: Christopher <
> > > > ctubb...@apache.org> Date:12/10/2014 8:44 PM (GMT-05:00)
> > > > To: Accumulo Dev List 
> > > > Cc: Subject: Planning for (eventual) removal of
> > > > instance.dfs.{uri,dir} 
> > > > So,
> > > >
> > > > instance.volumes replaces instance.dfs.uri and instance.dfs.dir in 1.6.
> > > > But, what's our long-term plan for these? I ask, because we still have
> > > > internal code that uses instance.dfs.uri to resolve relative paths.
> > > >
> > > > Should we force these to be re-written at some point (maybe on upgrade
> > to
> > > > 1.7)? Should we continue to support the deprecated properties
> > > indefinitely
> > > > and continue the lazy re-write-on-compact? Do we transition by
> > requiring
> > > > instance.volumes to specify the volumes, and only use the old
> > properties
> > > to
> > > > resolve relative paths?
> > > >
> > > > My personal view is that we should provide an upgrade-prep/check tool
> > > prior
> > > > to upgrade to 2.0, which checks and/or re-writes paths and verifies
> > that
> > > > instance.volumes is set.
> > > >
> > > > Does anybody have a different opinion on this?
> > > >
> > > > --
> > > > Christopher L Tubbs II
> > > > http://gravatar.com/ctubbsii
> > > >
> > >
> >
> >

Re: Review Request 28873: ACCUMULO-3393 Follow-on work for per-table volume chooser.

[Josh Elser]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28873/#review64737
---

Ship it!


LGTM otherwise.


server/base/src/main/java/org/apache/accumulo/server/fs/PreferredVolumeChooser.java


Should use StringUtils (either from Hadoop or commons-lang) instead of the 
regex-based implementation on String.


- Josh Elser


On Dec. 11, 2014, 8:14 a.m., Sean Busbey wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28873/
> ---
> 
> (Updated Dec. 11, 2014, 8:14 a.m.)
> 
> 
> Review request for accumulo and Jenna Huston.
> 
> 
> Bugs: ACCUMULO-3393
> https://issues.apache.org/jira/browse/ACCUMULO-3393
> 
> 
> Repository: accumulo
> 
> 
> Description
> ---
> 
> ACCUMULO-3393 Follow-on work for per-table volume chooser.
> 
> Still has TODOs for additional tickets I need to file. I'll update the review 
> to remove once I have them all filed. I think everything marked TODO can wait 
> for a later ticket. Please comment on relevant section if you think something 
> needs to be done now.
> 
> * docs clean up + code guideline compliance.
> * ensure RandomVolumeChoosers are independent when used per-table.
> * make sure that per-table choosers can keep state the way that global 
> choosers can
> * make sure that a chooser can only pick from the options it is presented.
> * reduce object creation in critical path for Tablet.split.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/accumulo/core/conf/Property.java 
> 4c2d0b41b3bce32449861da3ac42fa27deb2b182 
>   
> server/base/src/main/java/org/apache/accumulo/server/fs/PerTableVolumeChooser.java
>  7a825c796eb5a25de8c748e2aba642f483697b9a 
>   
> server/base/src/main/java/org/apache/accumulo/server/fs/PreferredVolumeChooser.java
>  7ed7bba809a4e5e4b2d472c3327b15adb37251a7 
>   
> server/base/src/main/java/org/apache/accumulo/server/fs/RandomVolumeChooser.java
>  f2eb2113cb848ed58ac5f41573c6ff2cde9b0a77 
>   server/base/src/main/java/org/apache/accumulo/server/fs/VolumeChooser.java 
> f523057b11a2dc42e82010675bb1ac8e3802f96d 
>   server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManager.java 
> 890651e92f4c34514cb839b7b9ee9d23ad55070a 
>   
> server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManagerImpl.java
>  dc1be739b634d91992894f8f27c2d9c184bd98cd 
>   server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java 
> 877d01c2233cca086c1ac1539eb81cc282a7ceb4 
>   
> server/base/src/main/java/org/apache/accumulo/server/util/TabletOperations.java
>  c0e1a9b991d61a4dbb127c74ae297f171434e7d5 
>   
> server/base/src/test/java/org/apache/accumulo/server/fs/VolumeManagerImplTest.java
>  582822a8ccbc398925a2184eed0b9d7fa853f9b4 
> 
> Diff: https://reviews.apache.org/r/28873/diff/
> 
> 
> Testing
> ---
> 
> Ran through VolumeManagerImplTest and VolumeChooserIT 
> 
> 
> Thanks,
> 
> Sean Busbey
> 
>

Re: Planning for (eventual) removal of instance.dfs.{uri,dir}

[Josh Elser]

Christopher wrote:

So,

instance.volumes replaces instance.dfs.uri and instance.dfs.dir in 1.6.
But, what's our long-term plan for these? I ask, because we still have
internal code that uses instance.dfs.uri to resolve relative paths.

Should we force these to be re-written at some point (maybe on upgrade to
1.7)? Should we continue to support the deprecated properties indefinitely
and continue the lazy re-write-on-compact? Do we transition by requiring
instance.volumes to specify the volumes, and only use the old properties to
resolve relative paths?


+1 Making a utility that does the upgrade would be so nice over trying 
to get everything else right ourselves. 2.0 sounds like a good time to 
do it (since we deprecated the properties in 1.6.0)



My personal view is that we should provide an upgrade-prep/check tool prior
to upgrade to 2.0, which checks and/or re-writes paths and verifies that
instance.volumes is set.

Does anybody have a different opinion on this?

--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

Re: Planning for (eventual) removal of instance.dfs.{uri,dir}

[Josh Elser]

You're building functions over the location of files in HDFS? Automated 
configuration of instances?


Jeremy Kepner wrote:

When we remove functions, do we have any official guidance to our users who may
have built applications that use those functions?

Right now, the official position is that the Accumulo developers can remove
as they see fit.  However, this provides no guidance to users as to what
they are suppose to do?  As it stands, our guidance is that they have the 
following choices:

(0) Diligently watch the Accumulo e-mail list and aggressively weigh in on
any vote to remove functions that may impact them.

(1) Find someone to modify the original source code of their applications,
build it, and *re-verify* the application.  I emphasise the re-verify because
that is usually the most costly part of the process that often won't get 
approved
by management.

(2) Don't upgrade Accumulo.





On Thu, Dec 11, 2014 at 10:03:56AM -0500, Christopher wrote:

Well, no, the database will start if we rely on instance.volumes, but we
may be unable to find files that have relative paths, if we incorrectly
assume /accumulo. Making it required is annoying if users don't have
relative paths.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Thu, Dec 11, 2014 at 8:15 AM,  wrote:


How so? If someone upgrades from another version and is using a different
dir, doesn't specify it in the configuration, and we assume /accumulo, then
their database won't start. The other option, which may be safer than
making any assumptions, is to make instance.volumes a required parameter
with no defaults.

- Original Message -

From: "Christopher"
To: "Accumulo Dev List"
Sent: Wednesday, December 10, 2014 11:51:39 PM
Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir}

The URI is probably reasonable, but the dir is potentially problematic if
you weren't using the default.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Wed, Dec 10, 2014 at 10:03 PM, dlmarion  wrote:


Looks like VolumeConfiguration falls back to fs.defaultFS for the uri and
/accumulo for the dir. You could remove both properties and still keep

this

as the documented default behavior if instance.volumes is not specified.



 Original message From: Christopher<
ctubb...@apache.org>  Date:12/10/2014 9:13 PM (GMT-05:00)
To: Accumulo Dev List
Cc:Subject: Re: Planning for (eventual) removal of
instance.dfs.{uri,dir}
My ACCUMULO-2589 branch in github (
https://github.com/ctubbsii/accumulo/tree/ACCUMULO-2589) does have a
commit
that drops a bunch of stuff (which may or may not be accepted as is for
2.0). The instance.dfs.{uri,dir} properties aren't so easy and require

more

planning, because it's not just removing a property... it's also dealing
with updating internal data by making relative paths absolute.

For what it's worth, I'm also looking at what changes if we drop Hadoop 1
support.

As for the validation of config, I think we do a sanity check on startup
already, which we can always extend. Doesn't solve this issue, though.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Wed, Dec 10, 2014 at 8:59 PM, dlmarion  wrote:


We should schedule a bunch of deprecated things for removal in 2.0 to

ease

maintenance. Do we have a way to validate the site.xml and zookeeper
settings before startup and fail with appropriate error message.



 Original message From: Christopher<
ctubb...@apache.org>  Date:12/10/2014 8:44 PM (GMT-05:00)
To: Accumulo Dev List
Cc:Subject: Planning for (eventual) removal of
instance.dfs.{uri,dir}
So,

instance.volumes replaces instance.dfs.uri and instance.dfs.dir in 1.6.
But, what's our long-term plan for these? I ask, because we still have
internal code that uses instance.dfs.uri to resolve relative paths.

Should we force these to be re-written at some point (maybe on upgrade

to

1.7)? Should we continue to support the deprecated properties

indefinitely

and continue the lazy re-write-on-compact? Do we transition by

requiring

instance.volumes to specify the volumes, and only use the old

properties

to

resolve relative paths?

My personal view is that we should provide an upgrade-prep/check tool

prior

to upgrade to 2.0, which checks and/or re-writes paths and verifies

that

instance.volumes is set.

Does anybody have a different opinion on this?

--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

Google analytics summaries

[Sean Busbey]

I just got the mail summary of our google analytics stats for
accumulo.apache.org.

Anyone opposed to me starting to forward the summary email to dev@accumulo?
It contains summary info like visitor count, countries, bounce rate, etc.

I figure it's an easy starting point to make more of our community stats
accessible.

-- 
Sean

Re: Planning for (eventual) removal of instance.dfs.{uri,dir}

[dlmarion]

It has been my experience that testing is always required when upgrading a 
major component. With the vote to adopt semver, our users will have more of a 
guarantee that their applications will work through patch and minor version 
upgrades. It might be useful though to have a document like CHANGES, but 
contain the items that have been removed in the major release and a note on 
what (if anything) is replacing the removed function. 


- Original Message -

From: "Jeremy Kepner"  
To: dev@accumulo.apache.org 
Sent: Thursday, December 11, 2014 10:44:21 AM 
Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir} 

When we remove functions, do we have any official guidance to our users who may 
have built applications that use those functions? 

Right now, the official position is that the Accumulo developers can remove 
as they see fit. However, this provides no guidance to users as to what 
they are suppose to do? As it stands, our guidance is that they have the 
following choices: 

(0) Diligently watch the Accumulo e-mail list and aggressively weigh in on 
any vote to remove functions that may impact them. 

(1) Find someone to modify the original source code of their applications, 
build it, and *re-verify* the application. I emphasise the re-verify because 
that is usually the most costly part of the process that often won't get 
approved 
by management. 

(2) Don't upgrade Accumulo. 





On Thu, Dec 11, 2014 at 10:03:56AM -0500, Christopher wrote: 
> Well, no, the database will start if we rely on instance.volumes, but we 
> may be unable to find files that have relative paths, if we incorrectly 
> assume /accumulo. Making it required is annoying if users don't have 
> relative paths. 
> 
> 
> -- 
> Christopher L Tubbs II 
> http://gravatar.com/ctubbsii 
> 
> On Thu, Dec 11, 2014 at 8:15 AM,  wrote: 
> 
> > How so? If someone upgrades from another version and is using a different 
> > dir, doesn't specify it in the configuration, and we assume /accumulo, then 
> > their database won't start. The other option, which may be safer than 
> > making any assumptions, is to make instance.volumes a required parameter 
> > with no defaults. 
> > 
> > - Original Message - 
> > 
> > From: "Christopher"  
> > To: "Accumulo Dev List"  
> > Sent: Wednesday, December 10, 2014 11:51:39 PM 
> > Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir} 
> > 
> > The URI is probably reasonable, but the dir is potentially problematic if 
> > you weren't using the default. 
> > 
> > 
> > -- 
> > Christopher L Tubbs II 
> > http://gravatar.com/ctubbsii 
> > 
> > On Wed, Dec 10, 2014 at 10:03 PM, dlmarion  wrote: 
> > 
> > > Looks like VolumeConfiguration falls back to fs.defaultFS for the uri and 
> > > /accumulo for the dir. You could remove both properties and still keep 
> > this 
> > > as the documented default behavior if instance.volumes is not specified. 
> > > 
> > > 
> > > 
> > >  Original message From: Christopher < 
> > > ctubb...@apache.org> Date:12/10/2014 9:13 PM (GMT-05:00) 
> > > To: Accumulo Dev List  
> > > Cc: Subject: Re: Planning for (eventual) removal of 
> > > instance.dfs.{uri,dir}  
> > > My ACCUMULO-2589 branch in github ( 
> > > https://github.com/ctubbsii/accumulo/tree/ACCUMULO-2589) does have a 
> > > commit 
> > > that drops a bunch of stuff (which may or may not be accepted as is for 
> > > 2.0). The instance.dfs.{uri,dir} properties aren't so easy and require 
> > more 
> > > planning, because it's not just removing a property... it's also dealing 
> > > with updating internal data by making relative paths absolute. 
> > > 
> > > For what it's worth, I'm also looking at what changes if we drop Hadoop 1 
> > > support. 
> > > 
> > > As for the validation of config, I think we do a sanity check on startup 
> > > already, which we can always extend. Doesn't solve this issue, though. 
> > > 
> > > 
> > > -- 
> > > Christopher L Tubbs II 
> > > http://gravatar.com/ctubbsii 
> > > 
> > > On Wed, Dec 10, 2014 at 8:59 PM, dlmarion  wrote: 
> > > 
> > > > We should schedule a bunch of deprecated things for removal in 2.0 to 
> > > ease 
> > > > maintenance. Do we have a way to validate the site.xml and zookeeper 
> > > > settings before startup and fail with appropriate error message. 
> > > > 
> > > > 
> > > > 
> > > >  Original message From: Christopher < 
> > > > ctubb...@apache.org> Date:12/10/2014 8:44 PM (GMT-05:00) 
> > > > To: Accumulo Dev List  
> > > > Cc: Subject: Planning for (eventual) removal of 
> > > > instance.dfs.{uri,dir}  
> > > > So, 
> > > > 
> > > > instance.volumes replaces instance.dfs.uri and instance.dfs.dir in 1.6. 
> > > > But, what's our long-term plan for these? I ask, because we still have 
> > > > internal code that uses instance.dfs.uri to resolve relative paths. 
> > > > 
> > > > Should we force these to be re-written at some point (maybe on upgrade 
> > to 
> > > > 1.7)? Should we continue 

Re: Planning for (eventual) removal of instance.dfs.{uri,dir}

[Christopher]

On Thu, Dec 11, 2014 at 10:52 AM,  wrote:

> "Making it required is annoying if users don't have relative paths."
>
> But this property is used to determine the location of new files.
>
>
No, it's not. Not if you're using instance.volumes. It should only be used
for resolving old files in that case.


> - Original Message -
>
> From: "Christopher" 
> To: "Accumulo Dev List" 
> Sent: Thursday, December 11, 2014 10:03:56 AM
> Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir}
>
> Well, no, the database will start if we rely on instance.volumes, but we
> may be unable to find files that have relative paths, if we incorrectly
> assume /accumulo. Making it required is annoying if users don't have
> relative paths.
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
> On Thu, Dec 11, 2014 at 8:15 AM,  wrote:
>
> > How so? If someone upgrades from another version and is using a different
> > dir, doesn't specify it in the configuration, and we assume /accumulo,
> then
> > their database won't start. The other option, which may be safer than
> > making any assumptions, is to make instance.volumes a required parameter
> > with no defaults.
> >
> > - Original Message -
> >
> > From: "Christopher" 
> > To: "Accumulo Dev List" 
> > Sent: Wednesday, December 10, 2014 11:51:39 PM
> > Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir}
> >
> > The URI is probably reasonable, but the dir is potentially problematic if
> > you weren't using the default.
> >
> >
> > --
> > Christopher L Tubbs II
> > http://gravatar.com/ctubbsii
> >
> > On Wed, Dec 10, 2014 at 10:03 PM, dlmarion  wrote:
> >
> > > Looks like VolumeConfiguration falls back to fs.defaultFS for the uri
> and
> > > /accumulo for the dir. You could remove both properties and still keep
> > this
> > > as the documented default behavior if instance.volumes is not
> specified.
> > >
> > >
> > >
> > >  Original message From: Christopher <
> > > ctubb...@apache.org> Date:12/10/2014 9:13 PM (GMT-05:00)
> > > To: Accumulo Dev List 
> > > Cc: Subject: Re: Planning for (eventual) removal
> of
> > > instance.dfs.{uri,dir} 
> > > My ACCUMULO-2589 branch in github (
> > > https://github.com/ctubbsii/accumulo/tree/ACCUMULO-2589) does have a
> > > commit
> > > that drops a bunch of stuff (which may or may not be accepted as is for
> > > 2.0). The instance.dfs.{uri,dir} properties aren't so easy and require
> > more
> > > planning, because it's not just removing a property... it's also
> dealing
> > > with updating internal data by making relative paths absolute.
> > >
> > > For what it's worth, I'm also looking at what changes if we drop
> Hadoop 1
> > > support.
> > >
> > > As for the validation of config, I think we do a sanity check on
> startup
> > > already, which we can always extend. Doesn't solve this issue, though.
> > >
> > >
> > > --
> > > Christopher L Tubbs II
> > > http://gravatar.com/ctubbsii
> > >
> > > On Wed, Dec 10, 2014 at 8:59 PM, dlmarion 
> wrote:
> > >
> > > > We should schedule a bunch of deprecated things for removal in 2.0 to
> > > ease
> > > > maintenance. Do we have a way to validate the site.xml and zookeeper
> > > > settings before startup and fail with appropriate error message.
> > > >
> > > >
> > > >
> > > >  Original message From: Christopher <
> > > > ctubb...@apache.org> Date:12/10/2014 8:44 PM (GMT-05:00)
> > > > To: Accumulo Dev List 
> > > > Cc: Subject: Planning for (eventual) removal of
> > > > instance.dfs.{uri,dir} 
> > > > So,
> > > >
> > > > instance.volumes replaces instance.dfs.uri and instance.dfs.dir in
> 1.6.
> > > > But, what's our long-term plan for these? I ask, because we still
> have
> > > > internal code that uses instance.dfs.uri to resolve relative paths.
> > > >
> > > > Should we force these to be re-written at some point (maybe on
> upgrade
> > to
> > > > 1.7)? Should we continue to support the deprecated properties
> > > indefinitely
> > > > and continue the lazy re-write-on-compact? Do we transition by
> > requiring
> > > > instance.volumes to specify the volumes, and only use the old
> > properties
> > > to
> > > > resolve relative paths?
> > > >
> > > > My personal view is that we should provide an upgrade-prep/check tool
> > > prior
> > > > to upgrade to 2.0, which checks and/or re-writes paths and verifies
> > that
> > > > instance.volumes is set.
> > > >
> > > > Does anybody have a different opinion on this?
> > > >
> > > > --
> > > > Christopher L Tubbs II
> > > > http://gravatar.com/ctubbsii
> > > >
> > >
> >
> >
>
>

Re: Google analytics summaries

[Christopher]

No objection here. Either here or notifications@ would be fine.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Thu, Dec 11, 2014 at 11:14 AM, Sean Busbey  wrote:

> I just got the mail summary of our google analytics stats for
> accumulo.apache.org.
>
> Anyone opposed to me starting to forward the summary email to dev@accumulo
> ?
> It contains summary info like visitor count, countries, bounce rate, etc.
>
> I figure it's an easy starting point to make more of our community stats
> accessible.
>
> --
> Sean
>

Re: Planning for (eventual) removal of instance.dfs.{uri,dir}

[Josh Elser]

To be clear -- if you want to address actual problems you've seen WRT 
compatibility, start up a new thread and we can work through the issues 
you have seen and what you are fearful of.


I'm not seeing the relevance to this discussion and don't want it to get 
derailed because dealing with the removal of the instance.dfs.* stuff is 
important for us to handle properly.


Josh Elser wrote:

You're building functions over the location of files in HDFS? Automated
configuration of instances?

Jeremy Kepner wrote:

When we remove functions, do we have any official guidance to our
users who may
have built applications that use those functions?

Right now, the official position is that the Accumulo developers can
remove
as they see fit. However, this provides no guidance to users as to what
they are suppose to do? As it stands, our guidance is that they have
the following choices:

(0) Diligently watch the Accumulo e-mail list and aggressively weigh
in on
any vote to remove functions that may impact them.

(1) Find someone to modify the original source code of their
applications,
build it, and *re-verify* the application. I emphasise the re-verify
because
that is usually the most costly part of the process that often won't
get approved
by management.

(2) Don't upgrade Accumulo.





On Thu, Dec 11, 2014 at 10:03:56AM -0500, Christopher wrote:

Well, no, the database will start if we rely on instance.volumes, but we
may be unable to find files that have relative paths, if we incorrectly
assume /accumulo. Making it required is annoying if users don't have
relative paths.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Thu, Dec 11, 2014 at 8:15 AM, wrote:


How so? If someone upgrades from another version and is using a
different
dir, doesn't specify it in the configuration, and we assume
/accumulo, then
their database won't start. The other option, which may be safer than
making any assumptions, is to make instance.volumes a required
parameter
with no defaults.

- Original Message -

From: "Christopher"
To: "Accumulo Dev List"
Sent: Wednesday, December 10, 2014 11:51:39 PM
Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir}

The URI is probably reasonable, but the dir is potentially
problematic if
you weren't using the default.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Wed, Dec 10, 2014 at 10:03 PM, dlmarion wrote:


Looks like VolumeConfiguration falls back to fs.defaultFS for the
uri and
/accumulo for the dir. You could remove both properties and still keep

this

as the documented default behavior if instance.volumes is not
specified.



 Original message From: Christopher<
ctubb...@apache.org> Date:12/10/2014 9:13 PM (GMT-05:00)
To: Accumulo Dev List
Cc:Subject: Re: Planning for (eventual)
removal of
instance.dfs.{uri,dir}
My ACCUMULO-2589 branch in github (
https://github.com/ctubbsii/accumulo/tree/ACCUMULO-2589) does have a
commit
that drops a bunch of stuff (which may or may not be accepted as is
for
2.0). The instance.dfs.{uri,dir} properties aren't so easy and require

more

planning, because it's not just removing a property... it's also
dealing
with updating internal data by making relative paths absolute.

For what it's worth, I'm also looking at what changes if we drop
Hadoop 1
support.

As for the validation of config, I think we do a sanity check on
startup
already, which we can always extend. Doesn't solve this issue, though.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Wed, Dec 10, 2014 at 8:59 PM, dlmarion wrote:


We should schedule a bunch of deprecated things for removal in 2.0 to

ease

maintenance. Do we have a way to validate the site.xml and zookeeper
settings before startup and fail with appropriate error message.



 Original message From: Christopher<
ctubb...@apache.org> Date:12/10/2014 8:44 PM (GMT-05:00)
To: Accumulo Dev List
Cc:Subject: Planning for (eventual) removal of
instance.dfs.{uri,dir}
So,

instance.volumes replaces instance.dfs.uri and instance.dfs.dir in
1.6.
But, what's our long-term plan for these? I ask, because we still
have
internal code that uses instance.dfs.uri to resolve relative paths.

Should we force these to be re-written at some point (maybe on
upgrade

to

1.7)? Should we continue to support the deprecated properties

indefinitely

and continue the lazy re-write-on-compact? Do we transition by

requiring

instance.volumes to specify the volumes, and only use the old

properties

to

resolve relative paths?

My personal view is that we should provide an upgrade-prep/check tool

prior

to upgrade to 2.0, which checks and/or re-writes paths and verifies

that

instance.volumes is set.

Does anybody have a different opinion on this?

--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

Re: Google analytics summaries

[Billie Rinaldi]

Sounds good to me.
On Dec 11, 2014 11:15 AM, "Sean Busbey"  wrote:

> I just got the mail summary of our google analytics stats for
> accumulo.apache.org.
>
> Anyone opposed to me starting to forward the summary email to dev@accumulo
> ?
> It contains summary info like visitor count, countries, bounce rate, etc.
>
> I figure it's an easy starting point to make more of our community stats
> accessible.
>
> --
> Sean
>

Re: Planning for (eventual) removal of instance.dfs.{uri,dir}

[dlmarion]

I think we crossed the streams. I'll talk to you offline. 


- Original Message -

From: "Christopher"  
To: "Accumulo Dev List"  
Sent: Thursday, December 11, 2014 11:27:48 AM 
Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir} 

On Thu, Dec 11, 2014 at 10:52 AM,  wrote: 

> "Making it required is annoying if users don't have relative paths." 
> 
> But this property is used to determine the location of new files. 
> 
> 
No, it's not. Not if you're using instance.volumes. It should only be used 
for resolving old files in that case. 


> - Original Message - 
> 
> From: "Christopher"  
> To: "Accumulo Dev List"  
> Sent: Thursday, December 11, 2014 10:03:56 AM 
> Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir} 
> 
> Well, no, the database will start if we rely on instance.volumes, but we 
> may be unable to find files that have relative paths, if we incorrectly 
> assume /accumulo. Making it required is annoying if users don't have 
> relative paths. 
> 
> 
> -- 
> Christopher L Tubbs II 
> http://gravatar.com/ctubbsii 
> 
> On Thu, Dec 11, 2014 at 8:15 AM,  wrote: 
> 
> > How so? If someone upgrades from another version and is using a different 
> > dir, doesn't specify it in the configuration, and we assume /accumulo, 
> then 
> > their database won't start. The other option, which may be safer than 
> > making any assumptions, is to make instance.volumes a required parameter 
> > with no defaults. 
> > 
> > - Original Message - 
> > 
> > From: "Christopher"  
> > To: "Accumulo Dev List"  
> > Sent: Wednesday, December 10, 2014 11:51:39 PM 
> > Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir} 
> > 
> > The URI is probably reasonable, but the dir is potentially problematic if 
> > you weren't using the default. 
> > 
> > 
> > -- 
> > Christopher L Tubbs II 
> > http://gravatar.com/ctubbsii 
> > 
> > On Wed, Dec 10, 2014 at 10:03 PM, dlmarion  wrote: 
> > 
> > > Looks like VolumeConfiguration falls back to fs.defaultFS for the uri 
> and 
> > > /accumulo for the dir. You could remove both properties and still keep 
> > this 
> > > as the documented default behavior if instance.volumes is not 
> specified. 
> > > 
> > > 
> > > 
> > >  Original message From: Christopher < 
> > > ctubb...@apache.org> Date:12/10/2014 9:13 PM (GMT-05:00) 
> > > To: Accumulo Dev List  
> > > Cc: Subject: Re: Planning for (eventual) removal 
> of 
> > > instance.dfs.{uri,dir}  
> > > My ACCUMULO-2589 branch in github ( 
> > > https://github.com/ctubbsii/accumulo/tree/ACCUMULO-2589) does have a 
> > > commit 
> > > that drops a bunch of stuff (which may or may not be accepted as is for 
> > > 2.0). The instance.dfs.{uri,dir} properties aren't so easy and require 
> > more 
> > > planning, because it's not just removing a property... it's also 
> dealing 
> > > with updating internal data by making relative paths absolute. 
> > > 
> > > For what it's worth, I'm also looking at what changes if we drop 
> Hadoop 1 
> > > support. 
> > > 
> > > As for the validation of config, I think we do a sanity check on 
> startup 
> > > already, which we can always extend. Doesn't solve this issue, though. 
> > > 
> > > 
> > > -- 
> > > Christopher L Tubbs II 
> > > http://gravatar.com/ctubbsii 
> > > 
> > > On Wed, Dec 10, 2014 at 8:59 PM, dlmarion  
> wrote: 
> > > 
> > > > We should schedule a bunch of deprecated things for removal in 2.0 to 
> > > ease 
> > > > maintenance. Do we have a way to validate the site.xml and zookeeper 
> > > > settings before startup and fail with appropriate error message. 
> > > > 
> > > > 
> > > > 
> > > >  Original message From: Christopher < 
> > > > ctubb...@apache.org> Date:12/10/2014 8:44 PM (GMT-05:00) 
> > > > To: Accumulo Dev List  
> > > > Cc: Subject: Planning for (eventual) removal of 
> > > > instance.dfs.{uri,dir}  
> > > > So, 
> > > > 
> > > > instance.volumes replaces instance.dfs.uri and instance.dfs.dir in 
> 1.6. 
> > > > But, what's our long-term plan for these? I ask, because we still 
> have 
> > > > internal code that uses instance.dfs.uri to resolve relative paths. 
> > > > 
> > > > Should we force these to be re-written at some point (maybe on 
> upgrade 
> > to 
> > > > 1.7)? Should we continue to support the deprecated properties 
> > > indefinitely 
> > > > and continue the lazy re-write-on-compact? Do we transition by 
> > requiring 
> > > > instance.volumes to specify the volumes, and only use the old 
> > properties 
> > > to 
> > > > resolve relative paths? 
> > > > 
> > > > My personal view is that we should provide an upgrade-prep/check tool 
> > > prior 
> > > > to upgrade to 2.0, which checks and/or re-writes paths and verifies 
> > that 
> > > > instance.volumes is set. 
> > > > 
> > > > Does anybody have a different opinion on this? 
> > > > 
> > > > -- 
> > > > Christopher L Tubbs II 
> > > > http://gravatar.com/ctubbsii 
> > > > 
> > > 
> > 
> > 
> 

Re: Google analytics summaries

[Adam Fuchs]

I'm for it.

Also, does anyone know how I get access to the google analytics
reports page for our site?

Thanks,
Adam


On Thu, Dec 11, 2014 at 11:14 AM, Sean Busbey  wrote:
> I just got the mail summary of our google analytics stats for
> accumulo.apache.org.
>
> Anyone opposed to me starting to forward the summary email to dev@accumulo?
> It contains summary info like visitor count, countries, bounce rate, etc.
>
> I figure it's an easy starting point to make more of our community stats
> accessible.
>
> --
> Sean

Official Guidance to users regarding removal of functions

[Kepner, Jeremy - 0553 - MITLL]

When we remove functions, do we have any official guidance to our users who may 
have built applications that use those functions?

Right now, the official position is that the Accumulo developers can remove 
based on a consensus vote. However, this provides no guidance to users as to 
what they are suppose to do? As it stands, our guidance is that they have the 
following choices:

(0) Diligently watch the Accumulo e-mail list and aggressively weigh in on any 
vote to remove functions that may impact them.

(1) Find someone to modify the original source code of their applications, 
build it, and *re-verify* the application. I emphasise the re-verify because 
that is usually the most costly part of the process that often won't get 
approved by management.

(2) Don't upgrade Accumulo.

Re: Official Guidance to users regarding removal of functions

[John Vines]

Wouldn't this be resolved with our SemVer sqwitch?

On Thu, Dec 11, 2014 at 11:36 AM, Kepner, Jeremy - 0553 - MITLL <
kep...@ll.mit.edu> wrote:

> When we remove functions, do we have any official guidance to our users
> who may have built applications that use those functions?
>
> Right now, the official position is that the Accumulo developers can
> remove based on a consensus vote. However, this provides no guidance to
> users as to what they are suppose to do? As it stands, our guidance is that
> they have the following choices:
>
> (0) Diligently watch the Accumulo e-mail list and aggressively weigh in on
> any vote to remove functions that may impact them.
>
> (1) Find someone to modify the original source code of their applications,
> build it, and *re-verify* the application. I emphasise the re-verify
> because that is usually the most costly part of the process that often
> won't get approved by management.
>
> (2) Don't upgrade Accumulo.
>

Re: Official Guidance to users regarding removal of functions

[Benson Margulies]

How do you see this as different from any other major open source platform?
Every one I've ever worked on does deprecate and remove things; generally
deprecation is accompanied by comments and release notes offering
adaptation advice.



On Thu, Dec 11, 2014 at 11:36 AM, Kepner, Jeremy - 0553 - MITLL <
kep...@ll.mit.edu> wrote:

> When we remove functions, do we have any official guidance to our users
> who may have built applications that use those functions?
>
> Right now, the official position is that the Accumulo developers can
> remove based on a consensus vote. However, this provides no guidance to
> users as to what they are suppose to do? As it stands, our guidance is that
> they have the following choices:
>
> (0) Diligently watch the Accumulo e-mail list and aggressively weigh in on
> any vote to remove functions that may impact them.
>
> (1) Find someone to modify the original source code of their applications,
> build it, and *re-verify* the application. I emphasise the re-verify
> because that is usually the most costly part of the process that often
> won't get approved by management.
>
> (2) Don't upgrade Accumulo.
>

Re: Official Guidance to users regarding removal of functions

[Sean Busbey]

Jeremy,

It is my understanding that our practice is to flag to-be-removed API
components with Deprecation annotations and a note for the replacement.
That notice should be around for a full major version[1], though obviously
that's of limited utility if the lifespan of the major release is short.

Right now, part of our release process is checking that the API changes
aren't "painful" (ref step 6 for major releases on [1]). What that means
isn't formally called out, but I'd expect a minimum would be that the above
warning happened.

Would expanding that guidance to include calling out removals (and their
replacements) in the release notes help your concerns?

What if we had a step where such changes were posted to user@accumulo in
advance of the release? That would give impacted users a chance to give
feedback without having to watch dev@ or commits@.

FYI, in the future you'll get better traction on these kinds of threads if
you start the subject line with "[DISCUSS]".

[1]: http://accumulo.apache.org/governance/releasing.html

On Thu, Dec 11, 2014 at 10:36 AM, Kepner, Jeremy - 0553 - MITLL <
kep...@ll.mit.edu> wrote:

> When we remove functions, do we have any official guidance to our users
> who may have built applications that use those functions?
>
> Right now, the official position is that the Accumulo developers can
> remove based on a consensus vote. However, this provides no guidance to
> users as to what they are suppose to do? As it stands, our guidance is that
> they have the following choices:
>
> (0) Diligently watch the Accumulo e-mail list and aggressively weigh in on
> any vote to remove functions that may impact them.
>
> (1) Find someone to modify the original source code of their applications,
> build it, and *re-verify* the application. I emphasise the re-verify
> because that is usually the most costly part of the process that often
> won't get approved by management.
>
> (2) Don't upgrade Accumulo.
>



-- 
Sean

Re: Official Guidance to users regarding removal of functions

[Billie Rinaldi]

To clarify John's question: if our vote to adopt semver 2.0.0 passes, our
intention will be to no longer have breaking public API changes unless the
major version number is incremented, i.e. 1.x.x -> 2.x.x. An important
aspect of semantic versioning is defining what is considered part of the
public API. So if there are things you need to remain consistent that are
not covered by Section 9 of the README, we should discuss adding them.
Actually, strengthening what we consider to be the public API is likely to
be a separate conversation in which (I hope) we will engage the user list.
On Dec 11, 2014 11:51 AM, "John Vines"  wrote:

> Wouldn't this be resolved with our SemVer sqwitch?
>
> On Thu, Dec 11, 2014 at 11:36 AM, Kepner, Jeremy - 0553 - MITLL <
> kep...@ll.mit.edu> wrote:
>
> > When we remove functions, do we have any official guidance to our users
> > who may have built applications that use those functions?
> >
> > Right now, the official position is that the Accumulo developers can
> > remove based on a consensus vote. However, this provides no guidance to
> > users as to what they are suppose to do? As it stands, our guidance is
> that
> > they have the following choices:
> >
> > (0) Diligently watch the Accumulo e-mail list and aggressively weigh in
> on
> > any vote to remove functions that may impact them.
> >
> > (1) Find someone to modify the original source code of their
> applications,
> > build it, and *re-verify* the application. I emphasise the re-verify
> > because that is usually the most costly part of the process that often
> > won't get approved by management.
> >
> > (2) Don't upgrade Accumulo.
> >
>

Re: Official Guidance to users regarding removal of functions

[Mike Drob]

Billie,

Not to be glib, but it reads like your suggestion to Jeremy for when we
have a 2.0.0 release (assuming semver passes) is to take option (2) Don't
upgrade Accumulo.

Please correct my misunderstanding.

Mike

On Thu, Dec 11, 2014 at 11:07 AM, Billie Rinaldi  wrote:

> To clarify John's question: if our vote to adopt semver 2.0.0 passes, our
> intention will be to no longer have breaking public API changes unless the
> major version number is incremented, i.e. 1.x.x -> 2.x.x. An important
> aspect of semantic versioning is defining what is considered part of the
> public API. So if there are things you need to remain consistent that are
> not covered by Section 9 of the README, we should discuss adding them.
> Actually, strengthening what we consider to be the public API is likely to
> be a separate conversation in which (I hope) we will engage the user list.
> On Dec 11, 2014 11:51 AM, "John Vines"  wrote:
>
> > Wouldn't this be resolved with our SemVer sqwitch?
> >
> > On Thu, Dec 11, 2014 at 11:36 AM, Kepner, Jeremy - 0553 - MITLL <
> > kep...@ll.mit.edu> wrote:
> >
> > > When we remove functions, do we have any official guidance to our users
> > > who may have built applications that use those functions?
> > >
> > > Right now, the official position is that the Accumulo developers can
> > > remove based on a consensus vote. However, this provides no guidance to
> > > users as to what they are suppose to do? As it stands, our guidance is
> > that
> > > they have the following choices:
> > >
> > > (0) Diligently watch the Accumulo e-mail list and aggressively weigh in
> > on
> > > any vote to remove functions that may impact them.
> > >
> > > (1) Find someone to modify the original source code of their
> > applications,
> > > build it, and *re-verify* the application. I emphasise the re-verify
> > > because that is usually the most costly part of the process that often
> > > won't get approved by management.
> > >
> > > (2) Don't upgrade Accumulo.
> > >
> >
>

Re: Official Guidance to users regarding removal of functions

[Benson Margulies]

On Thu, Dec 11, 2014 at 12:11 PM, Mike Drob  wrote:

> Billie,
>
> Not to be glib, but it reads like your suggestion to Jeremy for when we
> have a 2.0.0 release (assuming semver passes) is to take option (2) Don't
> upgrade Accumulo.
>
> Please correct my misunderstanding.
>


That is perfectly fine advice in my view. In stable, long-life products,
major versions are maintained for a _long_ time. Applications rarely move
to new major versions. (I'm using Apache Lucene as a point of comparison.)
So, once an application goes into production on x.y.0, its maintainers
expect that there will be bug fixes and compatible improvements for for
_years_. Moving to a new major version is a big deal, and no one is going
to do it routinely or lightly. Many applications will reach the end of
their own EOL first. There are people still using Apache Lucene 3.x, and
the community is still prepared to make security fixes. The community will
be making functional improvements to 4.x for a very long time, even as
they/we are on the verge of shipping 5.0.


>
> Mike
>
> On Thu, Dec 11, 2014 at 11:07 AM, Billie Rinaldi 
> wrote:
>
> > To clarify John's question: if our vote to adopt semver 2.0.0 passes, our
> > intention will be to no longer have breaking public API changes unless
> the
> > major version number is incremented, i.e. 1.x.x -> 2.x.x. An important
> > aspect of semantic versioning is defining what is considered part of the
> > public API. So if there are things you need to remain consistent that are
> > not covered by Section 9 of the README, we should discuss adding them.
> > Actually, strengthening what we consider to be the public API is likely
> to
> > be a separate conversation in which (I hope) we will engage the user
> list.
> > On Dec 11, 2014 11:51 AM, "John Vines"  wrote:
> >
> > > Wouldn't this be resolved with our SemVer sqwitch?
> > >
> > > On Thu, Dec 11, 2014 at 11:36 AM, Kepner, Jeremy - 0553 - MITLL <
> > > kep...@ll.mit.edu> wrote:
> > >
> > > > When we remove functions, do we have any official guidance to our
> users
> > > > who may have built applications that use those functions?
> > > >
> > > > Right now, the official position is that the Accumulo developers can
> > > > remove based on a consensus vote. However, this provides no guidance
> to
> > > > users as to what they are suppose to do? As it stands, our guidance
> is
> > > that
> > > > they have the following choices:
> > > >
> > > > (0) Diligently watch the Accumulo e-mail list and aggressively weigh
> in
> > > on
> > > > any vote to remove functions that may impact them.
> > > >
> > > > (1) Find someone to modify the original source code of their
> > > applications,
> > > > build it, and *re-verify* the application. I emphasise the re-verify
> > > > because that is usually the most costly part of the process that
> often
> > > > won't get approved by management.
> > > >
> > > > (2) Don't upgrade Accumulo.
> > > >
> > >
> >
>

Re: Official Guidance to users regarding removal of functions

[Christopher]

On Thu, Dec 11, 2014 at 11:36 AM, Kepner, Jeremy - 0553 - MITLL <
kep...@ll.mit.edu> wrote:

> When we remove functions, do we have any official guidance to our users
> who may have built applications that use those functions?
>
>
I assume by "functions", you mean API, not functionality. To my knowledge,
Accumulo has never removed functionality (except where it was broken, as in
support for ViewFS between 1.6.0 and 1.6.1), only changed the way that
functionality is leveraged.


> Right now, the official position is that the Accumulo developers can
> remove based on a consensus vote. However, this provides no guidance to
> users as to what they are suppose to do? As it stands, our guidance is that
> they have the following choices:
>
>
Well, lazy consensus, yes, though we've taken care in the last few releases
to ensure more compatibility, and that has been brought up in the
release-candidate testing.


> (0) Diligently watch the Accumulo e-mail list and aggressively weigh in on
> any vote to remove functions that may impact them.
>
>
With the (expected) adoption of Semver to guide our release planning, we'll
be able to communicate changes which impact users much more clearly, so you
don't have to watch the mailing lists or commits to anticipate them. Of
course, if you want to provide feedback on change proposals, more active
involvement will still be necessary.


> (1) Find someone to modify the original source code of their applications,
> build it, and *re-verify* the application. I emphasise the re-verify
> because that is usually the most costly part of the process that often
> won't get approved by management.
>
>
With Semver, you won't need to do this for major versions. You'll be able
to rely on the fact that no removals will occur in that major version.
You'll only need to go through this if you want to jump major versions.
Obviously, that's a choice you'll have to make. You could also lobby for or
contribute updates you'd like for the major version you use, if you
want/need new features and you don't want to update a major version.


> (2) Don't upgrade Accumulo.
>

That's a choice you'd have to make based on your individual requirements.
What we can do as developers is help you make an informed decision, by
communicating what it means to change a version number. Keep in mind also,
that a new version which drops old API doesn't mean that you must upgrade.
The previous major version can still be supported for some time, with new
features and bug fixes in minor and patch releases, respectively (though
obviously, that depends on the degree to which contributors provide patches
for that version).


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

Re: Official Guidance to users regarding removal of functions

[Billie Rinaldi]

Actually, I wasn't suggesting anything.  I was providing elaboration on
what John was referring to.  I imagine that stronger API guarantees will be
cold comfort in the face of a 1.0 -> 2.0 upgrade.  However, if we had been
using semver all along, there would have been much less pain for users in
the 1.x series.  Also, adopting semver would mean that going from 1.6 to a
hypothetical 1.7 would not suffer from the same upgrade issues.  I doubt
that we could retroactively mitigate the differences in minor versions,
though, so going from 1.3/1.4/1.5 to 1.7 would still be hard.

On Thu, Dec 11, 2014 at 9:11 AM, Mike Drob  wrote:

> Billie,
>
> Not to be glib, but it reads like your suggestion to Jeremy for when we
> have a 2.0.0 release (assuming semver passes) is to take option (2) Don't
> upgrade Accumulo.
>
> Please correct my misunderstanding.
>
> Mike
>
> On Thu, Dec 11, 2014 at 11:07 AM, Billie Rinaldi 
> wrote:
>
> > To clarify John's question: if our vote to adopt semver 2.0.0 passes, our
> > intention will be to no longer have breaking public API changes unless
> the
> > major version number is incremented, i.e. 1.x.x -> 2.x.x. An important
> > aspect of semantic versioning is defining what is considered part of the
> > public API. So if there are things you need to remain consistent that are
> > not covered by Section 9 of the README, we should discuss adding them.
> > Actually, strengthening what we consider to be the public API is likely
> to
> > be a separate conversation in which (I hope) we will engage the user
> list.
> > On Dec 11, 2014 11:51 AM, "John Vines"  wrote:
> >
> > > Wouldn't this be resolved with our SemVer sqwitch?
> > >
> > > On Thu, Dec 11, 2014 at 11:36 AM, Kepner, Jeremy - 0553 - MITLL <
> > > kep...@ll.mit.edu> wrote:
> > >
> > > > When we remove functions, do we have any official guidance to our
> users
> > > > who may have built applications that use those functions?
> > > >
> > > > Right now, the official position is that the Accumulo developers can
> > > > remove based on a consensus vote. However, this provides no guidance
> to
> > > > users as to what they are suppose to do? As it stands, our guidance
> is
> > > that
> > > > they have the following choices:
> > > >
> > > > (0) Diligently watch the Accumulo e-mail list and aggressively weigh
> in
> > > on
> > > > any vote to remove functions that may impact them.
> > > >
> > > > (1) Find someone to modify the original source code of their
> > > applications,
> > > > build it, and *re-verify* the application. I emphasise the re-verify
> > > > because that is usually the most costly part of the process that
> often
> > > > won't get approved by management.
> > > >
> > > > (2) Don't upgrade Accumulo.
> > > >
> > >
> >
>

Re: Official Guidance to users regarding removal of functions

[Christopher]

I don't know that it'd be "cold comfort". We can continue to support 1.x
for some time, if we choose.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Thu, Dec 11, 2014 at 12:53 PM, Billie Rinaldi  wrote:

> Actually, I wasn't suggesting anything.  I was providing elaboration on
> what John was referring to.  I imagine that stronger API guarantees will be
> cold comfort in the face of a 1.0 -> 2.0 upgrade.  However, if we had been
> using semver all along, there would have been much less pain for users in
> the 1.x series.  Also, adopting semver would mean that going from 1.6 to a
> hypothetical 1.7 would not suffer from the same upgrade issues.  I doubt
> that we could retroactively mitigate the differences in minor versions,
> though, so going from 1.3/1.4/1.5 to 1.7 would still be hard.
>
> On Thu, Dec 11, 2014 at 9:11 AM, Mike Drob  wrote:
>
> > Billie,
> >
> > Not to be glib, but it reads like your suggestion to Jeremy for when we
> > have a 2.0.0 release (assuming semver passes) is to take option (2) Don't
> > upgrade Accumulo.
> >
> > Please correct my misunderstanding.
> >
> > Mike
> >
> > On Thu, Dec 11, 2014 at 11:07 AM, Billie Rinaldi 
> > wrote:
> >
> > > To clarify John's question: if our vote to adopt semver 2.0.0 passes,
> our
> > > intention will be to no longer have breaking public API changes unless
> > the
> > > major version number is incremented, i.e. 1.x.x -> 2.x.x. An important
> > > aspect of semantic versioning is defining what is considered part of
> the
> > > public API. So if there are things you need to remain consistent that
> are
> > > not covered by Section 9 of the README, we should discuss adding them.
> > > Actually, strengthening what we consider to be the public API is likely
> > to
> > > be a separate conversation in which (I hope) we will engage the user
> > list.
> > > On Dec 11, 2014 11:51 AM, "John Vines"  wrote:
> > >
> > > > Wouldn't this be resolved with our SemVer sqwitch?
> > > >
> > > > On Thu, Dec 11, 2014 at 11:36 AM, Kepner, Jeremy - 0553 - MITLL <
> > > > kep...@ll.mit.edu> wrote:
> > > >
> > > > > When we remove functions, do we have any official guidance to our
> > users
> > > > > who may have built applications that use those functions?
> > > > >
> > > > > Right now, the official position is that the Accumulo developers
> can
> > > > > remove based on a consensus vote. However, this provides no
> guidance
> > to
> > > > > users as to what they are suppose to do? As it stands, our guidance
> > is
> > > > that
> > > > > they have the following choices:
> > > > >
> > > > > (0) Diligently watch the Accumulo e-mail list and aggressively
> weigh
> > in
> > > > on
> > > > > any vote to remove functions that may impact them.
> > > > >
> > > > > (1) Find someone to modify the original source code of their
> > > > applications,
> > > > > build it, and *re-verify* the application. I emphasise the
> re-verify
> > > > > because that is usually the most costly part of the process that
> > often
> > > > > won't get approved by management.
> > > > >
> > > > > (2) Don't upgrade Accumulo.
> > > > >
> > > >
> > >
> >
>

Re: Review Request 28873: ACCUMULO-3393 Follow-on work for per-table volume chooser.

[Christopher Tubbs]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28873/#review64763
---

Ship it!


Current implementation looks pretty good.

Small issue with type safety on ArrayList.
I prefer to avoid the "N.B." notes in favor of plain English (ASF development 
is typically done in English, and this is a relatively obscure abbreviation for 
a Latin phrase.)
You may want to clean up the TODOs (create issues, as needed) before pushing.

I think the sanity check on the return from the chooser should be done under 
ACCUMULO-3400 for organizational reasons (and because it should be applied to 
1.6 also). No issue with the content of that change, though. If you push it 
under 3393, that's fine, it just partially satisfies 3400, and 3400 can be 
closed when the check is back-ported to 1.6.

No need to re-review for me as long as the type safety issue is fixed. The rest 
are nice to have.


server/base/src/main/java/org/apache/accumulo/server/fs/PreferredVolumeChooser.java


Needs diamond for type safety: new ArrayList<>


- Christopher Tubbs


On Dec. 11, 2014, 3:14 a.m., Sean Busbey wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28873/
> ---
> 
> (Updated Dec. 11, 2014, 3:14 a.m.)
> 
> 
> Review request for accumulo and Jenna Huston.
> 
> 
> Bugs: ACCUMULO-3393
> https://issues.apache.org/jira/browse/ACCUMULO-3393
> 
> 
> Repository: accumulo
> 
> 
> Description
> ---
> 
> ACCUMULO-3393 Follow-on work for per-table volume chooser.
> 
> Still has TODOs for additional tickets I need to file. I'll update the review 
> to remove once I have them all filed. I think everything marked TODO can wait 
> for a later ticket. Please comment on relevant section if you think something 
> needs to be done now.
> 
> * docs clean up + code guideline compliance.
> * ensure RandomVolumeChoosers are independent when used per-table.
> * make sure that per-table choosers can keep state the way that global 
> choosers can
> * make sure that a chooser can only pick from the options it is presented.
> * reduce object creation in critical path for Tablet.split.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/accumulo/core/conf/Property.java 
> 4c2d0b41b3bce32449861da3ac42fa27deb2b182 
>   
> server/base/src/main/java/org/apache/accumulo/server/fs/PerTableVolumeChooser.java
>  7a825c796eb5a25de8c748e2aba642f483697b9a 
>   
> server/base/src/main/java/org/apache/accumulo/server/fs/PreferredVolumeChooser.java
>  7ed7bba809a4e5e4b2d472c3327b15adb37251a7 
>   
> server/base/src/main/java/org/apache/accumulo/server/fs/RandomVolumeChooser.java
>  f2eb2113cb848ed58ac5f41573c6ff2cde9b0a77 
>   server/base/src/main/java/org/apache/accumulo/server/fs/VolumeChooser.java 
> f523057b11a2dc42e82010675bb1ac8e3802f96d 
>   server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManager.java 
> 890651e92f4c34514cb839b7b9ee9d23ad55070a 
>   
> server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManagerImpl.java
>  dc1be739b634d91992894f8f27c2d9c184bd98cd 
>   server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java 
> 877d01c2233cca086c1ac1539eb81cc282a7ceb4 
>   
> server/base/src/main/java/org/apache/accumulo/server/util/TabletOperations.java
>  c0e1a9b991d61a4dbb127c74ae297f171434e7d5 
>   
> server/base/src/test/java/org/apache/accumulo/server/fs/VolumeManagerImplTest.java
>  582822a8ccbc398925a2184eed0b9d7fa853f9b4 
> 
> Diff: https://reviews.apache.org/r/28873/diff/
> 
> 
> Testing
> ---
> 
> Ran through VolumeManagerImplTest and VolumeChooserIT 
> 
> 
> Thanks,
> 
> Sean Busbey
> 
>

Re: Official Guidance to users regarding removal of functions

[Jeremy Kepner]

So the simple solution is to deprecate often, but remove almost never.
It is very rare that leaving a deprecated API in place actually has a negative 
impact.
The code gets a little less clean, but that's fine as long as things are 
clearly labeled as deprecated.
In fact, seeing the way something used to be done can often be an inspiration 
for something new.
If the past is deleted, then that knowledge is lost.

I am not saying deleting can never happen, I am just saying that when it does, 
it is because
there absolutely no choice.  Deletion to "clean up the code" shouldn't be a 
valid reason for deletion.

On Thu, Dec 11, 2014 at 12:58:13PM -0500, Christopher wrote:
> I don't know that it'd be "cold comfort". We can continue to support 1.x
> for some time, if we choose.
> 
> 
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
> 
> On Thu, Dec 11, 2014 at 12:53 PM, Billie Rinaldi  wrote:
> 
> > Actually, I wasn't suggesting anything.  I was providing elaboration on
> > what John was referring to.  I imagine that stronger API guarantees will be
> > cold comfort in the face of a 1.0 -> 2.0 upgrade.  However, if we had been
> > using semver all along, there would have been much less pain for users in
> > the 1.x series.  Also, adopting semver would mean that going from 1.6 to a
> > hypothetical 1.7 would not suffer from the same upgrade issues.  I doubt
> > that we could retroactively mitigate the differences in minor versions,
> > though, so going from 1.3/1.4/1.5 to 1.7 would still be hard.
> >
> > On Thu, Dec 11, 2014 at 9:11 AM, Mike Drob  wrote:
> >
> > > Billie,
> > >
> > > Not to be glib, but it reads like your suggestion to Jeremy for when we
> > > have a 2.0.0 release (assuming semver passes) is to take option (2) Don't
> > > upgrade Accumulo.
> > >
> > > Please correct my misunderstanding.
> > >
> > > Mike
> > >
> > > On Thu, Dec 11, 2014 at 11:07 AM, Billie Rinaldi 
> > > wrote:
> > >
> > > > To clarify John's question: if our vote to adopt semver 2.0.0 passes,
> > our
> > > > intention will be to no longer have breaking public API changes unless
> > > the
> > > > major version number is incremented, i.e. 1.x.x -> 2.x.x. An important
> > > > aspect of semantic versioning is defining what is considered part of
> > the
> > > > public API. So if there are things you need to remain consistent that
> > are
> > > > not covered by Section 9 of the README, we should discuss adding them.
> > > > Actually, strengthening what we consider to be the public API is likely
> > > to
> > > > be a separate conversation in which (I hope) we will engage the user
> > > list.
> > > > On Dec 11, 2014 11:51 AM, "John Vines"  wrote:
> > > >
> > > > > Wouldn't this be resolved with our SemVer sqwitch?
> > > > >
> > > > > On Thu, Dec 11, 2014 at 11:36 AM, Kepner, Jeremy - 0553 - MITLL <
> > > > > kep...@ll.mit.edu> wrote:
> > > > >
> > > > > > When we remove functions, do we have any official guidance to our
> > > users
> > > > > > who may have built applications that use those functions?
> > > > > >
> > > > > > Right now, the official position is that the Accumulo developers
> > can
> > > > > > remove based on a consensus vote. However, this provides no
> > guidance
> > > to
> > > > > > users as to what they are suppose to do? As it stands, our guidance
> > > is
> > > > > that
> > > > > > they have the following choices:
> > > > > >
> > > > > > (0) Diligently watch the Accumulo e-mail list and aggressively
> > weigh
> > > in
> > > > > on
> > > > > > any vote to remove functions that may impact them.
> > > > > >
> > > > > > (1) Find someone to modify the original source code of their
> > > > > applications,
> > > > > > build it, and *re-verify* the application. I emphasise the
> > re-verify
> > > > > > because that is usually the most costly part of the process that
> > > often
> > > > > > won't get approved by management.
> > > > > >
> > > > > > (2) Don't upgrade Accumulo.
> > > > > >
> > > > >
> > > >
> > >
> >

Re: Official Guidance to users regarding removal of functions

[Adam Fuchs]

Has anybody looked into separating the public API a bit more from the
core? It seems to me that a large number of the deprecation removal
issues are related to people failing to read section 9 of the README.
It would be great if we built an API jar that people could build
against, but didn't leak internal classes. Maybe this is something we
can shoot for in the 2.0 release?

Taking that a step further, it would be great if we released a 1.x API
compatible client jar for every 2.x or later release. Does anybody
have a feel for the maintenance costs of such a thing? Certainly
changes to configuration options and metadata table structures will
prove challenging. Given that we don't have a history of removing
functionality, this ought to at least be feasible.

Thoughts?

Adam


On Thu, Dec 11, 2014 at 1:54 PM, Jeremy Kepner  wrote:
> So the simple solution is to deprecate often, but remove almost never.
> It is very rare that leaving a deprecated API in place actually has a 
> negative impact.
> The code gets a little less clean, but that's fine as long as things are 
> clearly labeled as deprecated.
> In fact, seeing the way something used to be done can often be an inspiration 
> for something new.
> If the past is deleted, then that knowledge is lost.
>
> I am not saying deleting can never happen, I am just saying that when it 
> does, it is because
> there absolutely no choice.  Deletion to "clean up the code" shouldn't be a 
> valid reason for deletion.
>
> On Thu, Dec 11, 2014 at 12:58:13PM -0500, Christopher wrote:
>> I don't know that it'd be "cold comfort". We can continue to support 1.x
>> for some time, if we choose.
>>
>>
>> --
>> Christopher L Tubbs II
>> http://gravatar.com/ctubbsii
>>
>> On Thu, Dec 11, 2014 at 12:53 PM, Billie Rinaldi  wrote:
>>
>> > Actually, I wasn't suggesting anything.  I was providing elaboration on
>> > what John was referring to.  I imagine that stronger API guarantees will be
>> > cold comfort in the face of a 1.0 -> 2.0 upgrade.  However, if we had been
>> > using semver all along, there would have been much less pain for users in
>> > the 1.x series.  Also, adopting semver would mean that going from 1.6 to a
>> > hypothetical 1.7 would not suffer from the same upgrade issues.  I doubt
>> > that we could retroactively mitigate the differences in minor versions,
>> > though, so going from 1.3/1.4/1.5 to 1.7 would still be hard.
>> >
>> > On Thu, Dec 11, 2014 at 9:11 AM, Mike Drob  wrote:
>> >
>> > > Billie,
>> > >
>> > > Not to be glib, but it reads like your suggestion to Jeremy for when we
>> > > have a 2.0.0 release (assuming semver passes) is to take option (2) Don't
>> > > upgrade Accumulo.
>> > >
>> > > Please correct my misunderstanding.
>> > >
>> > > Mike
>> > >
>> > > On Thu, Dec 11, 2014 at 11:07 AM, Billie Rinaldi 
>> > > wrote:
>> > >
>> > > > To clarify John's question: if our vote to adopt semver 2.0.0 passes,
>> > our
>> > > > intention will be to no longer have breaking public API changes unless
>> > > the
>> > > > major version number is incremented, i.e. 1.x.x -> 2.x.x. An important
>> > > > aspect of semantic versioning is defining what is considered part of
>> > the
>> > > > public API. So if there are things you need to remain consistent that
>> > are
>> > > > not covered by Section 9 of the README, we should discuss adding them.
>> > > > Actually, strengthening what we consider to be the public API is likely
>> > > to
>> > > > be a separate conversation in which (I hope) we will engage the user
>> > > list.
>> > > > On Dec 11, 2014 11:51 AM, "John Vines"  wrote:
>> > > >
>> > > > > Wouldn't this be resolved with our SemVer sqwitch?
>> > > > >
>> > > > > On Thu, Dec 11, 2014 at 11:36 AM, Kepner, Jeremy - 0553 - MITLL <
>> > > > > kep...@ll.mit.edu> wrote:
>> > > > >
>> > > > > > When we remove functions, do we have any official guidance to our
>> > > users
>> > > > > > who may have built applications that use those functions?
>> > > > > >
>> > > > > > Right now, the official position is that the Accumulo developers
>> > can
>> > > > > > remove based on a consensus vote. However, this provides no
>> > guidance
>> > > to
>> > > > > > users as to what they are suppose to do? As it stands, our guidance
>> > > is
>> > > > > that
>> > > > > > they have the following choices:
>> > > > > >
>> > > > > > (0) Diligently watch the Accumulo e-mail list and aggressively
>> > weigh
>> > > in
>> > > > > on
>> > > > > > any vote to remove functions that may impact them.
>> > > > > >
>> > > > > > (1) Find someone to modify the original source code of their
>> > > > > applications,
>> > > > > > build it, and *re-verify* the application. I emphasise the
>> > re-verify
>> > > > > > because that is usually the most costly part of the process that
>> > > often
>> > > > > > won't get approved by management.
>> > > > > >
>> > > > > > (2) Don't upgrade Accumulo.
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >

Re: Official Guidance to users regarding removal of functions

[Josh Elser]

Adam Fuchs wrote:

Has anybody looked into separating the public API a bit more from the
core? It seems to me that a large number of the deprecation removal
issues are related to people failing to read section 9 of the README.
It would be great if we built an API jar that people could build
against, but didn't leak internal classes. Maybe this is something we
can shoot for in the 2.0 release?


Yup, this is already in the works by Christopher as a part of ACCUMULO-2589.


Taking that a step further, it would be great if we released a 1.x API
compatible client jar for every 2.x or later release. Does anybody
have a feel for the maintenance costs of such a thing? Certainly
changes to configuration options and metadata table structures will
prove challenging. Given that we don't have a history of removing
functionality, this ought to at least be feasible.

Thoughts?

Adam


On Thu, Dec 11, 2014 at 1:54 PM, Jeremy Kepner  wrote:

So the simple solution is to deprecate often, but remove almost never.
It is very rare that leaving a deprecated API in place actually has a negative 
impact.
The code gets a little less clean, but that's fine as long as things are 
clearly labeled as deprecated.
In fact, seeing the way something used to be done can often be an inspiration 
for something new.
If the past is deleted, then that knowledge is lost.

I am not saying deleting can never happen, I am just saying that when it does, 
it is because
there absolutely no choice.  Deletion to "clean up the code" shouldn't be a 
valid reason for deletion.

On Thu, Dec 11, 2014 at 12:58:13PM -0500, Christopher wrote:

I don't know that it'd be "cold comfort". We can continue to support 1.x
for some time, if we choose.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Thu, Dec 11, 2014 at 12:53 PM, Billie Rinaldi  wrote:


Actually, I wasn't suggesting anything.  I was providing elaboration on
what John was referring to.  I imagine that stronger API guarantees will be
cold comfort in the face of a 1.0 ->  2.0 upgrade.  However, if we had been
using semver all along, there would have been much less pain for users in
the 1.x series.  Also, adopting semver would mean that going from 1.6 to a
hypothetical 1.7 would not suffer from the same upgrade issues.  I doubt
that we could retroactively mitigate the differences in minor versions,
though, so going from 1.3/1.4/1.5 to 1.7 would still be hard.

On Thu, Dec 11, 2014 at 9:11 AM, Mike Drob  wrote:


Billie,

Not to be glib, but it reads like your suggestion to Jeremy for when we
have a 2.0.0 release (assuming semver passes) is to take option (2) Don't
upgrade Accumulo.

Please correct my misunderstanding.

Mike

On Thu, Dec 11, 2014 at 11:07 AM, Billie Rinaldi
wrote:


To clarify John's question: if our vote to adopt semver 2.0.0 passes,

our

intention will be to no longer have breaking public API changes unless

the

major version number is incremented, i.e. 1.x.x ->  2.x.x. An important
aspect of semantic versioning is defining what is considered part of

the

public API. So if there are things you need to remain consistent that

are

not covered by Section 9 of the README, we should discuss adding them.
Actually, strengthening what we consider to be the public API is likely

to

be a separate conversation in which (I hope) we will engage the user

list.

On Dec 11, 2014 11:51 AM, "John Vines"  wrote:


Wouldn't this be resolved with our SemVer sqwitch?

On Thu, Dec 11, 2014 at 11:36 AM, Kepner, Jeremy - 0553 - MITLL<
kep...@ll.mit.edu>  wrote:


When we remove functions, do we have any official guidance to our

users

who may have built applications that use those functions?

Right now, the official position is that the Accumulo developers

can

remove based on a consensus vote. However, this provides no

guidance

to

users as to what they are suppose to do? As it stands, our guidance

is

that

they have the following choices:

(0) Diligently watch the Accumulo e-mail list and aggressively

weigh

in

on

any vote to remove functions that may impact them.

(1) Find someone to modify the original source code of their

applications,

build it, and *re-verify* the application. I emphasise the

re-verify

because that is usually the most costly part of the process that

often

won't get approved by management.

(2) Don't upgrade Accumulo.

Re: Official Guidance to users regarding removal of functions

[Adam Fuchs]

Awesome -- ACCUMULO-2589 gets us at least halfway there. Given this,
what would be the challenges in having and maintaining one API project
for each major version ever released?

Adam

On Thu, Dec 11, 2014 at 2:24 PM, Josh Elser  wrote:
> Adam Fuchs wrote:
>>
>> Has anybody looked into separating the public API a bit more from the
>> core? It seems to me that a large number of the deprecation removal
>> issues are related to people failing to read section 9 of the README.
>> It would be great if we built an API jar that people could build
>> against, but didn't leak internal classes. Maybe this is something we
>> can shoot for in the 2.0 release?
>
>
> Yup, this is already in the works by Christopher as a part of ACCUMULO-2589.
>
>
>> Taking that a step further, it would be great if we released a 1.x API
>> compatible client jar for every 2.x or later release. Does anybody
>> have a feel for the maintenance costs of such a thing? Certainly
>> changes to configuration options and metadata table structures will
>> prove challenging. Given that we don't have a history of removing
>> functionality, this ought to at least be feasible.
>>
>> Thoughts?
>>
>> Adam
>>
>>
>> On Thu, Dec 11, 2014 at 1:54 PM, Jeremy Kepner  wrote:
>>>
>>> So the simple solution is to deprecate often, but remove almost never.
>>> It is very rare that leaving a deprecated API in place actually has a
>>> negative impact.
>>> The code gets a little less clean, but that's fine as long as things are
>>> clearly labeled as deprecated.
>>> In fact, seeing the way something used to be done can often be an
>>> inspiration for something new.
>>> If the past is deleted, then that knowledge is lost.
>>>
>>> I am not saying deleting can never happen, I am just saying that when it
>>> does, it is because
>>> there absolutely no choice.  Deletion to "clean up the code" shouldn't be
>>> a valid reason for deletion.
>>>
>>> On Thu, Dec 11, 2014 at 12:58:13PM -0500, Christopher wrote:

 I don't know that it'd be "cold comfort". We can continue to support 1.x
 for some time, if we choose.


 --
 Christopher L Tubbs II
 http://gravatar.com/ctubbsii

 On Thu, Dec 11, 2014 at 12:53 PM, Billie Rinaldi
 wrote:

> Actually, I wasn't suggesting anything.  I was providing elaboration on
> what John was referring to.  I imagine that stronger API guarantees
> will be
> cold comfort in the face of a 1.0 ->  2.0 upgrade.  However, if we had
> been
> using semver all along, there would have been much less pain for users
> in
> the 1.x series.  Also, adopting semver would mean that going from 1.6
> to a
> hypothetical 1.7 would not suffer from the same upgrade issues.  I
> doubt
> that we could retroactively mitigate the differences in minor versions,
> though, so going from 1.3/1.4/1.5 to 1.7 would still be hard.
>
> On Thu, Dec 11, 2014 at 9:11 AM, Mike Drob  wrote:
>
>> Billie,
>>
>> Not to be glib, but it reads like your suggestion to Jeremy for when
>> we
>> have a 2.0.0 release (assuming semver passes) is to take option (2)
>> Don't
>> upgrade Accumulo.
>>
>> Please correct my misunderstanding.
>>
>> Mike
>>
>> On Thu, Dec 11, 2014 at 11:07 AM, Billie Rinaldi
>> wrote:
>>
>>> To clarify John's question: if our vote to adopt semver 2.0.0 passes,
>
> our
>>>
>>> intention will be to no longer have breaking public API changes
>>> unless
>>
>> the
>>>
>>> major version number is incremented, i.e. 1.x.x ->  2.x.x. An
>>> important
>>> aspect of semantic versioning is defining what is considered part of
>
> the
>>>
>>> public API. So if there are things you need to remain consistent that
>
> are
>>>
>>> not covered by Section 9 of the README, we should discuss adding
>>> them.
>>> Actually, strengthening what we consider to be the public API is
>>> likely
>>
>> to
>>>
>>> be a separate conversation in which (I hope) we will engage the user
>>
>> list.
>>>
>>> On Dec 11, 2014 11:51 AM, "John Vines"  wrote:
>>>
 Wouldn't this be resolved with our SemVer sqwitch?

 On Thu, Dec 11, 2014 at 11:36 AM, Kepner, Jeremy - 0553 - MITLL<
 kep...@ll.mit.edu>  wrote:

> When we remove functions, do we have any official guidance to our
>>
>> users
>
> who may have built applications that use those functions?
>
> Right now, the official position is that the Accumulo developers
>
> can
>
> remove based on a consensus vote. However, this provides no
>
> guidance
>>
>> to
>
> users as to what they are suppose to do? As it stands, our guidance
>>
>> is

 that
>
> they have the following choices:
>
>>>

Re: Official Guidance to users regarding removal of functions

[John Vines]

More likely we'd have a fully backwards compatible API for each major
version. SemVer allows for it and I think that grants us enough room for
growth while still securing things for future releases.

On Thu, Dec 11, 2014 at 2:36 PM, Adam Fuchs  wrote:

> Awesome -- ACCUMULO-2589 gets us at least halfway there. Given this,
> what would be the challenges in having and maintaining one API project
> for each major version ever released?
>
> Adam
>
> On Thu, Dec 11, 2014 at 2:24 PM, Josh Elser  wrote:
> > Adam Fuchs wrote:
> >>
> >> Has anybody looked into separating the public API a bit more from the
> >> core? It seems to me that a large number of the deprecation removal
> >> issues are related to people failing to read section 9 of the README.
> >> It would be great if we built an API jar that people could build
> >> against, but didn't leak internal classes. Maybe this is something we
> >> can shoot for in the 2.0 release?
> >
> >
> > Yup, this is already in the works by Christopher as a part of
> ACCUMULO-2589.
> >
> >
> >> Taking that a step further, it would be great if we released a 1.x API
> >> compatible client jar for every 2.x or later release. Does anybody
> >> have a feel for the maintenance costs of such a thing? Certainly
> >> changes to configuration options and metadata table structures will
> >> prove challenging. Given that we don't have a history of removing
> >> functionality, this ought to at least be feasible.
> >>
> >> Thoughts?
> >>
> >> Adam
> >>
> >>
> >> On Thu, Dec 11, 2014 at 1:54 PM, Jeremy Kepner
> wrote:
> >>>
> >>> So the simple solution is to deprecate often, but remove almost never.
> >>> It is very rare that leaving a deprecated API in place actually has a
> >>> negative impact.
> >>> The code gets a little less clean, but that's fine as long as things
> are
> >>> clearly labeled as deprecated.
> >>> In fact, seeing the way something used to be done can often be an
> >>> inspiration for something new.
> >>> If the past is deleted, then that knowledge is lost.
> >>>
> >>> I am not saying deleting can never happen, I am just saying that when
> it
> >>> does, it is because
> >>> there absolutely no choice.  Deletion to "clean up the code" shouldn't
> be
> >>> a valid reason for deletion.
> >>>
> >>> On Thu, Dec 11, 2014 at 12:58:13PM -0500, Christopher wrote:
> 
>  I don't know that it'd be "cold comfort". We can continue to support
> 1.x
>  for some time, if we choose.
> 
> 
>  --
>  Christopher L Tubbs II
>  http://gravatar.com/ctubbsii
> 
>  On Thu, Dec 11, 2014 at 12:53 PM, Billie Rinaldi
>  wrote:
> 
> > Actually, I wasn't suggesting anything.  I was providing elaboration
> on
> > what John was referring to.  I imagine that stronger API guarantees
> > will be
> > cold comfort in the face of a 1.0 ->  2.0 upgrade.  However, if we
> had
> > been
> > using semver all along, there would have been much less pain for
> users
> > in
> > the 1.x series.  Also, adopting semver would mean that going from 1.6
> > to a
> > hypothetical 1.7 would not suffer from the same upgrade issues.  I
> > doubt
> > that we could retroactively mitigate the differences in minor
> versions,
> > though, so going from 1.3/1.4/1.5 to 1.7 would still be hard.
> >
> > On Thu, Dec 11, 2014 at 9:11 AM, Mike Drob
> wrote:
> >
> >> Billie,
> >>
> >> Not to be glib, but it reads like your suggestion to Jeremy for when
> >> we
> >> have a 2.0.0 release (assuming semver passes) is to take option (2)
> >> Don't
> >> upgrade Accumulo.
> >>
> >> Please correct my misunderstanding.
> >>
> >> Mike
> >>
> >> On Thu, Dec 11, 2014 at 11:07 AM, Billie Rinaldi
> >> wrote:
> >>
> >>> To clarify John's question: if our vote to adopt semver 2.0.0
> passes,
> >
> > our
> >>>
> >>> intention will be to no longer have breaking public API changes
> >>> unless
> >>
> >> the
> >>>
> >>> major version number is incremented, i.e. 1.x.x ->  2.x.x. An
> >>> important
> >>> aspect of semantic versioning is defining what is considered part
> of
> >
> > the
> >>>
> >>> public API. So if there are things you need to remain consistent
> that
> >
> > are
> >>>
> >>> not covered by Section 9 of the README, we should discuss adding
> >>> them.
> >>> Actually, strengthening what we consider to be the public API is
> >>> likely
> >>
> >> to
> >>>
> >>> be a separate conversation in which (I hope) we will engage the
> user
> >>
> >> list.
> >>>
> >>> On Dec 11, 2014 11:51 AM, "John Vines"  wrote:
> >>>
>  Wouldn't this be resolved with our SemVer sqwitch?
> 
>  On Thu, Dec 11, 2014 at 11:36 AM, Kepner, Jeremy - 0553 - MITLL<
>  kep...@ll.mit.edu>  wrote:
> 
> > When we remove functions, do we have any o

Re: Official Guidance to users regarding removal of functions

[Kepner, Jeremy - 0553 - MITLL]

So I think the bigger issue is to revisit the imperative to delete deprecated 
functions (both public & private).  How many instances have we had where 
deleting a deprecated API (public & private) did anything more than "clean up" 
the code?

On Dec 11, 2014, at 2:39 PM, John Vines  wrote:

> More likely we'd have a fully backwards compatible API for each major
> version. SemVer allows for it and I think that grants us enough room for
> growth while still securing things for future releases.
> 
> On Thu, Dec 11, 2014 at 2:36 PM, Adam Fuchs  wrote:
> 
>> Awesome -- ACCUMULO-2589 gets us at least halfway there. Given this,
>> what would be the challenges in having and maintaining one API project
>> for each major version ever released?
>> 
>> Adam
>> 
>> On Thu, Dec 11, 2014 at 2:24 PM, Josh Elser  wrote:
>>> Adam Fuchs wrote:
 
 Has anybody looked into separating the public API a bit more from the
 core? It seems to me that a large number of the deprecation removal
 issues are related to people failing to read section 9 of the README.
 It would be great if we built an API jar that people could build
 against, but didn't leak internal classes. Maybe this is something we
 can shoot for in the 2.0 release?
>>> 
>>> 
>>> Yup, this is already in the works by Christopher as a part of
>> ACCUMULO-2589.
>>> 
>>> 
 Taking that a step further, it would be great if we released a 1.x API
 compatible client jar for every 2.x or later release. Does anybody
 have a feel for the maintenance costs of such a thing? Certainly
 changes to configuration options and metadata table structures will
 prove challenging. Given that we don't have a history of removing
 functionality, this ought to at least be feasible.
 
 Thoughts?
 
 Adam
 
 
 On Thu, Dec 11, 2014 at 1:54 PM, Jeremy Kepner
>> wrote:
> 
> So the simple solution is to deprecate often, but remove almost never.
> It is very rare that leaving a deprecated API in place actually has a
> negative impact.
> The code gets a little less clean, but that's fine as long as things
>> are
> clearly labeled as deprecated.
> In fact, seeing the way something used to be done can often be an
> inspiration for something new.
> If the past is deleted, then that knowledge is lost.
> 
> I am not saying deleting can never happen, I am just saying that when
>> it
> does, it is because
> there absolutely no choice.  Deletion to "clean up the code" shouldn't
>> be
> a valid reason for deletion.
> 
> On Thu, Dec 11, 2014 at 12:58:13PM -0500, Christopher wrote:
>> 
>> I don't know that it'd be "cold comfort". We can continue to support
>> 1.x
>> for some time, if we choose.
>> 
>> 
>> --
>> Christopher L Tubbs II
>> http://gravatar.com/ctubbsii
>> 
>> On Thu, Dec 11, 2014 at 12:53 PM, Billie Rinaldi
>> wrote:
>> 
>>> Actually, I wasn't suggesting anything.  I was providing elaboration
>> on
>>> what John was referring to.  I imagine that stronger API guarantees
>>> will be
>>> cold comfort in the face of a 1.0 ->  2.0 upgrade.  However, if we
>> had
>>> been
>>> using semver all along, there would have been much less pain for
>> users
>>> in
>>> the 1.x series.  Also, adopting semver would mean that going from 1.6
>>> to a
>>> hypothetical 1.7 would not suffer from the same upgrade issues.  I
>>> doubt
>>> that we could retroactively mitigate the differences in minor
>> versions,
>>> though, so going from 1.3/1.4/1.5 to 1.7 would still be hard.
>>> 
>>> On Thu, Dec 11, 2014 at 9:11 AM, Mike Drob
>> wrote:
>>> 
 Billie,
 
 Not to be glib, but it reads like your suggestion to Jeremy for when
 we
 have a 2.0.0 release (assuming semver passes) is to take option (2)
 Don't
 upgrade Accumulo.
 
 Please correct my misunderstanding.
 
 Mike
 
 On Thu, Dec 11, 2014 at 11:07 AM, Billie Rinaldi
 wrote:
 
> To clarify John's question: if our vote to adopt semver 2.0.0
>> passes,
>>> 
>>> our
> 
> intention will be to no longer have breaking public API changes
> unless
 
 the
> 
> major version number is incremented, i.e. 1.x.x ->  2.x.x. An
> important
> aspect of semantic versioning is defining what is considered part
>> of
>>> 
>>> the
> 
> public API. So if there are things you need to remain consistent
>> that
>>> 
>>> are
> 
> not covered by Section 9 of the README, we should discuss adding
> them.
> Actually, strengthening what we consider to be the public API is
> likely
 
 to
> 
> be a separate conversation in which (I hope) we will engage th

Re: Official Guidance to users regarding removal of functions

[Sean Busbey]

On Thu, Dec 11, 2014 at 2:19 PM, Kepner, Jeremy - 0553 - MITLL <
kep...@ll.mit.edu> wrote:

> So I think the bigger issue is to revisit the imperative to delete
> deprecated functions (both public & private).  How many instances have we
> had where deleting a deprecated API (public & private) did anything more
> than "clean up" the code?
>
>

Personally, I don't think restrictions on private apis should be on the
table. We need flexibility to reduce our support burden and to make
improvements in the overall system that upend our current internal
structures. Often times, that means redoing how pieces work together. If we
have to rely on package and private scopes to delineate our support surface
it'll drive us to make poorer code organization choices.

That doesn't mean we should keep the public API where it is now. We can
expand it as needed to make sure we have a reasonable level of assurances
for downstream users. For example, it would be really great if we could get
to rolling upgrades as a tested thing, at least within minor versions and
preferably across adjacent major versions. I'm pretty sure that requires
expanding the public API to include RPC.

One thing that might make this easier for folks downstream, even without
waiting for a new client API, is to introduce annotations for our expected
audience and stability levels. This would allow us to mark the things that
are public under the current README and also let us flag things that aren't
there but are expected integration points (like custom balancers or
iterators). Hadoop and HBase both do this[1] and HBase even generates
custom javadocs for just those things that are flagged public[2].

[1]: http://hbase.apache.org/book/developing.html#d4209e21495
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/classification/package-summary.html

[2]: http://hbase.apache.org/apidocs/index.html

-- 
Sean

Re: Official Guidance to users regarding removal of functions

[Christopher]

In addition to the responses already provided, I think it'd be better to
just continue supporting 1.x with new features/bugfixes as long as there is
demand for it, would be preferred than trying to keep a 1.x shim API on top
of newer releases.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Thu, Dec 11, 2014 at 2:22 PM, Adam Fuchs  wrote:

> Has anybody looked into separating the public API a bit more from the
> core? It seems to me that a large number of the deprecation removal
> issues are related to people failing to read section 9 of the README.
> It would be great if we built an API jar that people could build
> against, but didn't leak internal classes. Maybe this is something we
> can shoot for in the 2.0 release?
>
> Taking that a step further, it would be great if we released a 1.x API
> compatible client jar for every 2.x or later release. Does anybody
> have a feel for the maintenance costs of such a thing? Certainly
> changes to configuration options and metadata table structures will
> prove challenging. Given that we don't have a history of removing
> functionality, this ought to at least be feasible.
>
> Thoughts?
>
> Adam
>
>
> On Thu, Dec 11, 2014 at 1:54 PM, Jeremy Kepner  wrote:
> > So the simple solution is to deprecate often, but remove almost never.
> > It is very rare that leaving a deprecated API in place actually has a
> negative impact.
> > The code gets a little less clean, but that's fine as long as things are
> clearly labeled as deprecated.
> > In fact, seeing the way something used to be done can often be an
> inspiration for something new.
> > If the past is deleted, then that knowledge is lost.
> >
> > I am not saying deleting can never happen, I am just saying that when it
> does, it is because
> > there absolutely no choice.  Deletion to "clean up the code" shouldn't
> be a valid reason for deletion.
> >
> > On Thu, Dec 11, 2014 at 12:58:13PM -0500, Christopher wrote:
> >> I don't know that it'd be "cold comfort". We can continue to support 1.x
> >> for some time, if we choose.
> >>
> >>
> >> --
> >> Christopher L Tubbs II
> >> http://gravatar.com/ctubbsii
> >>
> >> On Thu, Dec 11, 2014 at 12:53 PM, Billie Rinaldi 
> wrote:
> >>
> >> > Actually, I wasn't suggesting anything.  I was providing elaboration
> on
> >> > what John was referring to.  I imagine that stronger API guarantees
> will be
> >> > cold comfort in the face of a 1.0 -> 2.0 upgrade.  However, if we had
> been
> >> > using semver all along, there would have been much less pain for
> users in
> >> > the 1.x series.  Also, adopting semver would mean that going from 1.6
> to a
> >> > hypothetical 1.7 would not suffer from the same upgrade issues.  I
> doubt
> >> > that we could retroactively mitigate the differences in minor
> versions,
> >> > though, so going from 1.3/1.4/1.5 to 1.7 would still be hard.
> >> >
> >> > On Thu, Dec 11, 2014 at 9:11 AM, Mike Drob 
> wrote:
> >> >
> >> > > Billie,
> >> > >
> >> > > Not to be glib, but it reads like your suggestion to Jeremy for
> when we
> >> > > have a 2.0.0 release (assuming semver passes) is to take option (2)
> Don't
> >> > > upgrade Accumulo.
> >> > >
> >> > > Please correct my misunderstanding.
> >> > >
> >> > > Mike
> >> > >
> >> > > On Thu, Dec 11, 2014 at 11:07 AM, Billie Rinaldi  >
> >> > > wrote:
> >> > >
> >> > > > To clarify John's question: if our vote to adopt semver 2.0.0
> passes,
> >> > our
> >> > > > intention will be to no longer have breaking public API changes
> unless
> >> > > the
> >> > > > major version number is incremented, i.e. 1.x.x -> 2.x.x. An
> important
> >> > > > aspect of semantic versioning is defining what is considered part
> of
> >> > the
> >> > > > public API. So if there are things you need to remain consistent
> that
> >> > are
> >> > > > not covered by Section 9 of the README, we should discuss adding
> them.
> >> > > > Actually, strengthening what we consider to be the public API is
> likely
> >> > > to
> >> > > > be a separate conversation in which (I hope) we will engage the
> user
> >> > > list.
> >> > > > On Dec 11, 2014 11:51 AM, "John Vines"  wrote:
> >> > > >
> >> > > > > Wouldn't this be resolved with our SemVer sqwitch?
> >> > > > >
> >> > > > > On Thu, Dec 11, 2014 at 11:36 AM, Kepner, Jeremy - 0553 - MITLL
> <
> >> > > > > kep...@ll.mit.edu> wrote:
> >> > > > >
> >> > > > > > When we remove functions, do we have any official guidance to
> our
> >> > > users
> >> > > > > > who may have built applications that use those functions?
> >> > > > > >
> >> > > > > > Right now, the official position is that the Accumulo
> developers
> >> > can
> >> > > > > > remove based on a consensus vote. However, this provides no
> >> > guidance
> >> > > to
> >> > > > > > users as to what they are suppose to do? As it stands, our
> guidance
> >> > > is
> >> > > > > that
> >> > > > > > they have the following choices:
> >> > > > > >
> >> > > > > > (0) Diligently watch the Accumulo e-mail list and aggressively
> >> > weig

Re: Official Guidance to users regarding removal of functions

[Christopher]

You seem to be dismissing "clean up" as an invalid. I disagree. There are
costs to keeping around deprecated APIs. That's not to say we shouldn't
keep them around for a long time... we can and perhaps should. But "clean
up" is just shorthand for all the maintainability, readability, and
usability costs. As such, I feel it incapsulates many reasons which are
sometimes difficult to express, but still valid.

With semver in place, it seems like your position could be reduced to
"support major versions longer", which is a perfectly fine goal. As long as
we have that, there's really no need to upgrade to the "latest", unless
there's a feature that you want which is difficult to backport in a minor
release. In that case, I think it's perfectly reasonable to consider the
risks of upgrading and to make the choice to upgrade and have the feature
or not upgrade and avoid API changes which affect you (if any).


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Thu, Dec 11, 2014 at 3:19 PM, Kepner, Jeremy - 0553 - MITLL <
kep...@ll.mit.edu> wrote:

> So I think the bigger issue is to revisit the imperative to delete
> deprecated functions (both public & private).  How many instances have we
> had where deleting a deprecated API (public & private) did anything more
> than "clean up" the code?
>
> On Dec 11, 2014, at 2:39 PM, John Vines  wrote:
>
> > More likely we'd have a fully backwards compatible API for each major
> > version. SemVer allows for it and I think that grants us enough room for
> > growth while still securing things for future releases.
> >
> > On Thu, Dec 11, 2014 at 2:36 PM, Adam Fuchs  wrote:
> >
> >> Awesome -- ACCUMULO-2589 gets us at least halfway there. Given this,
> >> what would be the challenges in having and maintaining one API project
> >> for each major version ever released?
> >>
> >> Adam
> >>
> >> On Thu, Dec 11, 2014 at 2:24 PM, Josh Elser 
> wrote:
> >>> Adam Fuchs wrote:
> 
>  Has anybody looked into separating the public API a bit more from the
>  core? It seems to me that a large number of the deprecation removal
>  issues are related to people failing to read section 9 of the README.
>  It would be great if we built an API jar that people could build
>  against, but didn't leak internal classes. Maybe this is something we
>  can shoot for in the 2.0 release?
> >>>
> >>>
> >>> Yup, this is already in the works by Christopher as a part of
> >> ACCUMULO-2589.
> >>>
> >>>
>  Taking that a step further, it would be great if we released a 1.x API
>  compatible client jar for every 2.x or later release. Does anybody
>  have a feel for the maintenance costs of such a thing? Certainly
>  changes to configuration options and metadata table structures will
>  prove challenging. Given that we don't have a history of removing
>  functionality, this ought to at least be feasible.
> 
>  Thoughts?
> 
>  Adam
> 
> 
>  On Thu, Dec 11, 2014 at 1:54 PM, Jeremy Kepner
> >> wrote:
> >
> > So the simple solution is to deprecate often, but remove almost
> never.
> > It is very rare that leaving a deprecated API in place actually has a
> > negative impact.
> > The code gets a little less clean, but that's fine as long as things
> >> are
> > clearly labeled as deprecated.
> > In fact, seeing the way something used to be done can often be an
> > inspiration for something new.
> > If the past is deleted, then that knowledge is lost.
> >
> > I am not saying deleting can never happen, I am just saying that when
> >> it
> > does, it is because
> > there absolutely no choice.  Deletion to "clean up the code"
> shouldn't
> >> be
> > a valid reason for deletion.
> >
> > On Thu, Dec 11, 2014 at 12:58:13PM -0500, Christopher wrote:
> >>
> >> I don't know that it'd be "cold comfort". We can continue to support
> >> 1.x
> >> for some time, if we choose.
> >>
> >>
> >> --
> >> Christopher L Tubbs II
> >> http://gravatar.com/ctubbsii
> >>
> >> On Thu, Dec 11, 2014 at 12:53 PM, Billie Rinaldi
> >> wrote:
> >>
> >>> Actually, I wasn't suggesting anything.  I was providing
> elaboration
> >> on
> >>> what John was referring to.  I imagine that stronger API guarantees
> >>> will be
> >>> cold comfort in the face of a 1.0 ->  2.0 upgrade.  However, if we
> >> had
> >>> been
> >>> using semver all along, there would have been much less pain for
> >> users
> >>> in
> >>> the 1.x series.  Also, adopting semver would mean that going from
> 1.6
> >>> to a
> >>> hypothetical 1.7 would not suffer from the same upgrade issues.  I
> >>> doubt
> >>> that we could retroactively mitigate the differences in minor
> >> versions,
> >>> though, so going from 1.3/1.4/1.5 to 1.7 would still be hard.
> >>>
> >>> On Thu, Dec 11, 2014 at 9:11 AM, Mike Drob
> >> wrote:
> >>>
>  

Re: Official Guidance to users regarding removal of functions

[Mike Drob]

FWIW, Java 9 is dropping methods for the first time that the JDK API is
dropping methods. Almost 20 years. Just something to consider, when we
approach the conversation of how long to keep APIs around.

On Thu, Dec 11, 2014 at 3:46 PM, Christopher  wrote:

> You seem to be dismissing "clean up" as an invalid. I disagree. There are
> costs to keeping around deprecated APIs. That's not to say we shouldn't
> keep them around for a long time... we can and perhaps should. But "clean
> up" is just shorthand for all the maintainability, readability, and
> usability costs. As such, I feel it incapsulates many reasons which are
> sometimes difficult to express, but still valid.
>
> With semver in place, it seems like your position could be reduced to
> "support major versions longer", which is a perfectly fine goal. As long as
> we have that, there's really no need to upgrade to the "latest", unless
> there's a feature that you want which is difficult to backport in a minor
> release. In that case, I think it's perfectly reasonable to consider the
> risks of upgrading and to make the choice to upgrade and have the feature
> or not upgrade and avoid API changes which affect you (if any).
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
> On Thu, Dec 11, 2014 at 3:19 PM, Kepner, Jeremy - 0553 - MITLL <
> kep...@ll.mit.edu> wrote:
>
> > So I think the bigger issue is to revisit the imperative to delete
> > deprecated functions (both public & private).  How many instances have we
> > had where deleting a deprecated API (public & private) did anything more
> > than "clean up" the code?
> >
> > On Dec 11, 2014, at 2:39 PM, John Vines  wrote:
> >
> > > More likely we'd have a fully backwards compatible API for each major
> > > version. SemVer allows for it and I think that grants us enough room
> for
> > > growth while still securing things for future releases.
> > >
> > > On Thu, Dec 11, 2014 at 2:36 PM, Adam Fuchs  wrote:
> > >
> > >> Awesome -- ACCUMULO-2589 gets us at least halfway there. Given this,
> > >> what would be the challenges in having and maintaining one API project
> > >> for each major version ever released?
> > >>
> > >> Adam
> > >>
> > >> On Thu, Dec 11, 2014 at 2:24 PM, Josh Elser 
> > wrote:
> > >>> Adam Fuchs wrote:
> > 
> >  Has anybody looked into separating the public API a bit more from
> the
> >  core? It seems to me that a large number of the deprecation removal
> >  issues are related to people failing to read section 9 of the
> README.
> >  It would be great if we built an API jar that people could build
> >  against, but didn't leak internal classes. Maybe this is something
> we
> >  can shoot for in the 2.0 release?
> > >>>
> > >>>
> > >>> Yup, this is already in the works by Christopher as a part of
> > >> ACCUMULO-2589.
> > >>>
> > >>>
> >  Taking that a step further, it would be great if we released a 1.x
> API
> >  compatible client jar for every 2.x or later release. Does anybody
> >  have a feel for the maintenance costs of such a thing? Certainly
> >  changes to configuration options and metadata table structures will
> >  prove challenging. Given that we don't have a history of removing
> >  functionality, this ought to at least be feasible.
> > 
> >  Thoughts?
> > 
> >  Adam
> > 
> > 
> >  On Thu, Dec 11, 2014 at 1:54 PM, Jeremy Kepner
> > >> wrote:
> > >
> > > So the simple solution is to deprecate often, but remove almost
> > never.
> > > It is very rare that leaving a deprecated API in place actually
> has a
> > > negative impact.
> > > The code gets a little less clean, but that's fine as long as
> things
> > >> are
> > > clearly labeled as deprecated.
> > > In fact, seeing the way something used to be done can often be an
> > > inspiration for something new.
> > > If the past is deleted, then that knowledge is lost.
> > >
> > > I am not saying deleting can never happen, I am just saying that
> when
> > >> it
> > > does, it is because
> > > there absolutely no choice.  Deletion to "clean up the code"
> > shouldn't
> > >> be
> > > a valid reason for deletion.
> > >
> > > On Thu, Dec 11, 2014 at 12:58:13PM -0500, Christopher wrote:
> > >>
> > >> I don't know that it'd be "cold comfort". We can continue to
> support
> > >> 1.x
> > >> for some time, if we choose.
> > >>
> > >>
> > >> --
> > >> Christopher L Tubbs II
> > >> http://gravatar.com/ctubbsii
> > >>
> > >> On Thu, Dec 11, 2014 at 12:53 PM, Billie Rinaldi<
> bil...@apache.org>
> > >> wrote:
> > >>
> > >>> Actually, I wasn't suggesting anything.  I was providing
> > elaboration
> > >> on
> > >>> what John was referring to.  I imagine that stronger API
> guarantees
> > >>> will be
> > >>> cold comfort in the face of a 1.0 ->  2.0 upgrade.  However, if
> we
> > >> had
> > >>> been
> > >>> using se

Re: Official Guidance to users regarding removal of functions

[Josh Elser]

For context and as a good example of how we could be less abrasive, I'm 
unexpectedly spending today updating Hive with a bunch of new reflection 
because methods in 1.5 on AccumuloInputFormat no longer exist in 1.7. 
setZooKeeperInstance used to take two Strings, but in 1.7 this version 
of the method was removed and forces client to use ClientConfiguration.


This is a good example of where we didn't have to do this and there was 
next to no maintenance support on us to preserve this method.


Honestly, after writing this, I'm tempted to reinstate the old version 
because doing this amount of reflection is painful and silly for clients 
(and I even know what I'm doing most days!)


Mike Drob wrote:

FWIW, Java 9 is dropping methods for the first time that the JDK API is
dropping methods. Almost 20 years. Just something to consider, when we
approach the conversation of how long to keep APIs around.

On Thu, Dec 11, 2014 at 3:46 PM, Christopher  wrote:


You seem to be dismissing "clean up" as an invalid. I disagree. There are
costs to keeping around deprecated APIs. That's not to say we shouldn't
keep them around for a long time... we can and perhaps should. But "clean
up" is just shorthand for all the maintainability, readability, and
usability costs. As such, I feel it incapsulates many reasons which are
sometimes difficult to express, but still valid.

With semver in place, it seems like your position could be reduced to
"support major versions longer", which is a perfectly fine goal. As long as
we have that, there's really no need to upgrade to the "latest", unless
there's a feature that you want which is difficult to backport in a minor
release. In that case, I think it's perfectly reasonable to consider the
risks of upgrading and to make the choice to upgrade and have the feature
or not upgrade and avoid API changes which affect you (if any).


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Thu, Dec 11, 2014 at 3:19 PM, Kepner, Jeremy - 0553 - MITLL<
kep...@ll.mit.edu>  wrote:


So I think the bigger issue is to revisit the imperative to delete
deprecated functions (both public&  private).  How many instances have we
had where deleting a deprecated API (public&  private) did anything more
than "clean up" the code?

On Dec 11, 2014, at 2:39 PM, John Vines  wrote:


More likely we'd have a fully backwards compatible API for each major
version. SemVer allows for it and I think that grants us enough room

for

growth while still securing things for future releases.

On Thu, Dec 11, 2014 at 2:36 PM, Adam Fuchs  wrote:


Awesome -- ACCUMULO-2589 gets us at least halfway there. Given this,
what would be the challenges in having and maintaining one API project
for each major version ever released?

Adam

On Thu, Dec 11, 2014 at 2:24 PM, Josh Elser

wrote:

Adam Fuchs wrote:

Has anybody looked into separating the public API a bit more from

the

core? It seems to me that a large number of the deprecation removal
issues are related to people failing to read section 9 of the

README.

It would be great if we built an API jar that people could build
against, but didn't leak internal classes. Maybe this is something

we

can shoot for in the 2.0 release?


Yup, this is already in the works by Christopher as a part of

ACCUMULO-2589.



Taking that a step further, it would be great if we released a 1.x

API

compatible client jar for every 2.x or later release. Does anybody
have a feel for the maintenance costs of such a thing? Certainly
changes to configuration options and metadata table structures will
prove challenging. Given that we don't have a history of removing
functionality, this ought to at least be feasible.

Thoughts?

Adam


On Thu, Dec 11, 2014 at 1:54 PM, Jeremy Kepner

wrote:

So the simple solution is to deprecate often, but remove almost

never.

It is very rare that leaving a deprecated API in place actually

has a

negative impact.
The code gets a little less clean, but that's fine as long as

things

are

clearly labeled as deprecated.
In fact, seeing the way something used to be done can often be an
inspiration for something new.
If the past is deleted, then that knowledge is lost.

I am not saying deleting can never happen, I am just saying that

when

it

does, it is because
there absolutely no choice.  Deletion to "clean up the code"

shouldn't

be

a valid reason for deletion.

On Thu, Dec 11, 2014 at 12:58:13PM -0500, Christopher wrote:

I don't know that it'd be "cold comfort". We can continue to

support

1.x

for some time, if we choose.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Thu, Dec 11, 2014 at 12:53 PM, Billie Rinaldi<

bil...@apache.org>

wrote:


Actually, I wasn't suggesting anything.  I was providing

elaboration

on

what John was referring to.  I imagine that stronger API

guarantees

will be
cold comfort in the face of a 1.0 ->   2.0 upgrade.  However, if

we

had

been
using semver all along, there would have been much less pain for

user

Re: Official Guidance to users regarding removal of functions

[Christopher]

Yes, I agree you should just reintroduce the method if it was removed in
1.7, which is not yet released.
If it was removed in 1.6.{0,1}, you could also consider reintroducing it
for 1.6.2, to support those users also.

--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Thu, Dec 11, 2014 at 4:59 PM, Josh Elser  wrote:

> For context and as a good example of how we could be less abrasive, I'm
> unexpectedly spending today updating Hive with a bunch of new reflection
> because methods in 1.5 on AccumuloInputFormat no longer exist in 1.7.
> setZooKeeperInstance used to take two Strings, but in 1.7 this version of
> the method was removed and forces client to use ClientConfiguration.
>
> This is a good example of where we didn't have to do this and there was
> next to no maintenance support on us to preserve this method.
>
> Honestly, after writing this, I'm tempted to reinstate the old version
> because doing this amount of reflection is painful and silly for clients
> (and I even know what I'm doing most days!)
>
> Mike Drob wrote:
>
>> FWIW, Java 9 is dropping methods for the first time that the JDK API is
>> dropping methods. Almost 20 years. Just something to consider, when we
>> approach the conversation of how long to keep APIs around.
>>
>> On Thu, Dec 11, 2014 at 3:46 PM, Christopher  wrote:
>>
>>  You seem to be dismissing "clean up" as an invalid. I disagree. There are
>>> costs to keeping around deprecated APIs. That's not to say we shouldn't
>>> keep them around for a long time... we can and perhaps should. But "clean
>>> up" is just shorthand for all the maintainability, readability, and
>>> usability costs. As such, I feel it incapsulates many reasons which are
>>> sometimes difficult to express, but still valid.
>>>
>>> With semver in place, it seems like your position could be reduced to
>>> "support major versions longer", which is a perfectly fine goal. As long
>>> as
>>> we have that, there's really no need to upgrade to the "latest", unless
>>> there's a feature that you want which is difficult to backport in a minor
>>> release. In that case, I think it's perfectly reasonable to consider the
>>> risks of upgrading and to make the choice to upgrade and have the feature
>>> or not upgrade and avoid API changes which affect you (if any).
>>>
>>>
>>> --
>>> Christopher L Tubbs II
>>> http://gravatar.com/ctubbsii
>>>
>>> On Thu, Dec 11, 2014 at 3:19 PM, Kepner, Jeremy - 0553 - MITLL<
>>> kep...@ll.mit.edu>  wrote:
>>>
>>>  So I think the bigger issue is to revisit the imperative to delete
 deprecated functions (both public&  private).  How many instances have
 we
 had where deleting a deprecated API (public&  private) did anything more

 than "clean up" the code?

 On Dec 11, 2014, at 2:39 PM, John Vines  wrote:

  More likely we'd have a fully backwards compatible API for each major
> version. SemVer allows for it and I think that grants us enough room
>
 for
>>>
 growth while still securing things for future releases.
>
> On Thu, Dec 11, 2014 at 2:36 PM, Adam Fuchs  wrote:
>
>  Awesome -- ACCUMULO-2589 gets us at least halfway there. Given this,
>> what would be the challenges in having and maintaining one API project
>> for each major version ever released?
>>
>> Adam
>>
>> On Thu, Dec 11, 2014 at 2:24 PM, Josh Elser
>>
> wrote:

> Adam Fuchs wrote:
>>>
 Has anybody looked into separating the public API a bit more from

>>> the
>>>
 core? It seems to me that a large number of the deprecation removal
 issues are related to people failing to read section 9 of the

>>> README.
>>>
 It would be great if we built an API jar that people could build
 against, but didn't leak internal classes. Maybe this is something

>>> we
>>>
 can shoot for in the 2.0 release?

>>>
>>> Yup, this is already in the works by Christopher as a part of
>>>
>> ACCUMULO-2589.
>>
>>>
>>>  Taking that a step further, it would be great if we released a 1.x

>>> API
>>>
 compatible client jar for every 2.x or later release. Does anybody
 have a feel for the maintenance costs of such a thing? Certainly
 changes to configuration options and metadata table structures will
 prove challenging. Given that we don't have a history of removing
 functionality, this ought to at least be feasible.

 Thoughts?

 Adam


 On Thu, Dec 11, 2014 at 1:54 PM, Jeremy Kepner

>>> wrote:
>>
>>> So the simple solution is to deprecate often, but remove almost
>
 never.

> It is very rare that leaving a deprecated API in place actually
>
 has a
>>>
 negative impact.
> The code gets a little less clean, but that's fine as long as
>
 things

Re: Official Guidance to users regarding removal of functions

[Josh Elser]

Great, thanks for the feedback. I just committed this in ACCUMULO-3403

Christopher wrote:

Yes, I agree you should just reintroduce the method if it was removed in
1.7, which is not yet released.
If it was removed in 1.6.{0,1}, you could also consider reintroducing it
for 1.6.2, to support those users also.

--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Thu, Dec 11, 2014 at 4:59 PM, Josh Elser  wrote:


For context and as a good example of how we could be less abrasive, I'm
unexpectedly spending today updating Hive with a bunch of new reflection
because methods in 1.5 on AccumuloInputFormat no longer exist in 1.7.
setZooKeeperInstance used to take two Strings, but in 1.7 this version of
the method was removed and forces client to use ClientConfiguration.

This is a good example of where we didn't have to do this and there was
next to no maintenance support on us to preserve this method.

Honestly, after writing this, I'm tempted to reinstate the old version
because doing this amount of reflection is painful and silly for clients
(and I even know what I'm doing most days!)

Mike Drob wrote:


FWIW, Java 9 is dropping methods for the first time that the JDK API is
dropping methods. Almost 20 years. Just something to consider, when we
approach the conversation of how long to keep APIs around.

On Thu, Dec 11, 2014 at 3:46 PM, Christopher   wrote:

  You seem to be dismissing "clean up" as an invalid. I disagree. There are

costs to keeping around deprecated APIs. That's not to say we shouldn't
keep them around for a long time... we can and perhaps should. But "clean
up" is just shorthand for all the maintainability, readability, and
usability costs. As such, I feel it incapsulates many reasons which are
sometimes difficult to express, but still valid.

With semver in place, it seems like your position could be reduced to
"support major versions longer", which is a perfectly fine goal. As long
as
we have that, there's really no need to upgrade to the "latest", unless
there's a feature that you want which is difficult to backport in a minor
release. In that case, I think it's perfectly reasonable to consider the
risks of upgrading and to make the choice to upgrade and have the feature
or not upgrade and avoid API changes which affect you (if any).


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Thu, Dec 11, 2014 at 3:19 PM, Kepner, Jeremy - 0553 - MITLL<
kep...@ll.mit.edu>   wrote:

  So I think the bigger issue is to revisit the imperative to delete

deprecated functions (both public&   private).  How many instances have
we
had where deleting a deprecated API (public&   private) did anything more

than "clean up" the code?

On Dec 11, 2014, at 2:39 PM, John Vines   wrote:

  More likely we'd have a fully backwards compatible API for each major

version. SemVer allows for it and I think that grants us enough room


for
growth while still securing things for future releases.

On Thu, Dec 11, 2014 at 2:36 PM, Adam Fuchs   wrote:

  Awesome -- ACCUMULO-2589 gets us at least halfway there. Given this,

what would be the challenges in having and maintaining one API project
for each major version ever released?

Adam

On Thu, Dec 11, 2014 at 2:24 PM, Josh Elser


wrote:
Adam Fuchs wrote:

Has anybody looked into separating the public API a bit more from


the

core? It seems to me that a large number of the deprecation removal

issues are related to people failing to read section 9 of the


README.

It would be great if we built an API jar that people could build

against, but didn't leak internal classes. Maybe this is something


we

can shoot for in the 2.0 release?

Yup, this is already in the works by Christopher as a part of


ACCUMULO-2589.


  Taking that a step further, it would be great if we released a 1.x
API

compatible client jar for every 2.x or later release. Does anybody

have a feel for the maintenance costs of such a thing? Certainly
changes to configuration options and metadata table structures will
prove challenging. Given that we don't have a history of removing
functionality, this ought to at least be feasible.

Thoughts?

Adam


On Thu, Dec 11, 2014 at 1:54 PM, Jeremy Kepner


wrote:
So the simple solution is to deprecate often, but remove almost

never.

It is very rare that leaving a deprecated API in place actually

has a

negative impact.

The code gets a little less clean, but that's fine as long as


things

are

clearly labeled as deprecated.

In fact, seeing the way something used to be done can often be an
inspiration for something new.
If the past is deleted, then that knowledge is lost.

I am not saying deleting can never happen, I am just saying that


when

it

does, it is because

there absolutely no choice.  Deletion to "clean up the code"


shouldn't

be

a valid reason for deletion.

On Thu, Dec 11, 2014 at 12:58:13PM -0500, Christopher wrote:


I don't know that it'd be "cold comfort". We can continue to


support

1.x

for some time, if we choose.

Re: accumulo Scanner

[panqing...@163.com]

I try to use the WholeRowIterator, the same rowkey data into a line, Now,
Value contains ColumnFamily, ColumnQualifier, value，but the value of Value
should be how to analysis? 

 for (Entry entry : scanner) {
log.info("" + entry.getKey() + "," + entry.getValue());
}




--
View this message in context: 
http://apache-accumulo.1065345.n5.nabble.com/accumulo-Scanner-tp12506p12552.html
Sent from the Developers mailing list archive at Nabble.com.

Re: accumulo Scanner

[Corey Nolet]

You're going to want to use WholeRowIterator.decodeRow(entry.getKey(),
entry.getValue()) for that one. You can do:

for(Entry entry : scanner) {
   for(Entry actualEntry :
WholeRowIterator.decodeRow(entry.getKey(), entry.getValue()).entrySet()) {
// do something with actualEntry
   }
}

On Thu, Dec 11, 2014 at 10:24 PM, panqing...@163.com 
wrote:

> I try to use the WholeRowIterator, the same rowkey data into a line, Now,
> Value contains ColumnFamily, ColumnQualifier, value，but the value of Value
> should be how to analysis?
>
>  for (Entry entry : scanner) {
> log.info("" + entry.getKey() + "," + entry.getValue());
> }
>
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/accumulo-Scanner-tp12506p12552.html
> Sent from the Developers mailing list archive at Nabble.com.
>