Re: [VOTE] Apache Accumulo 1.6.1 RC1
Rescinding my -1 vote. Josh helped me figure this one out on IRC. There used to be a tag named 1.6.1-rc1. That no longer exists at apache repo, now there is a branch called 1.6.1-rc. I had both the old tag and the new branch, git was taking the tag. git fetch --prune did not remove the tag, I had to manually remove it. When I compare src tar to correct branch, no diffs. On Wed, Sep 24, 2014 at 10:53 AM, Keith Turner ke...@deenlo.com wrote: -1 GOOD Sigs and hashes for bin.tar.gz and src.tar.gz look good Successfully ran mutslam[1] against 1.6.1 rc1, using staging repo. Ci w/ Agitation verify ran successfully. Env hadoop 2.3.0, ZK 3.4.5, Centos 6, 20 node EC2. 17B ingested. [1]: https://github.com/keith-turner/mutslam BAD I am seeing differences between the src tar ball and the tag. Can anyone else confirm? [git]$ tar xzf accumulo-1.6.1-src.tar.gz [git]$ md5sum accumulo-1.6.1-src.tar.gz c9f0c51e97cb6ec6cb55190abfbd9aff accumulo-1.6.1-src.tar.gz [git]$ cd accumulo [accumulo]$ git status # HEAD detached at 1.6.1-rc1 nothing to commit, working directory clean [accumulo]$ git clean -xfd [accumulo]$ git log | head commit 93d1cbe3663014286219ebe6fd6d594bbd90667b Author: Corey J. Nolet cjno...@gmail.com Date: Fri Sep 19 00:45:07 2014 -0400 [maven-release-plugin] prepare release 1.6.1 commit ddd2c3bc098531d8dadee1b5142cfb80c50f4c41 Author: Corey J. Nolet cjno...@gmail.com Date: Thu Sep 18 20:11:41 2014 -0400 [accumulo]$ cd .. [git]$ diff -r -q accumulo accumulo-1.6.1 | fgrep -v .git Files accumulo/CHANGES and accumulo-1.6.1/CHANGES differ Files accumulo/minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloConfigImpl.java and accumulo-1.6.1/minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloConfigImpl.java differ Files accumulo/minicluster/src/test/java/org/apache/accumulo/minicluster/MiniAccumuloClusterTest.java and accumulo-1.6.1/minicluster/src/test/java/org/apache/accumulo/minicluster/MiniAccumuloClusterTest.java differ Files accumulo/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java and accumulo-1.6.1/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java differ Files accumulo/server/monitor/src/main/java/org/apache/accumulo/monitor/servlets/ScanServlet.java and accumulo-1.6.1/server/monitor/src/main/java/org/apache/accumulo/monitor/servlets/ScanServlet.java differ Files accumulo/server/tserver/src/main/java/org/apache/accumulo/tserver/Tablet.java and accumulo-1.6.1/server/tserver/src/main/java/org/apache/accumulo/tserver/Tablet.java differ Files accumulo/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java and accumulo-1.6.1/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java differ Files accumulo/test/src/test/java/org/apache/accumulo/test/functional/HalfDeadTServerIT.java and accumulo-1.6.1/test/src/test/java/org/apache/accumulo/test/functional/HalfDeadTServerIT.java differ Files accumulo/test/system/continuous/continuous-env.sh.example and accumulo-1.6.1/test/system/continuous/continuous-env.sh.example differ On Fri, Sep 19, 2014 at 10:49 PM, Corey Nolet cjno...@gmail.com wrote: Devs, Please consider the following candidate for Apache Accumulo 1.6.1 Branch: 1.6.1-rc1 SHA1: 88c5473b3b49d797d3dabebd12fe517e9b248ba2 Staging Repository: * https://repository.apache.org/content/repositories/orgapacheaccumulo-1017/ https://repository.apache.org/content/repositories/orgapacheaccumulo-1017/ * Source tarball: * http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-src.tar.gz http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-src.tar.gz * Binary tarball: * http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-bin.tar.gz http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-bin.tar.gz * (Append .sha1, .md5 or .asc to download the signature/hash for a given artifact.) Signing keys available at: https://www.apache.org/dist/accumulo/KEYS Over 1.6.1, we have 188 issues resolved * https://git-wip-us.apache.org/repos/asf?p=accumulo.git;a=blob;f=CHANGES;h=91b9d31e3b9dc53f1a576cc49bbc061919eb0070;hb=1.6.1-rc1 https://git-wip-us.apache.org/repos/asf?p=accumulo.git;a=blob;f=CHANGES;h=91b9d31e3b9dc53f1a576cc49bbc061919eb0070;hb=1.6.1-rc1 * Testing: All unit and functional tests are passing. Vote will be open until Thursday, September 25th 12:00AM UTC (9/24 8:00PM ET, 9/24 5:00PM PT)
Re: [VOTE] Apache Accumulo 1.6.1 RC1
Accumulo server logs are ~118M. The CI logs are tiny. Hadoop logs I have are around 126M (although probably relevant bits are much smaller). Sadly, I didn't run with archived WALs, so I'm not sure how useful the server logs are on their own. Sean Busbey wrote: Josh, how big are all the logs? On Tue, Sep 23, 2014 at 9:43 PM, Josh Elserjosh.el...@gmail.com wrote: Well, color me shocked -- the verify found some bad data. It looks like two keys have bad checksums (which I assume is what created the UNDEFINEDs, too?). CORRUPT 2 REFERENCED 219908 UNDEFINED 2 UNREFERENCED 874770 I ran two tabletservers on my desktop, turned on hflush instead of hsync, switched from GZ to snappy and upped the splits threshold for 4g and let CI run for ~5 hours. I killed the tservers about a dozen times by hand throughout the day (kill -9), and the master once or twice. The datanode was left alone. This was running on 2.6.0-SNAPSHOT from around 9/14/2014. The offending keys are: 389a85668b6ebf8e 2ff6:4a78 [] 1411499115242 3a10885b-d481-4d00-be00-0477e231ey65:8576b169:0cd98965c9ccc1d0:ba15529e and 7e56b58a0c7df128 5fa0:6249 [] 1411499311578 3a10885b-d481-4d00-be00-0477e231e965:p000872d60eb:499fa72752d82a7c:5c5f19e8 which both happened a little after 3:00pm eastern (I stopped CI around 3:30pm eastern). I don't see anything immediately wrong in the tserver logs (nor does it appear that I had restarted either of them around the timestamp of the above keys). I see no errors in the DN logs either around that time window. I don't have a clue how to even start looking at this to figure out if something indeed went wrong, or if it's some other sort of issue. To be clear, this as it stands isn't sufficient to make me change my vote. On Tue, Sep 23, 2014 at 3:04 PM, Josh Elserjosh.el...@gmail.com wrote: +1 * Verified checksums+sigs * Build from source tarball and ran all unit+functional tests against Apache Hadoop 2.5.1 and 2.6.0-SNAPSHOT * Ingested 2B records w/ CI + clean verify with single tserver (Apache Hadoop 2.6.0-SNAPSHOT + Apache ZooKeeper 3.4.5) * Ingested ~2.5B records w/ CI with 2 tservers and some manual agitation (Apache Hadoop 2.6.0-SNAPSHOT + Apache ZooKeeper 3.4.5) - Currently running verify, will report if I get a failed verify * Ran some Hive queries (w/ Apache Hive-0.14.0-SNAPSHOT Apache Tez 0.6.0-SNAPSHOT) * Ran some Pig queries (w/ Apache Pig-0.13.0) Thanks for organizing this, Corey!! On Fri, Sep 19, 2014 at 10:49 PM, Corey Noletcjno...@gmail.com wrote: Devs, Please consider the following candidate for Apache Accumulo 1.6.1 Branch: 1.6.1-rc1 SHA1: 88c5473b3b49d797d3dabebd12fe517e9b248ba2 Staging Repository: * https://repository.apache.org/content/repositories/orgapacheaccumulo-1017/ https://repository.apache.org/content/repositories/orgapacheaccumulo-1017/ * Source tarball: * http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-src.tar.gz http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-src.tar.gz * Binary tarball: * http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-bin.tar.gz http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-bin.tar.gz * (Append .sha1, .md5 or .asc to download the signature/hash for a given artifact.) Signing keys available at: https://www.apache.org/dist/accumulo/KEYS Over 1.6.1, we have 188 issues resolved * https://git-wip-us.apache.org/repos/asf?p=accumulo.git;a=blob;f=CHANGES;h=91b9d31e3b9dc53f1a576cc49bbc061919eb0070;hb=1.6.1-rc1 https://git-wip-us.apache.org/repos/asf?p=accumulo.git;a=blob;f=CHANGES;h=91b9d31e3b9dc53f1a576cc49bbc061919eb0070;hb=1.6.1-rc1 * Testing: All unit and functional tests are passing. Vote will be open until Thursday, September 25th 12:00AM UTC (9/24 8:00PM ET, 9/24 5:00PM PT)
Re: [VOTE] Apache Accumulo 1.6.1 RC1
+1 I did a little more poking at 1.6.1 and it looks good. Thanks Corey and Josh for putting these releases together. Sigs and hashes for bin.tar.gz and src.tar.gz look good Successfully ran mutslam[1] against 1.6.1 rc1, using staging repo. Ci w/ Agitation verify ran successfully. Env hadoop 2.3.0, ZK 3.4.5, Centos 6, 20 node EC2. 17B ingested. [1]: https://github.com/keith-turner/mutslam On Fri, Sep 19, 2014 at 10:49 PM, Corey Nolet cjno...@gmail.com wrote: Devs, Please consider the following candidate for Apache Accumulo 1.6.1 Branch: 1.6.1-rc1 SHA1: 88c5473b3b49d797d3dabebd12fe517e9b248ba2 Staging Repository: * https://repository.apache.org/content/repositories/orgapacheaccumulo-1017/ https://repository.apache.org/content/repositories/orgapacheaccumulo-1017/ * Source tarball: * http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-src.tar.gz http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-src.tar.gz * Binary tarball: * http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-bin.tar.gz http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-bin.tar.gz * (Append .sha1, .md5 or .asc to download the signature/hash for a given artifact.) Signing keys available at: https://www.apache.org/dist/accumulo/KEYS Over 1.6.1, we have 188 issues resolved * https://git-wip-us.apache.org/repos/asf?p=accumulo.git;a=blob;f=CHANGES;h=91b9d31e3b9dc53f1a576cc49bbc061919eb0070;hb=1.6.1-rc1 https://git-wip-us.apache.org/repos/asf?p=accumulo.git;a=blob;f=CHANGES;h=91b9d31e3b9dc53f1a576cc49bbc061919eb0070;hb=1.6.1-rc1 * Testing: All unit and functional tests are passing. Vote will be open until Thursday, September 25th 12:00AM UTC (9/24 8:00PM ET, 9/24 5:00PM PT)
Re: [VOTE] Apache Accumulo 1.6.1 RC1
Bill, I've been having that same IT issue and said the same thing It's not happening to others. I lifted the timeout completely and it never finished. On Wed, Sep 24, 2014 at 1:13 PM, Mike Drob mad...@cloudera.com wrote: Any chance the IRC chats can make it only the ML for posterity? Mike On Wed, Sep 24, 2014 at 12:04 PM, Keith Turner ke...@deenlo.com wrote: On Wed, Sep 24, 2014 at 12:44 PM, Russ Weeks rwe...@newbrightidea.com wrote: Interesting that y (0x79) and 9 (0x39) are one bit away from each other. I blame cosmic rays! It is interesting, and thats only half of the story. Its been interesting chatting w/ Josh about this on irc and hearing about his findings. On Wed, Sep 24, 2014 at 9:05 AM, Josh Elser josh.el...@gmail.com wrote: The offending keys are: 389a85668b6ebf8e 2ff6:4a78 [] 1411499115242 3a10885b-d481-4d00-be00-0477e231ey65:8576b169: 0cd98965c9ccc1d0:ba15529e The careful eye will notice that the UUID in the first component of the value has a different suffix than the next corrupt key/value (ends with ey65 instead of e965). Fixing this in the Value and re-running the CRC makes it pass. and 7e56b58a0c7df128 5fa0:6249 [] 1411499311578 3a10885b-d481-4d00-be00-0477e231e965:p000872d60eb: 499fa72752d82a7c:5c5f19e8
[GitHub] accumulo pull request: ACCUMULO-3089: Create volume chooser from t...
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/16#discussion_r17987075
--- Diff:
core/src/main/java/org/apache/accumulo/core/client/admin/TableOperations.java
---
@@ -103,6 +103,25 @@
void create(String tableName, boolean versioningIter, TimeType timeType)
throws AccumuloException, AccumuloSecurityException, TableExistsException;
/**
+ * @param tableName
+ * the name of the table
+ * @param limitVersion
+ * Enables/disables the versioning iterator, which will limit
the number of Key versions kept.
+ * @param timeType
+ * specifies logical or real-time based time recording for
entries in the table
+ * @param properties
+ * initial table properties the user wants
+ * @throws AccumuloException
+ * if a general error occurs
+ * @throws AccumuloSecurityException
+ * if the user does not have permission
+ * @throws TableExistsException
+ * if the table already exists
+ */
+ void create(String tableName, boolean limitVersion, TimeType timeType,
MapString,String properties) throws AccumuloException,
AccumuloSecurityException,
--- End diff --
Instead of introducing yet another create table method, we could create a
NewTableConfiguration class and pass that.
```java
public class NewTableConfiguration {
public NewTableConfiguration setTimeType(TimeType tt){...}
public NewTableConfiguration setLimitVersions(boolean lv){...}
public NewTableConfiguration setProperties(MapString,String
properties){...}
}
```
And add a method like the following, and possibly deprecated existing
create table methods (except for create(String) )
```java
void create(String tableName, NewTableConfig ntc)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request: ACCUMULO-3089: Create volume chooser from t...
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/16#discussion_r17987372
--- Diff: test/src/test/java/org/apache/accumulo/test/VolumeChooserIT.java
---
@@ -0,0 +1,670 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.Map.Entry;
+
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.BatchWriterConfig;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.metadata.MetadataTable;
+import
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.minicluster.impl.MiniAccumuloConfigImpl;
+import org.apache.accumulo.test.functional.ConfigurableMacIT;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RawLocalFileSystem;
+import org.apache.hadoop.io.Text;
+import org.junit.Test;
+
+/**
+ *
+ */
+public class VolumeChooserIT extends ConfigurableMacIT {
+
+ private static final Text EMPTY = new Text();
+ private static final Value EMPTY_VALUE = new Value(new byte[] {});
+ private File volDirBase;
+ private Path v1, v2, v3, v4;
+
+ @Override
+ public void configure(MiniAccumuloConfigImpl cfg, Configuration
hadoopCoreSite) {
+// Get 2 tablet servers
+cfg.setNumTservers(2);
+
+// Set the general volume chooser to the GeneralVolumeChooser so that
different choosers can be specified
+MapString,String siteConfig = new HashMapString,String();
+siteConfig.put(Property.GENERAL_VOLUME_CHOOSER.getKey(),
org.apache.accumulo.server.fs.GeneralVolumeChooser.class.getName());
+cfg.setSiteConfig(siteConfig);
+
+// Set up 4 different volume paths
+File baseDir = cfg.getDir();
+volDirBase = new File(baseDir, volumes);
+File v1f = new File(volDirBase, v1);
+File v2f = new File(volDirBase, v2);
+File v3f = new File(volDirBase, v3);
+File v4f = new File(volDirBase, v4);
+v1f.mkdir();
+v2f.mkdir();
+v4f.mkdir();
+v1 = new Path(file:// + v1f.getAbsolutePath());
+v2 = new Path(file:// + v2f.getAbsolutePath());
+v3 = new Path(file:// + v3f.getAbsolutePath());
+v4 = new Path(file:// + v4f.getAbsolutePath());
+
+// Only add volumes 1, 2, and 4 to the list of instance volumes to
have one volume that isn't in the options list when they are choosing
+cfg.setProperty(Property.INSTANCE_VOLUMES, v1.toString() + , +
v2.toString() + , + v4.toString());
+
+// use raw local file system so walogs sync and flush will work
+hadoopCoreSite.set(fs.file.impl, RawLocalFileSystem.class.getName());
+
+super.configure(cfg, hadoopCoreSite);
+
+ }
+
+ // Test that uses two tables with 10 split points each. They each use
the StaticVolumeChooser to choose volumes.
+ @Test(timeout = 60 * 1000)
+ public void twoTablesStaticVolumeChooser() throws Exception {
+log.info(Starting StaticVolumeChooser);
+
+// Create and populate initial properties map for creating table 1
+MapString,String properties = new HashMapString,String();
+String propertyName = table.custom.chooser;
+String volume =
[GitHub] accumulo pull request: ACCUMULO-3089: Create volume chooser from t...
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/16#discussion_r17987487
--- Diff: test/src/test/java/org/apache/accumulo/test/VolumeChooserIT.java
---
@@ -0,0 +1,670 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.Map.Entry;
+
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.BatchWriterConfig;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.metadata.MetadataTable;
+import
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.minicluster.impl.MiniAccumuloConfigImpl;
+import org.apache.accumulo.test.functional.ConfigurableMacIT;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RawLocalFileSystem;
+import org.apache.hadoop.io.Text;
+import org.junit.Test;
+
+/**
+ *
+ */
+public class VolumeChooserIT extends ConfigurableMacIT {
+
+ private static final Text EMPTY = new Text();
+ private static final Value EMPTY_VALUE = new Value(new byte[] {});
+ private File volDirBase;
+ private Path v1, v2, v3, v4;
+
+ @Override
+ public void configure(MiniAccumuloConfigImpl cfg, Configuration
hadoopCoreSite) {
+// Get 2 tablet servers
+cfg.setNumTservers(2);
+
+// Set the general volume chooser to the GeneralVolumeChooser so that
different choosers can be specified
+MapString,String siteConfig = new HashMapString,String();
+siteConfig.put(Property.GENERAL_VOLUME_CHOOSER.getKey(),
org.apache.accumulo.server.fs.GeneralVolumeChooser.class.getName());
+cfg.setSiteConfig(siteConfig);
+
+// Set up 4 different volume paths
+File baseDir = cfg.getDir();
+volDirBase = new File(baseDir, volumes);
+File v1f = new File(volDirBase, v1);
+File v2f = new File(volDirBase, v2);
+File v3f = new File(volDirBase, v3);
+File v4f = new File(volDirBase, v4);
+v1f.mkdir();
+v2f.mkdir();
+v4f.mkdir();
+v1 = new Path(file:// + v1f.getAbsolutePath());
+v2 = new Path(file:// + v2f.getAbsolutePath());
+v3 = new Path(file:// + v3f.getAbsolutePath());
+v4 = new Path(file:// + v4f.getAbsolutePath());
+
+// Only add volumes 1, 2, and 4 to the list of instance volumes to
have one volume that isn't in the options list when they are choosing
+cfg.setProperty(Property.INSTANCE_VOLUMES, v1.toString() + , +
v2.toString() + , + v4.toString());
+
+// use raw local file system so walogs sync and flush will work
+hadoopCoreSite.set(fs.file.impl, RawLocalFileSystem.class.getName());
+
+super.configure(cfg, hadoopCoreSite);
+
+ }
+
+ // Test that uses two tables with 10 split points each. They each use
the StaticVolumeChooser to choose volumes.
+ @Test(timeout = 60 * 1000)
+ public void twoTablesStaticVolumeChooser() throws Exception {
+log.info(Starting StaticVolumeChooser);
+
+// Create and populate initial properties map for creating table 1
+MapString,String properties = new HashMapString,String();
+String propertyName = table.custom.chooser;
+String volume =
[GitHub] accumulo pull request: ACCUMULO-3089: Create volume chooser from t...
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/16#discussion_r17987756
--- Diff:
server/base/src/main/java/org/apache/accumulo/server/fs/VolumeChooser.java ---
@@ -18,5 +18,7 @@
public interface VolumeChooser {
+ String choose(VolumeChooserEnvironment env, String[] options);
--- End diff --
ugh, this will break any current code that has implemented the
interface Java 8 has a solution for this, but not 7
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request: ACCUMULO-3089: Create volume chooser from t...
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/16#discussion_r17987986
--- Diff:
server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManagerImpl.java
---
@@ -560,8 +565,22 @@ public ContentSummary getContentSummary(Path dir)
throws IOException {
}
@Override
- public String choose(String[] options) {
-return chooser.choose(options);
+ public String choose(OptionalString tableId, String[] options) {
+// If the tableId is present, use that to create a
VolumeChooserEnvironment variable
+if (tableId.isPresent()) {
+ // Get the current instance and from that the
ServerConfigurationFactory and in turn the tableId
+ Instance instance = HdfsZooInstance.getInstance();
+ ServerConfigurationFactory serverConf = new
ServerConfigurationFactory(instance);
+ TableConfiguration tableConf =
serverConf.getTableConfiguration(tableId.get());
+
+ // Create the environment and then choose the volume using the
currently defined chooser
+ VolumeChooserEnvironment env = new
VolumeChooserEnvironment(tableConf);
+ return chooser.choose(env, options);
+} else {
+ // If the tableId is missing, then just choose using the current
chooser, without using the per table properties
+ log.info(TABLE ID MISSING);
--- End diff --
this is not a very useful log message, and I assume it will occur regularly
in the case when a volume is chosen for walogs... maybe drop it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request: ACCUMULO-3089: Create volume chooser from t...
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/16#discussion_r17988361
--- Diff: test/src/test/java/org/apache/accumulo/test/VolumeChooserIT.java
---
@@ -0,0 +1,670 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.Map.Entry;
+
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.BatchWriterConfig;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.metadata.MetadataTable;
+import
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.minicluster.impl.MiniAccumuloConfigImpl;
+import org.apache.accumulo.test.functional.ConfigurableMacIT;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RawLocalFileSystem;
+import org.apache.hadoop.io.Text;
+import org.junit.Test;
+
+/**
+ *
+ */
+public class VolumeChooserIT extends ConfigurableMacIT {
+
+ private static final Text EMPTY = new Text();
+ private static final Value EMPTY_VALUE = new Value(new byte[] {});
+ private File volDirBase;
+ private Path v1, v2, v3, v4;
+
+ @Override
+ public void configure(MiniAccumuloConfigImpl cfg, Configuration
hadoopCoreSite) {
+// Get 2 tablet servers
+cfg.setNumTservers(2);
+
+// Set the general volume chooser to the GeneralVolumeChooser so that
different choosers can be specified
+MapString,String siteConfig = new HashMapString,String();
+siteConfig.put(Property.GENERAL_VOLUME_CHOOSER.getKey(),
org.apache.accumulo.server.fs.GeneralVolumeChooser.class.getName());
+cfg.setSiteConfig(siteConfig);
+
+// Set up 4 different volume paths
+File baseDir = cfg.getDir();
+volDirBase = new File(baseDir, volumes);
+File v1f = new File(volDirBase, v1);
+File v2f = new File(volDirBase, v2);
+File v3f = new File(volDirBase, v3);
+File v4f = new File(volDirBase, v4);
+v1f.mkdir();
+v2f.mkdir();
+v4f.mkdir();
+v1 = new Path(file:// + v1f.getAbsolutePath());
+v2 = new Path(file:// + v2f.getAbsolutePath());
+v3 = new Path(file:// + v3f.getAbsolutePath());
+v4 = new Path(file:// + v4f.getAbsolutePath());
+
+// Only add volumes 1, 2, and 4 to the list of instance volumes to
have one volume that isn't in the options list when they are choosing
+cfg.setProperty(Property.INSTANCE_VOLUMES, v1.toString() + , +
v2.toString() + , + v4.toString());
+
+// use raw local file system so walogs sync and flush will work
+hadoopCoreSite.set(fs.file.impl, RawLocalFileSystem.class.getName());
+
+super.configure(cfg, hadoopCoreSite);
+
+ }
+
+ // Test that uses two tables with 10 split points each. They each use
the StaticVolumeChooser to choose volumes.
+ @Test(timeout = 60 * 1000)
+ public void twoTablesStaticVolumeChooser() throws Exception {
+log.info(Starting StaticVolumeChooser);
+
+// Create and populate initial properties map for creating table 1
+MapString,String properties = new HashMapString,String();
+String propertyName = table.custom.chooser;
+String volume =
[GitHub] accumulo pull request: ACCUMULO-3089: Create volume chooser from t...
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/16#discussion_r17989515
--- Diff:
server/base/src/main/java/org/apache/accumulo/server/fs/GeneralVolumeChooser.java
---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.accumulo.server.fs;
+
+import java.util.Map;
+
+import org.apache.accumulo.core.conf.AccumuloConfiguration.AllFilter;
+import org.apache.accumulo.core.conf.AccumuloConfiguration.PropertyFilter;
+import org.apache.accumulo.start.classloader.vfs.AccumuloVFSClassLoader;
+import org.apache.log4j.Logger;
+
+public class GeneralVolumeChooser implements VolumeChooser {
--- End diff --
User facing documentation is needed somewhere to inform users how they can
put these differently volume choosers and the custom props together. Should
also tell explain why they should use create table w/ props for this.
Could do that as javadoc on these volume choosers and/or in the user manual.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request: ACCUMULO-3089: Create volume chooser from t...
Github user keith-turner commented on a diff in the pull request: https://github.com/apache/accumulo/pull/16#discussion_r17989615 --- Diff: core/src/main/java/org/apache/accumulo/core/client/admin/TableOperations.java --- @@ -103,6 +103,25 @@ void create(String tableName, boolean versioningIter, TimeType timeType) throws AccumuloException, AccumuloSecurityException, TableExistsException; /** + * @param tableName + * the name of the table + * @param limitVersion + * Enables/disables the versioning iterator, which will limit the number of Key versions kept. + * @param timeType + * specifies logical or real-time based time recording for entries in the table + * @param properties + * initial table properties the user wants + * @throws AccumuloException + * if a general error occurs + * @throws AccumuloSecurityException + * if the user does not have permission + * @throws TableExistsException + * if the table already exists + */ + void create(String tableName, boolean limitVersion, TimeType timeType, MapString,String properties) throws AccumuloException, AccumuloSecurityException, --- End diff -- This new functionality probably needs to be exposed in the shell and proxy. This occurred to me when I was thinking about writing user documentation for this. The documentation would probably use the shell. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Thoughts on scaling back github integration
Devs, Github integration is nice... but, I'm not a fan of how comments on diffs are posted to the JIRAs. Each line note is a separate comment and it includes a ton of irrelevant diff output. Looking at ACCUMULO-3089 is what triggered this reaction for me. Does anybody else have thoughts on this? I don't know what our integration options are, but I would prefer something less intrusive. It's really easy to lose the actual discussion with what we have now. Mike
Re: Thoughts on scaling back github integration
I was thinking about this... and I was wondering if this is done by the ASF integration could batch these comments for mailing to the dev list and/or commenting on JIRA. I really like using GitHub for reviews/comments. Issuing pull requests is very seamless, but the comment activity is a bit annoying. -- Christopher L Tubbs II http://gravatar.com/ctubbsii On Wed, Sep 24, 2014 at 2:47 PM, Mike Drob mad...@cloudera.com wrote: Devs, Github integration is nice... but, I'm not a fan of how comments on diffs are posted to the JIRAs. Each line note is a separate comment and it includes a ton of irrelevant diff output. Looking at ACCUMULO-3089 is what triggered this reaction for me. Does anybody else have thoughts on this? I don't know what our integration options are, but I would prefer something less intrusive. It's really easy to lose the actual discussion with what we have now. Mike
