[jira] [Commented] (ACCUMULO-2907) Invalidate "this may not be applicable for your security setup" warning from initialize
[ https://issues.apache.org/jira/browse/ACCUMULO-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175711#comment-16175711 ] Mark Owens commented on ACCUMULO-2907: -- Hi [~ctubbsii], Thanks for the comment. I proceeded with the change based on the comment from [~elserj] above where he stated it was completely irrelevant given prior comments. If that is not the case I'm willing to take a deeper look into the issue, although I'm sure I would need some assistance given that I'm just getting into the code base. > Invalidate "this may not be applicable for your security setup" warning from > initialize > > > Key: ACCUMULO-2907 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2907 > Project: Accumulo > Issue Type: Improvement >Reporter: Josh Elser >Assignee: Mark Owens >Priority: Minor > Labels: newbie > > The warning that is printed about setting a root password may not be > applicable for your security setup is invalid because the plugable > authentication modules do not manage the root user and it is expected that > there will always be a local root user with a password. > Remove the warning. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] jmark99 opened a new pull request #301: Accumulo 2907 Invalidate "this may not be applicable for your security setup" warning from initialize
jmark99 opened a new pull request #301: Accumulo 2907 Invalidate "this may not be applicable for your security setup" warning from initialize URL: https://github.com/apache/accumulo/pull/301 ACCUMULO-2907 Invalidate "this may not be applicable for your security setup" warning from initialize Updated console message in the getRootPassword method to remove unnecessary warning about the users security setup. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (ACCUMULO-2907) Invalidate "this may not be applicable for your security setup" warning from initialize
[ https://issues.apache.org/jira/browse/ACCUMULO-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175650#comment-16175650 ] Christopher Tubbs commented on ACCUMULO-2907: - Hi [~jmark99]. Although this issue is currently labeled "newbie", I think you may find that this depends on the work in ACCUMULO-1300 being complete, which would be much further in depth. I do not recall whether or not the warning is still valid in the current pluggable security implementation. If you're still interested, and need assistance, please feel free to reach out to me, or anybody else on the mailing list. > Invalidate "this may not be applicable for your security setup" warning from > initialize > > > Key: ACCUMULO-2907 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2907 > Project: Accumulo > Issue Type: Improvement >Reporter: Josh Elser >Assignee: Mark Owens >Priority: Minor > Labels: newbie > > The warning that is printed about setting a root password may not be > applicable for your security setup is invalid because the plugable > authentication modules do not manage the root user and it is expected that > there will always be a local root user with a password. > Remove the warning. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] ctubbsii commented on a change in pull request #300: ACCUMULO-4708 Limit RFile block size to 2GB
ctubbsii commented on a change in pull request #300: ACCUMULO-4708 Limit RFile
block size to 2GB
URL: https://github.com/apache/accumulo/pull/300#discussion_r140381442
##
File path:
core/src/test/java/org/apache/accumulo/core/conf/DefaultConfigurationTest.java
##
@@ -51,4 +51,11 @@ public void testGetProperties() {
public void testSanityCheck() {
ConfigSanityCheck.validate(c);
}
+
+ @Test
+ public void testDefaultBlockSizeIsAllowed() {
+long bsi = c.getAsBytes(Property.TABLE_FILE_COMPRESSED_BLOCK_SIZE_INDEX);
+assert (bsi > 0);
+assert (bsi < Integer.MAX_VALUE);
Review comment:
Probably best to avoid assert keyword statements, because assertions can be
disabled in the JVM. Best to use JUnit provided APIs for assertions:
`Assert.assertTrue( expr )`
Also, might be good to add a small one-line comment explaining what we're
checking for. As I understand it, this is mostly just a sanity check on our
default.
We can check more generally that any value in the config is suitable by
putting this check in:
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/conf/ConfigSanityCheck.java#L46
in the `validate()` method. This gets executed whenever we read various
configurations. We could add a check to ensure that whatever the user has
configured, whichever configuration they've configured it in, that it has a
reasonable value. This is very similar to how we're already checking that the
INSTANCE_ZK_TIMEOUT value is within an allowable range.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ctubbsii commented on a change in pull request #300: ACCUMULO-4708 Limit RFile block size to 2GB
ctubbsii commented on a change in pull request #300: ACCUMULO-4708 Limit RFile
block size to 2GB
URL: https://github.com/apache/accumulo/pull/300#discussion_r140379315
##
File path:
core/src/main/java/org/apache/accumulo/core/file/rfile/bcfile/BCFile.java
##
@@ -287,8 +287,13 @@ public void finish() throws IOException {
*/
public long getRawSize() throws IOException {
/**
- * Expecting the size() of a block not exceeding 4GB. Assuming the
size() will wrap to negative integer if it exceeds 2GB.
+ * size() comes from DataOutputStream, which returns Integer.MAX_VALUE
on an overflow, which means we do not know if 2GB or more data has been written.
+ * Because the data is of an unknown length, we cannot know the block
size. To avoid corrupt RFiles, we throw an exception. This should be addressed
by
+ * whatever object is putting data into the stream to ensure this
condition is never reached.
*/
+if (size() == Integer.MAX_VALUE) {
Review comment:
Just to be safe, could also do `>=` in case implementation changes. We know
it should never be larger than max int, but if implementation starts returning
some larger value (for whatever reason), we don't want to let that through
either.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (ACCUMULO-4707) Simple maven profile that skips plugins
[
https://issues.apache.org/jira/browse/ACCUMULO-4707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175565#comment-16175565
]
Christopher Tubbs commented on ACCUMULO-4707:
-
{{-Dcheckstyle.skip}} works fine for me. The version of the current formatter
plugin does not have a skip feature, but if we update to the latest, it should
have a {{-Dformatter.skip}}. Also, there already exists a profile to skip all
the formatting plugins as a group: {{-DskipFormat}} or {{-P !autoformat}}.
There's even an inline comment in the pom.xml where the formatter is executed
to explain this.
You can also create different build profiles in your {{~/.m2/settings.xml}}
file, and activate them on the command-line, or set default activations in that
same file. You can also create a shell alias or simple bash function or simple
wrapper script if you want to have conveniently-named command-line argument
sets.
There's generally not a need to create a new profile in the POM for such
things, and I'm generally against it (for several reasons, some of which I've
articulated on ACCUMULO-4423 and on the Fluo issue linked above).
> Simple maven profile that skips plugins
> ---
>
> Key: ACCUMULO-4707
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4707
> Project: Accumulo
> Issue Type: Improvement
>Reporter: Michael Miller
>Priority: Minor
>
> I am not sure if this is a bug or how we have the build setup but I can't
> skip checkstyle. Passing in -Dcheckstyle.skip or -Dcheckstyle.skip=true has
> no effect on whatever maven command I run. Maybe this was intentional? I
> was trying to come up with a way to just build code as quickly as possible
> without running any maven plugins.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ACCUMULO-4706) Create official Accumulo Docker image
[
https://issues.apache.org/jira/browse/ACCUMULO-4706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175507#comment-16175507
]
Christopher Tubbs commented on ACCUMULO-4706:
-
In the past, we had RPMs and DEBs built upstream as "official" packaging of
Accumulo. We ended up abandoning those "official" RPMs and DEBs, and stripped
out downstream packaging elements, in favor of delegating that maintenance to
the proper downstream communities (if they desired to pick it up).
There is an analogy here with Docker. I see Docker as another downstream
packaging effort. That said, I think there are some important differences with
Docker that differentiate it from the RPM/DEB packaging that we did before.
* First, it's clear that the way this would happen is as a separate, sibling
module from the main upstream Accumulo build. This is different from what we
did before, by baking in the RPM/DEB builds into our main artifact builds.
Keeping this as a separate git repo, which depends on the main build is a great
idea.
* Second, unlike RPM/DEB, which are tightly coupled to the downstream packaging
conventions of FedoraDocker is somewhat independent of any particular
downstream environment. It makes sense that if a community of people want to
work on that (even if that is just one person), that we could host that effort,
since there's no other logical place for it to exist.
Overall, I'm in favor of this effort, and support it as long as there is
somebody interested in doing the work.
As for the idea of uploading config to ZK during init, I think that's sensible.
I think we could hijack the existing "SystemConfiguration" mechanism to push
configuration there. We'll need a bootstrap mechanism to get these services
connected to ZK in the first place, though. For that, I think we should
seriously consider revamping our configuration mechanisms, to use
commons-configuration2's CompositeConfiguration with the composition being:
accumulo-site.properties file, overridden by java system properties, which we
can set on the command-line. Then, we can simply inject the command line args
for the system properties which provide ZK connection details into the workers.
I've wanted this change (or something like it) for a while, and I think it
would easily help with this case.
> Create official Accumulo Docker image
> -
>
> Key: ACCUMULO-4706
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4706
> Project: Accumulo
> Issue Type: New Feature
> Components: scripts, start
>Affects Versions: 2.0.0
>Reporter: Mike Walch
>Assignee: Mike Walch
>
> While there are some [Accumulo
> images|https://hub.docker.com/search/?isAutomated=0=0=1=0=accumulo=0]
> on DockerHub, it looks the majority of the them are designed to run a
> single-node Accumulo instance in a Docker container for development and
> testing.
> It would be great if Accumulo had an official image for running Accumulo
> processes in containers on a production cluster. The image could be be
> published as an official image 'apache/accumulo' to DockerHub.
> In order to make this possible, I think work needs to be done to allow
> configuration to be passed to the Accumulo process in the docker container
> without using configuration files (as passing files to a running container is
> hard in Docker). One way to do this is to add an option called
> {{--upload-accumulo-site}} to 'accumulo init' command which is called outside
> of Docker by the user. This would set properties in accumulo-site.xml as
> system properties in Zookeeper during Accumulo initialization. Accumulo
> processes in Docker containers could be started with minimal configuration by
> updating 'accumulo ' commands to have a {{-o key=value}} option to
> override configuration. These changes to Accumulo would enable the following
> commands to start an Accumulo cluster in Docker:
> {noformat}
> accumulo init --upload-accumulo-site
> docker pull apache/accumulo
> docker run apache/accumulo master -o instance.zookeeper.host=zkhost:2181
> docker run apache/accumulo tserver -o instance.zookeeper.host=zkhost:2181
> docker run apache/accumulo monitor -o instance.zookeeper.host=zkhost:2181
> docker run apache/accumulo tracer -o instance.zookeeper.host=zkhost:2181
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ACCUMULO-4703) Attempt to pull all dependencies to latest version
[ https://issues.apache.org/jira/browse/ACCUMULO-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174913#comment-16174913 ] Christopher Tubbs commented on ACCUMULO-4703: - Yeah, if we're updating to newer plugins which require 3.2.5 or later, then it makes more sense to bump it. > Attempt to pull all dependencies to latest version > -- > > Key: ACCUMULO-4703 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4703 > Project: Accumulo > Issue Type: Task >Reporter: Keith Turner >Assignee: Michael Miller >Priority: Blocker > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > This is issue is motivated by discussion in ACCUMULO-4701. For 2.0.0 we > should attempt to use the latest version of any direct dependencies. Not > doing so may force user to use older versions of dependencies with bugs and > security problems. > ACCUMULO-4701 provides an example of this where Accumulo using methods that > exist in an older version of Guava but are dropped in a new version prevent a > user from using newer Guava. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ACCUMULO-4703) Attempt to pull all dependencies to latest version
[ https://issues.apache.org/jira/browse/ACCUMULO-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174870#comment-16174870 ] Michael Miller commented on ACCUMULO-4703: -- I thought there might be a reason for keeping it at that version but the comment mentioned maven release plugin. I went with the version required by versions-maven-plugin. I thought 3.2.5 was a good compromise from going all the way to the latest, 3.5.0. Thankfully it looks like 3.0.5 is the oldest recommended version: https://maven.apache.org/security.html I can revert it and add a comment. > Attempt to pull all dependencies to latest version > -- > > Key: ACCUMULO-4703 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4703 > Project: Accumulo > Issue Type: Task >Reporter: Keith Turner >Assignee: Michael Miller >Priority: Blocker > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > This is issue is motivated by discussion in ACCUMULO-4701. For 2.0.0 we > should attempt to use the latest version of any direct dependencies. Not > doing so may force user to use older versions of dependencies with bugs and > security problems. > ACCUMULO-4701 provides an example of this where Accumulo using methods that > exist in an older version of Guava but are dropped in a new version prevent a > user from using newer Guava. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ACCUMULO-4640) Accumulo shell is expecting instance.volumes in client.conf instead of accumulo-site.xml
[ https://issues.apache.org/jira/browse/ACCUMULO-4640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174339#comment-16174339 ] Jorge Machado commented on ACCUMULO-4640: - [~luigidifraia] We end up changing the hdfs path, because that change does not seem clear to me. Regards and I hope that on 1.8.2 it gets fixed. Regards > Accumulo shell is expecting instance.volumes in client.conf instead of > accumulo-site.xml > > > Key: ACCUMULO-4640 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4640 > Project: Accumulo > Issue Type: Bug > Components: shell >Affects Versions: 1.8.1 > Environment: [accumulo@master ~]$ cat /etc/redhat-release > CentOS Linux release 7.3.1611 (Core) > [accumulo@master ~]$ uname -a > Linux master 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > [accumulo@master ~]$ java -version > java version "1.8.0_112" > Java(TM) SE Runtime Environment (build 1.8.0_112-b15) > Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode) > Hadoop: 2.8.0 > ZooKeeper: 3.4.10 > Accumulo: 1.8.1 >Reporter: Luigi Di Fraia >Assignee: Christopher Tubbs > Fix For: 1.7.4, 1.8.2, 2.0.0 > > Attachments: accumulo-env.sh, accumulo-site.xml > > Time Spent: 1h > Remaining Estimate: 0h > > Starting with Accumulo 1.8.1 we are getting the exception as per below when > the Accumulo shell is invoked: > [accumulo@master ~]$ /usr/local/accumulo/bin/accumulo shell -u root -p > ***removed*** > 2017-05-18 14:31:23,667 [zookeeper.ZooUtil] ERROR: unable obtain instance id > at hdfs://master:9000/accumulo/instance_id > 2017-05-18 14:31:23,669 [start.Main] ERROR: Thread 'shell' died. > java.lang.RuntimeException: Accumulo not initialized, there is no instance id > at hdfs://master:9000/accumulo/instance_id > at > org.apache.accumulo.core.zookeeper.ZooUtil.getInstanceIDFromHdfs(ZooUtil.java:66) > at > org.apache.accumulo.core.zookeeper.ZooUtil.getInstanceIDFromHdfs(ZooUtil.java:51) > at org.apache.accumulo.shell.Shell.getZooInstance(Shell.java:511) > at org.apache.accumulo.shell.Shell.setInstance(Shell.java:473) > at org.apache.accumulo.shell.Shell.config(Shell.java:322) > at org.apache.accumulo.shell.Shell.execute(Shell.java:589) > at org.apache.accumulo.start.Main$1.run(Main.java:120) > at java.lang.Thread.run(Thread.java:745) > The location used to look up the instance_id > (hdfs://master:9000/accumulo/instance_id) does not appear to be consistent > with this site configuration and I am not quite sure how it is being worked > out. > The Accumulo Overview web page shows that the instance exists: > Instance ID: 4264f11a-704f-4c1f-880f-08f6154232b2 > Thu May 18 14:44:03 BST 2017 > The instance_id file is in Hadoop's HDFS: > [accumulo@master ~]$ /usr/local/hadoop/bin/hdfs dfs -ls > /user/accumulo/accumulo/instance_id > Found 1 items > -rw-r--r-- 3 accumulo supergroup 0 2017-05-18 14:27 > /user/accumulo/accumulo/instance_id/4264f11a-704f-4c1f-880f-08f6154232b2 > Again, within the same environment as selected above apart for using Accumulo > 1.8.0, the shell can be invoked without issues. > Here's a snippet from accumulo-site.xml: > > instance.volumes > hdfs://master:9000/user/accumulo/accumulo > comma separated list of URIs for volumes. example: > hdfs://localhost:9000/accumulo > > For historical reasons our "root volume" is not / but /user/accumulo although > the above stack trace suggests that / is being used as root volume. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
