[GitHub] ctubbsii opened a new issue #422: Token file functionality for mapred code deprecated, but not replaced
ctubbsii opened a new issue #422: Token file functionality for mapred code deprecated, but not replaced URL: https://github.com/apache/accumulo/issues/422 As fallout from the recent Connector builder stuff that @mikewalch worked on for 2.0, many of the old APIs were deprecated in the MapReduce code to be replaced with `ConnectionInfo`. However, the mapreduce APIs for storing an authentication token in a job's distributed cache, and reading it from within the mappers does not have an equivalent in the new code. What is needed is two things: 1. The ability to serialize the entire connection info into the job, without exposing the credentials in the Hadoop configuration, and 2. The ability to serialize arbitrary authentication tokens, not just the known ones that the current Connector builder has special code to handle. (A whole authentication token serializer was previously created for this purpose, but it is not used in the new code.) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] milleruntime closed pull request #21: Add debug flag to runex Fixes #18
milleruntime closed pull request #21: Add debug flag to runex Fixes #18
URL: https://github.com/apache/accumulo-examples/pull/21
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/bin/runex b/bin/runex
index fc273ad..8bf7479 100755
--- a/bin/runex
+++ b/bin/runex
@@ -15,6 +15,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+# For debugging run this script like so:
+# RUNEX_MVN_ARG="-X" ./bin/runex client.RandomBatchScanner
+mvn_arg="${RUNEX_MVN_ARG:--q}"
+
main_class="$1"
main_args="${*:2}"
@@ -22,4 +26,4 @@ if command -v accumulo > /dev/null 2>&1 ; then
av_arg="-Daccumulo.version=`accumulo version | tail -n 1`"
fi
-mvn -q exec:java -Dlog4j.configuration="file:./conf/log4j.properties"
-Dexec.mainClass="org.apache.accumulo.examples.$main_class" $av_arg
-Dexec.args="$main_args"
+mvn $mvn_arg exec:java -Dlog4j.configuration="file:./conf/log4j.properties"
-Dexec.mainClass="org.apache.accumulo.examples.$main_class" $av_arg
-Dexec.args="$main_args"
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] milleruntime closed issue #18: Add debug flag to runex
milleruntime closed issue #18: Add debug flag to runex URL: https://github.com/apache/accumulo-examples/issues/18 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmark99 closed pull request #420: ACCUMULO-4697 tablet server could kick off too many idle compactions
jmark99 closed pull request #420: ACCUMULO-4697 tablet server could kick off
too many idle compactions
URL: https://github.com/apache/accumulo/pull/420
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
b/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
index 52aaf055dc..99262955fd 100644
--- a/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
+++ b/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
@@ -2140,8 +2140,6 @@ public void run() {
// modification
}
- int numMajorCompactionsInProgress = 0;
-
Iterator> iter =
copyOnlineTablets.entrySet().iterator();
// bail early now if we're shutting down
@@ -2168,29 +2166,7 @@ public void run() {
}
synchronized (tablet) {
- if (tablet.initiateMajorCompaction(MajorCompactionReason.NORMAL)
- || tablet.isMajorCompactionQueued() ||
tablet.isMajorCompactionRunning()) {
-numMajorCompactionsInProgress++;
-continue;
- }
-}
- }
-
- int idleCompactionsToStart = Math.max(1,
- getConfiguration().getCount(Property.TSERV_MAJC_MAXCONCURRENT) /
2);
-
- if (numMajorCompactionsInProgress < idleCompactionsToStart) {
-// system is not major compacting, can schedule some
-// idle compactions
-iter = copyOnlineTablets.entrySet().iterator();
-
-while (iter.hasNext() && numMajorCompactionsInProgress <
idleCompactionsToStart) {
- Entry entry = iter.next();
- Tablet tablet = entry.getValue();
-
- if (tablet.initiateMajorCompaction(MajorCompactionReason.IDLE)) {
-numMajorCompactionsInProgress++;
- }
+ tablet.initiateMajorCompaction(MajorCompactionReason.NORMAL);
}
}
} catch (Throwable t) {
diff --git
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
index b3139e77c1..14836be115 100644
---
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
+++
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
@@ -1531,18 +1531,16 @@ public Path getLocation() {
return location;
}
- public synchronized boolean initiateMajorCompaction(MajorCompactionReason
reason) {
+ public synchronized void initiateMajorCompaction(MajorCompactionReason
reason) {
if (isClosing() || isClosed() || !needsMajorCompaction(reason) ||
isMajorCompactionRunning()
|| majorCompactionQueued.contains(reason)) {
- return false;
+ return;
}
majorCompactionQueued.add(reason);
getTabletResources().executeMajorCompaction(getExtent(), new
CompactionRunner(this, reason));
-
-return false;
}
/**
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] jmark99 commented on issue #420: ACCUMULO-4697 tablet server could kick off too many idle compactions
jmark99 commented on issue #420: ACCUMULO-4697 tablet server could kick off too many idle compactions URL: https://github.com/apache/accumulo/pull/420#issuecomment-380530899 I'll leave it as is unless someone else has other thoughts. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ctubbsii commented on issue #420: ACCUMULO-4697 tablet server could kick off too many idle compactions
ctubbsii commented on issue #420: ACCUMULO-4697 tablet server could kick off too many idle compactions URL: https://github.com/apache/accumulo/pull/420#issuecomment-380527177 It's up to you. Idle compactions never took into account other tserver activity, like scans, so they were probably never a good idea... but I defer to you, or another reviewer's judgment... I'm mostly indifferent. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmark99 commented on issue #420: ACCUMULO-4697 tablet server could kick off too many idle compactions
jmark99 commented on issue #420: ACCUMULO-4697 tablet server could kick off too many idle compactions URL: https://github.com/apache/accumulo/pull/420#issuecomment-380526195 I thought that would have been a possible fix as well, but ended up going Adam's suggestion. If you feel it is worth it for possible future needs I can refactor the method to once again return boolean but with true after the queuing. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] milleruntime opened a new issue #421: Organize Fate Operations
milleruntime opened a new issue #421: Organize Fate Operations URL: https://github.com/apache/accumulo/issues/421 The RepOs of every fate operation are ALL in the org.apache.accumulo.master.tableOps package. It is annoying to track down which RepO goes with which operation. Creating a package in o.a.a.master.tableOps for each operation would organize them nicely. This would have to be done in 2.0 since it will break serialization. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (ACCUMULO-4697) tablet server could kick off too many idle compactions
[ https://issues.apache.org/jira/browse/ACCUMULO-4697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Owens closed ACCUMULO-4697. Duplicated by Github issue [#420|https://github.com/apache/accumulo/pull/420] > tablet server could kick off too many idle compactions > -- > > Key: ACCUMULO-4697 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4697 > Project: Accumulo > Issue Type: Bug > Components: tserver >Reporter: Adam Fuchs >Assignee: Mark Owens >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Tablet.initiateMajorCompaction() always returns false, but we check the > return value when accounting for the number of idle major compactions started > in TabletServer.MajorCompactor.run() and continue to kick off idle > compactions on every tablet. > DefaultCompactionStrategy doesn't do anything special for idle compactions, > so this will only be an issue with a custom CompactionStrategy. > My guess would be the best thing to do would be to get rid of idle > compactions and get rid of the return value on initiateMajorCompaction(). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ACCUMULO-4697) tablet server could kick off too many idle compactions
[ https://issues.apache.org/jira/browse/ACCUMULO-4697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Owens resolved ACCUMULO-4697. -- Resolution: Duplicate This ticket is now represented by Github Issue [#420|https://github.com/apache/accumulo/pull/420] > tablet server could kick off too many idle compactions > -- > > Key: ACCUMULO-4697 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4697 > Project: Accumulo > Issue Type: Bug > Components: tserver >Reporter: Adam Fuchs >Assignee: Mark Owens >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Tablet.initiateMajorCompaction() always returns false, but we check the > return value when accounting for the number of idle major compactions started > in TabletServer.MajorCompactor.run() and continue to kick off idle > compactions on every tablet. > DefaultCompactionStrategy doesn't do anything special for idle compactions, > so this will only be an issue with a custom CompactionStrategy. > My guess would be the best thing to do would be to get rid of idle > compactions and get rid of the return value on initiateMajorCompaction(). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] jmark99 opened a new pull request #420: ACCUMULO-4697 tablet server could kick off too many idle compactions
jmark99 opened a new pull request #420: ACCUMULO-4697 tablet server could kick off too many idle compactions URL: https://github.com/apache/accumulo/pull/420 ACCUMULO-4697 tablet server could kick off too many idle compactions Tablet.initiateMajorCompaction() always returns false, but we check the return value when accounting for the number of idle major compactions started in TabletServer.MajorCompactor.run() and continue to kick off idle compactions on every tablet. DefaultCompactionStrategy doesn't do anything special for idle compactions, so this will only be an issue with a custom CompactionStrategy. My guess would be the best thing to do would be to get rid of idle compactions and get rid of the return value on initiateMajorCompaction(). This update follows the reporters suggestion and removes the idle compaction code and refactors the initiateMajorCompaction() method to return void. This ticket references Jira ticket [ACCUMULO-4697](https://issues.apache.org/jira/browse/ACCUMULO-4697) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] milleruntime commented on a change in pull request #21: Add debug flag to runex Fixes #18
milleruntime commented on a change in pull request #21: Add debug flag to runex
Fixes #18
URL: https://github.com/apache/accumulo-examples/pull/21#discussion_r180805567
##
File path: bin/runex
##
@@ -16,10 +16,24 @@
# limitations under the License.
main_class="$1"
-main_args="${*:2}"
+mvn_opts="-q"
+main_args=()
+
+shift
+while [[ $# -gt 0 ]] ; do
+ case $1 in
+-d|--debug)
+mvn_opts="-X"
+;;
+*)
+main_args+=("$1")
+;;
+ esac
+ shift
+done
if command -v accumulo > /dev/null 2>&1 ; then
av_arg="-Daccumulo.version=`accumulo version | tail -n 1`"
fi
-mvn -q exec:java -Dlog4j.configuration="file:./conf/log4j.properties"
-Dexec.mainClass="org.apache.accumulo.examples.$main_class" $av_arg
-Dexec.args="$main_args"
+mvn $mvn_opts exec:java -Dlog4j.configuration="file:./conf/log4j.properties"
-Dexec.mainClass="org.apache.accumulo.examples.$main_class" $av_arg
-Dexec.args="${main_args[@]}"
Review comment:
It took me a second to wrap my head around this one but thank you for the
example and explanation. I wouldn't need to worry about constructing an array
if I use your idea of the env variable below.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] milleruntime commented on a change in pull request #21: Add debug flag to runex Fixes #18
milleruntime commented on a change in pull request #21: Add debug flag to runex
Fixes #18
URL: https://github.com/apache/accumulo-examples/pull/21#discussion_r180804911
##
File path: bin/runex
##
@@ -16,10 +16,24 @@
# limitations under the License.
main_class="$1"
-main_args="${*:2}"
+mvn_opts="-q"
+main_args=()
+
+shift
+while [[ $# -gt 0 ]] ; do
+ case $1 in
+-d|--debug)
Review comment:
I like using an environment variable, I think that's the best solution.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
