[GitHub] phoenix pull request: merging latest avatica
Github user codymarcel commented on the pull request: https://github.com/apache/phoenix/pull/157#issuecomment-209524084 @gkanade Any updates on this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] phoenix pull request: merging latest avatica
Github user codymarcel commented on the pull request: https://github.com/apache/phoenix/pull/157#issuecomment-205858884 General Comments. 1. Please add some unit tests. Especially around the instanceResolver and RulesApplier. We user https://github.com/apache/phoenix/blob/master/phoenix-pherf/src/test/resources/scenario/test_scenario.xml as both test base and doc for supported features. 2. Formatting needs some work across the board. Make sure you have the formatter for the project applied and your ide is set to convert tabs to spaces. 3. Remove all code that's in comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] phoenix pull request: merging latest avatica
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/157#discussion_r58557911
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/workload/WriteWorkload.java
---
@@ -253,13 +255,22 @@ private void waitForBatches(DataLoadTimeSummary
dataLoadTimeSummary, Scenario sc
last = start = System.currentTimeMillis();
String sql = buildSql(columns, tableName);
+//System.out.print("going to create statement");
--- End diff --
remove commented out code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: merging latest avatica
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/157#discussion_r58555009
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/QuerySet.java
---
@@ -31,7 +31,24 @@
private long numberOfExecutions =
PherfConstants.DEFAULT_NUMBER_OF_EXECUTIONS;
private long executionDurationInMs =
PherfConstants.DEFAULT_THREAD_DURATION_IN_MS;
private ExecutionType executionType = ExecutionType.SERIAL;
-
+ private boolean randomPointRead = false;
+ private String primaryKey = PherfConstants.DEFAULT_PRIMARY_KEY;
+
+ @XmlAttribute
+ public String getPrimaryKey() {
+ return primaryKey;
--- End diff --
nit: more formatting. Make sure you import the code stye preferences into
your project.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: merging latest avatica
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/157#discussion_r58554617
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/DataTypeMapping.java
---
@@ -25,7 +25,8 @@
CHAR("CHAR", Types.CHAR),
DECIMAL("DECIMAL", Types.DECIMAL),
INTEGER("INTEGER", Types.INTEGER),
-DATE("DATE", Types.DATE);
+DATE("DATE", Types.DATE),
+ YCSBKEY("YCSBKEY", Types.VARCHAR);
--- End diff --
nit: formatting
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: merging latest avatica
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/157#discussion_r58554472
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java
---
@@ -254,6 +254,7 @@ public void run() throws Exception {
// Schema and Data Load
if (preLoadData) {
logger.info("\nStarting Data Load...");
+System.out.print("Starting Data Load ...");
--- End diff --
Should be able to use the logger config for stdout instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: merging latest avatica
Github user codymarcel commented on the pull request:
https://github.com/apache/phoenix/pull/157#issuecomment-205847652
Yea, I'll take a look. What is the associated Jira?
On Mon, Apr 4, 2016 at 8:02 PM, James Taylor <notificati...@github.com>
wrote:
> @codymarcel <https://github.com/codymarcel> - would it be possible for
> you to review this pull as Mujtaba is on PTO?
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/phoenix/pull/157#issuecomment-205598163>
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38470844
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java
---
@@ -179,6 +188,15 @@ public void run() throws Exception {
}
return;
}
+
+// Compare results and exit
+ if (null != compareResults) {
--- End diff --
if(!compareResults){}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38474158
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/util/ResultsComparator.java
---
@@ -0,0 +1,353 @@
+package org.apache.phoenix.pherf.util;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.PrintWriter;
+import java.io.UnsupportedEncodingException;
+import java.text.DecimalFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVParser;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+
+/**
+ * Compare results based on set threshold and render results as Google
Charts
+ */
+public class ResultsComparator {
+
+ private String[] labels;
+ private final Map<String, DataNode> datanodes = new TreeMap<String,
DataNode>();
+ private final PherfConstants constants = PherfConstants.create();
+ private final String resultDir =
constants.getProperty("pherf.default.results.dir");
+ private final double threshold =
Double.parseDouble(constants.getProperty("pherf.default.comparison.threshold"));
+
+ public ResultsComparator(String labels) {
+ this.setLabelsAsString(labels);
+ }
+
+ String[] getLabels() {
+ return labels;
+ }
+
+ void setLabels(String[] labels) {
+ this.labels = labels;
+ }
+
+ void setLabelsAsString(String labels) {
+ this.labels = labels.split(",");
+ }
+
+ public void readAndRender() {
+ try {
+ for (String label : labels) {
+ read(label);
+ }
+ renderAsGoogleChartsHTML();
+
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+
+ /**
+* Reads aggregate file and convert it to DataNode
+* @param label
+* @throws Exception
+*/
+private void read(String label) throws Exception {
+ String resultFileName = resultDir
+ + PherfConstants.PATH_SEPARATOR
+ + PherfConstants.RESULT_PREFIX
+ + label
+ +
ResultFileDetails.CSV_AGGREGATE_PERFORMANCE.getExtension();
+
+ FileReader in = new FileReader(resultFileName);
+ final CSVParser parser = new CSVParser(in,
CSVFormat.DEFAULT.withHeader());
+
+for (CSVRecord record : parser) {
+String group = record.get("QUERY_GROUP");
+String query = record.get("QUERY");
+String explain = record.get("EXPLAIN_PLAN");
+String tenantId = record.get("TENANT_ID");
+long avgTime = Long.parseLong(record.get("AVG_TIME_MS"));
+long minTime = Long.parseLong(record.get("AVG_MIN_TIME_MS"));
+long numRuns = Long.parseLong(record.get("RUN_COUNT"));
+long rowCount = Long.parseLong(record.get("RESULT_ROW_COUNT"));
+Node node = new Node(minTime, avgTime, numRuns, explain,
query, tenantId, label, rowCount);
+
+if (datanodes.containsKey(group)) {
+ datanodes.get(group).getDataSet().put(label, node);
+} else {
+ datanodes.put(group, new DataNode(label, node));
+}
+}
+parser.close();
+}
+
+/**
+ * Verifies if the first result is within the set
+ * threshold of pherf.default.comparison.threshold
+ * set in pherf.properties files
+ * @param threshold
+ * @return
+ */
+private boolean verifyWithinThreshold(double threshold) {
+ long resetTimeToCompare = -1;
+ long timeToCompare = resetTimeToCompare;
+ for (Map.Entry<String, DataNode> dn : datanodes.entrySet()) {
+ for (Map.Entry<String, Node> node :
dn.getValue().getDataSet().entrySet()) {
+ if (timeToCompare == -1) {
+ timeToCompare = node.getValue().getMinTime();
+ }
+ if double) (timeToCompare -
node.getValue().getMinTime())) / (double) node
+
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38476254
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/util/ResultsComparator.java
---
@@ -0,0 +1,353 @@
+package org.apache.phoenix.pherf.util;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.PrintWriter;
+import java.io.UnsupportedEncodingException;
+import java.text.DecimalFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVParser;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+
+/**
+ * Compare results based on set threshold and render results as Google
Charts
+ */
+public class ResultsComparator {
+
+ private String[] labels;
+ private final Map<String, DataNode> datanodes = new TreeMap<String,
DataNode>();
+ private final PherfConstants constants = PherfConstants.create();
+ private final String resultDir =
constants.getProperty("pherf.default.results.dir");
+ private final double threshold =
Double.parseDouble(constants.getProperty("pherf.default.comparison.threshold"));
+
+ public ResultsComparator(String labels) {
+ this.setLabelsAsString(labels);
+ }
+
+ String[] getLabels() {
+ return labels;
+ }
+
+ void setLabels(String[] labels) {
+ this.labels = labels;
+ }
+
+ void setLabelsAsString(String labels) {
+ this.labels = labels.split(",");
+ }
+
+ public void readAndRender() {
+ try {
+ for (String label : labels) {
+ read(label);
+ }
+ renderAsGoogleChartsHTML();
+
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+
+ /**
+* Reads aggregate file and convert it to DataNode
+* @param label
+* @throws Exception
+*/
+private void read(String label) throws Exception {
--- End diff --
Can we use a CSVResultHandler with a
ResultFileDetails.CSV_AGGREGATE_PERFORMANCE? It looks like there is a fair
amount of redundancy with that and the ResultWriter. It makes sense that you
need to wrap the compare logic into this, but result reading and writing should
use the existing framework for that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38473654
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/util/ResultsComparator.java
---
@@ -0,0 +1,353 @@
+package org.apache.phoenix.pherf.util;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.PrintWriter;
+import java.io.UnsupportedEncodingException;
+import java.text.DecimalFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVParser;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+
+/**
+ * Compare results based on set threshold and render results as Google
Charts
+ */
+public class ResultsComparator {
+
+ private String[] labels;
+ private final Map<String, DataNode> datanodes = new TreeMap<String,
DataNode>();
+ private final PherfConstants constants = PherfConstants.create();
+ private final String resultDir =
constants.getProperty("pherf.default.results.dir");
+ private final double threshold =
Double.parseDouble(constants.getProperty("pherf.default.comparison.threshold"));
+
+ public ResultsComparator(String labels) {
+ this.setLabelsAsString(labels);
+ }
+
+ String[] getLabels() {
+ return labels;
+ }
+
+ void setLabels(String[] labels) {
+ this.labels = labels;
+ }
+
+ void setLabelsAsString(String labels) {
--- End diff --
This is setting as array. Why not just call it setLabels() and let
overloading do it's thing?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request: https://github.com/apache/phoenix/pull/115#discussion_r38472004 --- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/PherfConstants.java --- @@ -62,6 +62,8 @@ public static final int MONITOR_FREQUENCY = 5000; public static final String MONITOR_FILE_NAME = "STATS_MONITOR"; +public static final String SUMMARY_HTML_FILE_NAME = "summary.html"; --- End diff -- I am trying to make everything here configurable from properties. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38471394
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java
---
@@ -136,6 +142,8 @@ public Pherf(String[] args) throws Exception {
command.getOptionValue("writerThreadSize",
properties.getProperty("pherf.default.dataloader.threadpool"));
properties.setProperty("pherf. default.dataloader.threadpool",
writerThreadPoolSize);
+label = command.getOptionValue("label", null);
+compareResults = command.getOptionValue("compare", null);
--- End diff --
Use boolean and set to false as default
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38472365
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/result/ResultUtil.java ---
@@ -147,9 +147,9 @@ public String getSuffix() {
if (null == FILE_SUFFIX) {
Date date = new Date();
Format formatter = new SimpleDateFormat("-MM-dd_hh-mm-ss");
-FILE_SUFFIX = "_" + formatter.format(date);
+FILE_SUFFIX = formatter.format(date);
--- End diff --
Should you call setFileSuffix()?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38473156
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/util/ResultsComparator.java
---
@@ -0,0 +1,353 @@
+package org.apache.phoenix.pherf.util;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.PrintWriter;
+import java.io.UnsupportedEncodingException;
+import java.text.DecimalFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVParser;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+
+/**
+ * Compare results based on set threshold and render results as Google
Charts
+ */
+public class ResultsComparator {
+
+ private String[] labels;
+ private final Map<String, DataNode> datanodes = new TreeMap<String,
DataNode>();
+ private final PherfConstants constants = PherfConstants.create();
+ private final String resultDir =
constants.getProperty("pherf.default.results.dir");
+ private final double threshold =
Double.parseDouble(constants.getProperty("pherf.default.comparison.threshold"));
+
+ public ResultsComparator(String labels) {
--- End diff --
nit: label should be singular ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38470727
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java
---
@@ -76,6 +78,8 @@
options.addOption("d", "debug", false, "Put tool in debug mode");
options.addOption("stats", false,
"Update Phoenix Statistics after data is loaded with -l
argument");
+ options.addOption("label", true, "Label a run. Result file name
will be suffixed with specified label");
--- End diff --
Is this an override to then dynamically generated label? Also, it would be
good if the label was applied as a column value in the result data. That way we
can distinguish between runs on analysis.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38470777
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java
---
@@ -136,6 +142,8 @@ public Pherf(String[] args) throws Exception {
command.getOptionValue("writerThreadSize",
properties.getProperty("pherf.default.dataloader.threadpool"));
properties.setProperty("pherf. default.dataloader.threadpool",
writerThreadPoolSize);
+label = command.getOptionValue("label", null);
+compareResults = command.getOptionValue("compare", null);
--- End diff --
default to false instead of null
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request: https://github.com/apache/phoenix/pull/115#discussion_r38471909 --- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/PherfConstants.java --- @@ -62,6 +62,8 @@ public static final int MONITOR_FREQUENCY = 5000; public static final String MONITOR_FILE_NAME = "STATS_MONITOR"; +public static final String SUMMARY_HTML_FILE_NAME = "summary.html"; --- End diff -- Can you put this in the properties file? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38473049
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/util/ResultsComparator.java
---
@@ -0,0 +1,353 @@
+package org.apache.phoenix.pherf.util;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.PrintWriter;
+import java.io.UnsupportedEncodingException;
+import java.text.DecimalFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVParser;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+
+/**
+ * Compare results based on set threshold and render results as Google
Charts
+ */
+public class ResultsComparator {
--- End diff --
Should this implement Comparable? Calling it Comparator is misleading if
it's not doing that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38477433
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/workload/MultithreadedDiffer.java
---
@@ -51,7 +53,7 @@ private void diffQuery() throws Exception {
Date startDate = Calendar.getInstance().getTime();
String newCSV = queryVerifier.exportCSV(query);
boolean verifyResult = queryVerifier.doDiff(query, newCSV);
-String explainPlan = queryVerifier.getExplainPlan(query);
--- End diff --
Could you add the explainPlan stuff as a different Jira/CL? It might get
checked in faster if it's detached from this CL. This one might need a little
more love.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38471489
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java
---
@@ -179,6 +188,15 @@ public void run() throws Exception {
}
return;
}
+
+// Compare results and exit
+ if (null != compareResults) {
+ logger.info("\nStarting to compare results and
exiting for " + compareResults);
+ ResultsComparator rc = new
ResultsComparator(compareResults);
--- End diff --
Should this be the label?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38471433
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java
---
@@ -179,6 +188,15 @@ public void run() throws Exception {
}
return;
}
+
+// Compare results and exit
+ if (null != compareResults) {
--- End diff --
if(!compareResults){}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38472866
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/util/PhoenixUtil.java ---
@@ -274,4 +274,34 @@ public void updatePhoenixStats(String tableName,
Scenario scenario) throws Excep
logger.info("Updating stats for " + tableName);
executeStatement("UPDATE STATISTICS " + tableName, scenario);
}
+
+/**
+ * Get explain plan for a query
+ *
+ * @param query
+ * @return
+ * @throws SQLException
+ */
+public String getExplainPlan(Query query) throws SQLException {
+Connection conn = null;
+ResultSet rs = null;
+PreparedStatement statement = null;
+StringBuilder buf = new StringBuilder();
+try {
--- End diff --
You might want to consider using the JdbcSession() to handle all the boiler
plate try/catch/finally stuff on the connection.
Check out testRWWorkload()
https://github.com/apache/phoenix/blob/master/phoenix-pherf/src/it/java/org/apache/phoenix/pherf/DataIngestIT.java
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38477085
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/util/ResultsComparator.java
---
@@ -0,0 +1,353 @@
+package org.apache.phoenix.pherf.util;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.PrintWriter;
+import java.io.UnsupportedEncodingException;
+import java.text.DecimalFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVParser;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+
+/**
+ * Compare results based on set threshold and render results as Google
Charts
+ */
+public class ResultsComparator {
+
+ private String[] labels;
+ private final Map<String, DataNode> datanodes = new TreeMap<String,
DataNode>();
+ private final PherfConstants constants = PherfConstants.create();
+ private final String resultDir =
constants.getProperty("pherf.default.results.dir");
+ private final double threshold =
Double.parseDouble(constants.getProperty("pherf.default.comparison.threshold"));
+
+ public ResultsComparator(String labels) {
+ this.setLabelsAsString(labels);
+ }
+
+ String[] getLabels() {
+ return labels;
+ }
+
+ void setLabels(String[] labels) {
+ this.labels = labels;
+ }
+
+ void setLabelsAsString(String labels) {
+ this.labels = labels.split(",");
+ }
+
+ public void readAndRender() {
+ try {
+ for (String label : labels) {
+ read(label);
+ }
+ renderAsGoogleChartsHTML();
+
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+
+ /**
+* Reads aggregate file and convert it to DataNode
+* @param label
+* @throws Exception
+*/
+private void read(String label) throws Exception {
+ String resultFileName = resultDir
+ + PherfConstants.PATH_SEPARATOR
+ + PherfConstants.RESULT_PREFIX
+ + label
+ +
ResultFileDetails.CSV_AGGREGATE_PERFORMANCE.getExtension();
+
+ FileReader in = new FileReader(resultFileName);
+ final CSVParser parser = new CSVParser(in,
CSVFormat.DEFAULT.withHeader());
+
+for (CSVRecord record : parser) {
+String group = record.get("QUERY_GROUP");
+String query = record.get("QUERY");
+String explain = record.get("EXPLAIN_PLAN");
+String tenantId = record.get("TENANT_ID");
+long avgTime = Long.parseLong(record.get("AVG_TIME_MS"));
+long minTime = Long.parseLong(record.get("AVG_MIN_TIME_MS"));
+long numRuns = Long.parseLong(record.get("RUN_COUNT"));
+long rowCount = Long.parseLong(record.get("RESULT_ROW_COUNT"));
+Node node = new Node(minTime, avgTime, numRuns, explain,
query, tenantId, label, rowCount);
+
+if (datanodes.containsKey(group)) {
+ datanodes.get(group).getDataSet().put(label, node);
+} else {
+ datanodes.put(group, new DataNode(label, node));
+}
+}
+parser.close();
+}
+
+/**
+ * Verifies if the first result is within the set
+ * threshold of pherf.default.comparison.threshold
+ * set in pherf.properties files
+ * @param threshold
+ * @return
+ */
+private boolean verifyWithinThreshold(double threshold) {
+ long resetTimeToCompare = -1;
+ long timeToCompare = resetTimeToCompare;
+ for (Map.Entry<String, DataNode> dn : datanodes.entrySet()) {
+ for (Map.Entry<String, Node> node :
dn.getValue().getDataSet().entrySet()) {
+ if (timeToCompare == -1) {
+ timeToCompare = node.getValue().getMinTime();
+ }
+ if double) (timeToCompare -
node.getValue().getMinTime())) / (double) node
+
[GitHub] phoenix pull request: PHOENIX-1791. Adding ability for Pherf Scena...
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/105#discussion_r36532442
--- Diff:
phoenix-pherf/src/it/java/org/apache/phoenix/pherf/DataIngestIT.java ---
@@ -144,4 +148,52 @@ public void testRWWorkload() throws Exception {
fail(Failed to load data. An exception was thrown: +
e.getMessage());
}
}
+
+
+@Test
+/**
+ * Validates that Pherf can write data to a Multi-Tenant View in
addition to
+ * standard Phoenix tables.
+ */
+public void testMultiTenantViewWriteWorkload() throws Exception {
+// Arrange
+Scenario scenario =
parser.getScenarioByName(testMTWriteScenario);
+WorkloadExecutor executor = new WorkloadExecutor();
+executor.add(new WriteWorkload(util, parser, scenario));
+
+// Act
+try {
+// Wait for data to load up.
+executor.get();
+executor.shutdown();
+} catch (Exception e) {
+fail(Failed to load data. An exception was thrown: +
e.getMessage());
+}
+
+// Assert
+assertExpectedNumberOfRecordsWritten(scenario);
+
+}
+
+private void assertExpectedNumberOfRecordsWritten(Scenario scenario)
throws Exception,
+SQLException {
+Connection connection = util.getConnection(scenario.getTenantId());
+String sql = select count(*) from + scenario.getTableName();
+Integer count = new JdbcSession(connection).sql(sql).select(new
OutcomeInteger() {
+@Override public Integer handle(ResultSet resultSet, Statement
statement)
+throws SQLException {
+while (resultSet.next()) {
+return resultSet.getInt(1);
+}
+return null;
+}
+});
+assertNotNull(Could not retrieve count. + count);
+assertTrue(Could not query any rows for in +
scenario.getTableName(), count 0);
--- End diff --
This assertion is redundant with the one below. The next one is better. I
can remove this one on commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: Mixed R/W workload support
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/89#discussion_r32847155
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/workload/WorkloadExecutor.java
---
@@ -19,95 +19,92 @@
package org.apache.phoenix.pherf.workload;
import org.apache.phoenix.pherf.PherfConstants;
-import org.apache.phoenix.pherf.PherfConstants.RunMode;
-import org.apache.phoenix.pherf.configuration.XMLConfigParser;
-import org.apache.phoenix.pherf.jmx.MonitorManager;
-import org.apache.phoenix.pherf.loaddata.DataLoader;
-
-import org.apache.phoenix.pherf.util.ResourceList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.util.Properties;
+import java.util.*;
+import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
-import java.util.concurrent.TimeUnit;
public class WorkloadExecutor {
private static final Logger logger =
LoggerFactory.getLogger(WorkloadExecutor.class);
-private final XMLConfigParser parser;
-private MonitorManager monitor;
-private Future monitorThread;
private final int poolSize;
+private final MapWorkload, Future jobs = new HashMap();
private final ExecutorService pool;
-
public WorkloadExecutor() throws Exception {
this(PherfConstants.create().getProperties(PherfConstants.PHERF_PROPERTIES));
}
-public WorkloadExecutor(Properties properties) throws Exception{
-this(properties,PherfConstants.DEFAULT_FILE_PATTERN);
+public WorkloadExecutor(Properties properties) throws Exception {
+this(properties, new ArrayList());
}
-public WorkloadExecutor(Properties properties, String filePattern)
throws Exception {
-this(properties,
-new XMLConfigParser(filePattern),
-true);
+public WorkloadExecutor(Properties properties, ListWorkload
workloads) throws Exception {
+this.poolSize =
+(properties.getProperty(pherf.default.threadpool) ==
null) ?
+PherfConstants.DEFAULT_THREAD_POOL_SIZE :
+
Integer.parseInt(properties.getProperty(pherf.default.threadpool));
+
+this.pool = Executors.newFixedThreadPool(this.poolSize);
+init(workloads);
}
-public WorkloadExecutor(Properties properties, XMLConfigParser parser,
boolean monitor) throws Exception {
-this.parser = parser;
-this.poolSize =
(properties.getProperty(pherf.default.threadpool) == null)
-? PherfConstants.DEFAULT_THREAD_POOL_SIZE
-:
Integer.parseInt(properties.getProperty(pherf.default.threadpool));
+public void add(Workload workload) throws Exception {
+this.jobs.put(workload, pool.submit(workload.execute()));
+}
-this.pool = Executors.newFixedThreadPool(this.poolSize);
-if (monitor) {
-
initMonitor(Integer.parseInt(properties.getProperty(pherf.default.monitorFrequency)));
+/**
+ * Blocks on waiting for all workloads to finish. If a
+ * {@link org.apache.phoenix.pherf.workload.Workload} Requires
complete() to be called, it must
+ * be called prior to using this method. Otherwise it will block
infinitely.
+ */
+public synchronized void get() {
--- End diff --
Yep, I can remove synchronized.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: Mixed R/W workload support
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/89#discussion_r32848223
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/WriteParams.java
---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.pherf.configuration;
+
+import javax.xml.bind.annotation.XmlAttribute;
+
+public class WriteParams {
+private int writerThreadCount;
+private long threadSleepDuration;
+private long batchSize;
+private long executionDurationInMs;
+
+public WriteParams() {
+this.batchSize = Long.MIN_VALUE;
+this.writerThreadCount = Integer.MIN_VALUE;
+this.threadSleepDuration = Long.MIN_VALUE;
+this.executionDurationInMs = Long.MAX_VALUE;
+}
+
+public long getThreadSleepDuration() {
+return threadSleepDuration;
+}
+
+@SuppressWarnings(unused)
--- End diff --
No choice here. The setters are being called by the xml bindings. You just
don't see the references to them because they are called dynamically. They are
being called though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: Mixed R/W workload support
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/89#discussion_r32848731
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/result/DataModelResult.java
---
@@ -18,61 +18,57 @@
package org.apache.phoenix.pherf.result;
-import java.util.ArrayList;
-import java.util.List;
+import org.apache.phoenix.pherf.configuration.DataModel;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.XmlRootElement;
+import java.util.ArrayList;
+import java.util.List;
-import org.apache.phoenix.pherf.configuration.DataModel;
+@XmlRootElement(namespace = org.apache.phoenix.pherf.result) public
class DataModelResult
+extends DataModel {
+private ListScenarioResult scenarioResult = new
ArrayListScenarioResult();
+private String zookeeper;
-@XmlRootElement(namespace = org.apache.phoenix.pherf.result)
-public class DataModelResult extends DataModel {
- private ListScenarioResult scenarioResult = new
ArrayListScenarioResult();
- private String zookeeper;
+public ListScenarioResult getScenarioResult() {
+return scenarioResult;
+}
- public ListScenarioResult getScenarioResult() {
- return scenarioResult;
- }
+@SuppressWarnings(unused) public void
setScenarioResult(ListScenarioResult scenarioResult) {
--- End diff --
All the xml data bindings require the setters
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: Mixed R/W workload support
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/89#discussion_r32848856
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/result/QuerySetResult.java
---
@@ -18,31 +18,30 @@
package org.apache.phoenix.pherf.result;
+import org.apache.phoenix.pherf.configuration.QuerySet;
+
import java.util.ArrayList;
import java.util.List;
-import org.apache.phoenix.pherf.configuration.QuerySet;
-
public class QuerySetResult extends QuerySet {
-
- private ListQueryResult queryResults = new ArrayListQueryResult();
-
- public QuerySetResult(QuerySet querySet) {
- this.setConcurrency(querySet.getConcurrency());
- this.setNumberOfExecutions(querySet.getNumberOfExecutions());
-
this.setExecutionDurationInMs(querySet.getExecutionDurationInMs());
- this.setExecutionType(querySet.getExecutionType());
- }
-
- public QuerySetResult() {
- }
-
- public ListQueryResult getQueryResults() {
- return queryResults;
- }
-
-@SuppressWarnings(unused)
-public void setQueryResults(ListQueryResult queryResults) {
- this.queryResults = queryResults;
- }
+
+private ListQueryResult queryResults = new ArrayListQueryResult();
+
+public QuerySetResult(QuerySet querySet) {
+this.setConcurrency(querySet.getConcurrency());
+this.setNumberOfExecutions(querySet.getNumberOfExecutions());
+this.setExecutionDurationInMs(querySet.getExecutionDurationInMs());
+this.setExecutionType(querySet.getExecutionType());
+}
+
+public QuerySetResult() {
+}
+
+public ListQueryResult getQueryResults() {
+return queryResults;
+}
+
+@SuppressWarnings(unused) public void
setQueryResults(ListQueryResult queryResults) {
--- End diff --
Good point. I think we can remove this one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: Mixed R/W workload support
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/89#discussion_r32848091
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java
---
@@ -146,49 +173,77 @@ public void run() throws Exception {
}
return;
}
-workloadExec = (scenarioFile == null)
-? new WorkloadExecutor(properties,
-new
XMLConfigParser(PherfConstants.DEFAULT_FILE_PATTERN),
-monitor)
-: new WorkloadExecutor(properties,
-new XMLConfigParser(scenarioFile),
-monitor);
+XMLConfigParser parser = new XMLConfigParser(scenarioFile);
// Drop tables with PHERF schema and regex comparison
if (null != dropPherfTablesRegEx) {
-logger.info(\nDropping existing table with PHERF namename
and
-+ dropPherfTablesRegEx + regex expression.);
-new PhoenixUtil().deleteTables(dropPherfTablesRegEx);
+logger.info(
+\nDropping existing table with PHERF namename and
+ dropPherfTablesRegEx
++ regex expression.);
+phoenixUtil.deleteTables(dropPherfTablesRegEx);
}
-// Schema and Data Load
-if (loadData) {
+if (monitor) {
+MonitorManager
+monitorManager =
+new MonitorManager(Integer.parseInt(
+
properties.getProperty(pherf.default.monitorFrequency)));
+workloadExecutor.add(monitorManager);
--- End diff --
We want the monitor to run through the duration of the workloads. If we
call get here, it would block on the monitor until it completed then exit
before the other workloads finish.
I am glad you commented on this though. I just realized I am not calling
complete() on the monitor at the end. So it can block indefinitely right now or
leave the monitor dangling. I'll make that change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: Mixed R/W workload support
Github user codymarcel commented on a diff in the pull request: https://github.com/apache/phoenix/pull/89#discussion_r32848277 --- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/jmx/MonitorManager.java --- @@ -51,13 +58,11 @@ private volatile boolean shouldStop = false; --- End diff -- Atomic is better. I will change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] phoenix pull request: Mixed R/W workload support
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/89#discussion_r32850256
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/result/DataModelResult.java
---
@@ -18,61 +18,57 @@
package org.apache.phoenix.pherf.result;
-import java.util.ArrayList;
-import java.util.List;
+import org.apache.phoenix.pherf.configuration.DataModel;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.XmlRootElement;
+import java.util.ArrayList;
+import java.util.List;
-import org.apache.phoenix.pherf.configuration.DataModel;
+@XmlRootElement(namespace = org.apache.phoenix.pherf.result) public
class DataModelResult
+extends DataModel {
+private ListScenarioResult scenarioResult = new
ArrayListScenarioResult();
+private String zookeeper;
-@XmlRootElement(namespace = org.apache.phoenix.pherf.result)
-public class DataModelResult extends DataModel {
- private ListScenarioResult scenarioResult = new
ArrayListScenarioResult();
- private String zookeeper;
+public ListScenarioResult getScenarioResult() {
+return scenarioResult;
+}
- public ListScenarioResult getScenarioResult() {
- return scenarioResult;
- }
+@SuppressWarnings(unused) public void
setScenarioResult(ListScenarioResult scenarioResult) {
--- End diff --
James barked at them the last time I left it off.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: Mixed R/W workload support
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/89#discussion_r32765859
--- Diff:
phoenix-pherf/src/it/java/org/apache/phoenix/pherf/PherfMainIT.java ---
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.pherf;
+
+import org.junit.Test;
+
+import static org.junit.Assert.fail;
+
+public class PherfMainIT extends ResultBaseTestIT {
+
+@Test
+public void testPherfMain() throws Exception{
+String[] args = {-q, --scenarioFile,
.*prod_test_unsalted_scenario.* };
+try {
+Pherf.main(args);
+} catch (Exception e) {
+fail(Test encountered an exception.);
+throw e;
--- End diff --
yep, will remove the try. I think junit will automatically fail if an
exception is thrown. This is just wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: Mixed R/W workload support
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/89#discussion_r32763915
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/workload/QueryExecutor.java
---
@@ -18,227 +18,271 @@
package org.apache.phoenix.pherf.workload;
-import java.sql.Connection;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Map;
-
-import org.apache.phoenix.pherf.PherfConstants.RunMode;
-import org.apache.phoenix.pherf.configuration.XMLConfigParser;
-
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.phoenix.pherf.PherfConstants.RunMode;
+import org.apache.phoenix.pherf.configuration.*;
import org.apache.phoenix.pherf.result.*;
+import org.apache.phoenix.pherf.util.PhoenixUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import org.apache.phoenix.pherf.configuration.DataModel;
-import org.apache.phoenix.pherf.configuration.ExecutionType;
-import org.apache.phoenix.pherf.configuration.Query;
-import org.apache.phoenix.pherf.configuration.QuerySet;
-import org.apache.phoenix.pherf.configuration.Scenario;
-import org.apache.phoenix.pherf.util.PhoenixUtil;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+public class QueryExecutor implements Workload {
+private static final Logger logger =
LoggerFactory.getLogger(QueryExecutor.class);
+private ListDataModel dataModels;
+private String queryHint;
+private final RunMode runMode;
+private final boolean exportCSV;
+private final ExecutorService pool;
+private final XMLConfigParser parser;
+private final PhoenixUtil util;
+
+public QueryExecutor(XMLConfigParser parser, PhoenixUtil util,
ExecutorService pool) {
+this(parser, util, pool, parser.getDataModels(), null, false,
RunMode.PERFORMANCE);
+}
+
+public QueryExecutor(XMLConfigParser parser,
+PhoenixUtil util,
+ExecutorService pool,
+ListDataModel dataModels,
+String queryHint,
+boolean exportCSV, RunMode runMode) {
+this.parser = parser;
+this.queryHint = queryHint;
+this.exportCSV = exportCSV;
+this.runMode = runMode;
+this.dataModels = dataModels;
+this.pool = pool;
+this.util = util;
+}
+
+@Override
+public void complete() {
+
+}
+
+/**
+ * Calls in Multithreaded Query Executor for all datamodels
+ *
+ * @throws Exception
+ */
+public Runnable execute() throws Exception {
+Runnable runnable = null;
+for (DataModel dataModel : dataModels) {
+if (exportCSV) {
+runnable = exportAllScenarios(dataModel);
+} else {
+runnable = executeAllScenarios(dataModel);
+}
+}
+return runnable;
+}
+
+/**
+ * Export all queries results to CSV
+ *
+ * @param dataModel
+ * @throws Exception
+ */
+protected Runnable exportAllScenarios(final DataModel dataModel)
throws Exception {
+return new Runnable() {
+@Override
+public void run() {
+try {
+
+ListScenario scenarios = dataModel.getScenarios();
+QueryVerifier exportRunner = new QueryVerifier(false);
+for (Scenario scenario : scenarios) {
+for (QuerySet querySet : scenario.getQuerySet()) {
+util.executeQuerySetDdls(querySet);
+for (Query query : querySet.getQuery()) {
+exportRunner.exportCSV(query);
+}
+}
+}
+} catch (Exception e) {
+logger.warn(, e);
+}
+}
+};
+}
+
+/**
+ * Execute all scenarios
+ *
+ * @param dataModel
+ * @throws Exception
+ */
+protected Runnable executeAllScenarios(final DataModel dataModel)
throws Exception {
+return new Runnable() {
+@Override
+public void run() {
+ListDataModelResult dataModelResults = new ArrayList();
+DataModelResult
[GitHub] phoenix pull request: Mixed R/W workload support
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/89#discussion_r32766179
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/jmx/MonitorManager.java ---
@@ -135,10 +154,7 @@ public synchronized boolean shouldStop() {
return shouldStop;
}
-public synchronized void stop() {
-this.shouldStop = true;
-}
-
+@SuppressWarnings(unused)
--- End diff --
I found it useful for testing and debugging. I just didn't leave any of the
calling code in. I didn't want to have to re-add it later. I can remove it you
don't like it though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: Mixed R/W workload support
Github user codymarcel commented on the pull request: https://github.com/apache/phoenix/pull/89#issuecomment-112984238 @mujtabachohan @twdsilva --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] phoenix pull request: Mixed R/W workload support
Github user codymarcel commented on the pull request: https://github.com/apache/phoenix/pull/89#issuecomment-112984696 @samarthjain --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/41#discussion_r26231562
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/loaddata/DataLoader.java
---
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.pherf.loaddata;
+
+import java.math.BigDecimal;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.sql.Types;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+import org.apache.phoenix.pherf.result.ResultUtil;
+import org.apache.phoenix.pherf.util.ResourceList;
+import org.apache.phoenix.pherf.util.RowCalculator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.configuration.Column;
+import org.apache.phoenix.pherf.configuration.DataModel;
+import org.apache.phoenix.pherf.configuration.Scenario;
+import org.apache.phoenix.pherf.configuration.XMLConfigParser;
+import org.apache.phoenix.pherf.exception.PherfException;
+import org.apache.phoenix.pherf.result.DataLoadThreadTime;
+import org.apache.phoenix.pherf.result.DataLoadTimeSummary;
+import org.apache.phoenix.pherf.rules.DataValue;
+import org.apache.phoenix.pherf.rules.RulesApplier;
+import org.apache.phoenix.pherf.util.PhoenixUtil;
+
+public class DataLoader {
+private static final Logger logger =
LoggerFactory.getLogger(DataLoader.class);
+private final PhoenixUtil pUtil = new PhoenixUtil();
+private final XMLConfigParser parser;
+private final RulesApplier rulesApplier;
+private final ResultUtil resultUtil;
+private final ExecutorService pool;
+private final Properties properties;
+
+private final int threadPoolSize;
+private final int batchSize;
+
+public DataLoader(XMLConfigParser parser) throws Exception {
+this(new ResourceList().getProperties(), parser);
+}
+
+/**
+ * Default the writers to use up all available cores for threads.
+ *
+ * @param parser
+ * @throws Exception
+ */
+public DataLoader(Properties properties, XMLConfigParser parser)
throws Exception {
+this.parser = parser;
+this.properties = properties;
+this.rulesApplier = new RulesApplier(this.parser);
+this.resultUtil = new ResultUtil();
+int size =
Integer.parseInt(properties.getProperty(pherf.default.dataloader.threadpool));
+this.threadPoolSize = (size == 0) ?
Runtime.getRuntime().availableProcessors() : size;
+this.pool = Executors.newFixedThreadPool(this.threadPoolSize);
+String bSize =
properties.getProperty(pherf.default.dataloader.batchsize);
+this.batchSize = (bSize == null) ?
PherfConstants.DEFAULT_BATCH_SIZE : Integer.parseInt(bSize);
+}
+
+public void execute() throws Exception {
+try {
+DataModel model = getParser().getDataModels().get(0);
+DataLoadTimeSummary dataLoadTimeSummary = new
DataLoadTimeSummary();
+DataLoadThreadTime dataLoadThreadTime = new
DataLoadThreadTime();
+
+for (Scenario scenario : getParser().getScenarios()) {
+ListFuture writeBatches = new ArrayListFuture();
+logger.info(\nLoading + scenario.getRowCount()
++ rows for + scenario.getTableName());
+long start = System.currentTimeMillis();
+
+RowCalculator rowCalculator = new
RowCalculator
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request: https://github.com/apache/phoenix/pull/41#discussion_r26239765 --- Diff: phoenix-pherf/pom.xml --- @@ -0,0 +1,291 @@ +!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ License); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, software + ~ distributed under the License is distributed on an AS IS BASIS, + ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + ~ See the License for the specific language governing permissions and + ~ limitations under the License. + -- + +project xmlns=http://maven.apache.org/POM/4.0.0; xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation=http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd; +modelVersion4.0.0/modelVersion +parent +groupIdorg.apache.phoenix/groupId +artifactIdphoenix/artifactId +version5.0.0-SNAPSHOT/version +/parent + +artifactIdpherf/artifactId +packagingjar/packaging +!--version1.0-SNAPSHOT/version-- +namePhoenix - Pherf/name + +properties +phoenix.version4.3.0/phoenix.version --- End diff -- Removed. This is no longer needed since nothing is using it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/41#discussion_r26339859
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/jmx/MonitorManager.java ---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.pherf.jmx;
+
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.exception.FileLoaderRuntimeException;
+import org.apache.phoenix.pherf.jmx.monitors.Monitor;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+import org.apache.phoenix.pherf.result.impl.CSVResultHandler;
+import org.apache.phoenix.pherf.result.Result;
+import org.apache.phoenix.pherf.result.ResultHandler;
+import org.apache.phoenix.util.DateUtil;
+
+import javax.management.*;
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.util.*;
+
+/**
+ * This class starts JMX stats for the configured monitors. Monitors
should be configured in MonitorDetails Enum.
+ * Each stat implements {@link
org.apache.phoenix.pherf.jmx.monitors.Monitor}.
+ *
+ * For the duration of any Pherf run, when the configured {@link
org.apache.phoenix.pherf.PherfConstants#MONITOR_FREQUENCY}
+ * is reached a snapshot of each monitor is taken and dumped out to a log
file.
+ */
+public class MonitorManager implements Runnable {
+// List of MonitorDetails for all the running monitors.
+// TODO Move this out to config. Possible use Guice and use IOC to
inject it in.
+private static final ListMonitorDetails MONITOR_DETAILS_LIST =
+Arrays.asList(MonitorDetails.values());
+private final ResultHandler resultHandler;
+private final long monitorFrequency;
+private volatile long rowCount;
+private volatile boolean shouldStop = false;
+private volatile boolean isRunning = false;
+
+public MonitorManager() throws Exception {
+this(PherfConstants.MONITOR_FREQUENCY);
+}
+
+/**
+ *
+ * @param monitorFrequency Frequency at which monitor stats are
written to a log file.
+ * @throws Exception
+ */
+public MonitorManager(long monitorFrequency) throws Exception {
+this.monitorFrequency = monitorFrequency;
+MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
+
+// Register all the monitors to JMX
+for (MonitorDetails monitorDetails : MONITOR_DETAILS_LIST) {
+StandardMBean bean = new
StandardMBean(monitorDetails.getMonitor(), Monitor.class);
+ObjectName monitorThreadStatName = new
ObjectName(monitorDetails.toString());
+try {
+mbs.registerMBean(bean, monitorThreadStatName);
+} catch (InstanceAlreadyExistsException e) {
+mbs.unregisterMBean(monitorThreadStatName);
+mbs.registerMBean(bean, monitorThreadStatName);
+}
+}
+this.rowCount = 0;
+this.resultHandler = new
CSVResultHandler(PherfConstants.MONITOR_FILE_NAME, ResultFileDetails.CSV);
+}
+
+@Override
+public void run() {
+try {
+while (!shouldStop()) {
+isRunning = true;
+List rowValues = new ArrayListString();
+synchronized (resultHandler) {
+for (MonitorDetails monitorDetails :
MONITOR_DETAILS_LIST) {
+rowValues.clear();
+try {
+StandardMBean bean = new
StandardMBean(monitorDetails.getMonitor(), Monitor.class);
+
+Calendar calendar = new GregorianCalendar();
+rowValues.add(monitorDetails);
+
+rowValues.add(((Monitor)
bean.getImplementation()).getStat
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on the pull request: https://github.com/apache/phoenix/pull/41#issuecomment-78597830 I think I have addressed any outstanding concerns. Let me know if there is anything else. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request: https://github.com/apache/phoenix/pull/41#discussion_r26230867 --- Diff: phoenix-pherf/README.md --- @@ -0,0 +1,105 @@ +Pherf is a performance test framework that exercises HBase through Apache Phoenix, a SQL layer interface. --- End diff -- Sure, want it in the pull or in a subsequent patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request: https://github.com/apache/phoenix/pull/41#discussion_r26231675 --- Diff: phoenix-pherf/src/main/resources/scenario/prod_test_unsalted_scenario.xml --- @@ -0,0 +1,342 @@ +?xml version=1.0 encoding=UTF-8 standalone=yes? +!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ License); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, software + ~ distributed under the License is distributed on an AS IS BASIS, + ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + ~ See the License for the specific language governing permissions and + ~ limitations under the License. + -- + +datamodel release=194 name=PROD_TEST_UNSALTED +datamapping +column +!-- This column type defines what will generally happen to VARCHAR fields unless they are explicitly defined or overridden elsewhere -- +typeVARCHAR/type +dataSequenceRANDOM/dataSequence +length15/length +nameGENERAL_VARCHAR/name +/column +column +typeCHAR/type +dataSequenceRANDOM/dataSequence +length15/length +nameGENERAL_CHAR/name +/column +column +typeDATE/type +!--SEQUENTIAL is unsupported for DATE -- +dataSequenceRANDOM/dataSequence +!-- Number [0-100] that represents the probability of creating a null value -- +!-- The higher the number, the more like the value will returned will be null -- +!-- Leaving this tag out is equivalent to having a 0 probability. i.e. never null -- +nullChance0/nullChance +minValue1975/minValue +maxValue2025/maxValue +nameGENERAL_DATE/name +/column +column +typeDECIMAL/type +dataSequenceRANDOM/dataSequence +minValue0/minValue +maxValue1/maxValue + +!-- Precision is limited to 18 -- +precision18/precision +!-- Number [0-100] that represents the probability of creating a null value -- +!-- The higher the number, the more like the value will returned will be null -- +!-- Leaving this tag out is equivalent to having a 0 probability. i.e. never null -- +nullChance90/nullChance +nameGENERAL_DECIMAL/name +/column +column +typeINTEGER/type +dataSequenceRANDOM/dataSequence +minValue1/minValue +maxValue5000/maxValue +!-- Number [0-100] that represents the probability of creating a null value -- +!-- The higher the number, the more like the value will returned will be null -- +!-- Leaving this tag out is equivalent to having a 0 probability. i.e. never null -- +nullChance100/nullChance +nameGENERAL_INTEGER/name +/column +column +typeCHAR/type +userDefinedtrue/userDefined +dataSequenceLIST/dataSequenceID --- End diff -- Nice catch. I am not sure how that one sneaked in. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request: https://github.com/apache/phoenix/pull/41#discussion_r26231197 --- Diff: phoenix-pherf/cluster/pherf.sh --- @@ -0,0 +1,33 @@ +#!/bin/bash --- End diff -- Yea, I planned to convert this to python post commit as a new JIRA. Moving the scripts will require some build to Pherf. This pull is already huge so I was trying to limit the functionality changes that could possibly impact phoenix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/41#discussion_r26256441
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/jmx/MonitorManager.java ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.pherf.jmx;
+
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.exception.FileLoaderRuntimeException;
+import org.apache.phoenix.pherf.jmx.monitors.Monitor;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+import org.apache.phoenix.pherf.result.impl.CSVResultHandler;
+import org.apache.phoenix.pherf.result.Result;
+import org.apache.phoenix.pherf.result.ResultHandler;
+import org.apache.phoenix.util.DateUtil;
+
+import javax.management.*;
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.util.*;
+
+/**
+ * This class starts JMX stats for the configured monitors. Monitors
should be configured in MonitorDetails Enum.
+ * Each stat implements {@link
org.apache.phoenix.pherf.jmx.monitors.Monitor}.
+ *
+ * For the duration of any Pherf run, when the configured {@link
org.apache.phoenix.pherf.PherfConstants#MONITOR_FREQUENCY}
+ * is reached a snapshot of each monitor is taken and dumped out to a log
file.
+ */
+public class MonitorManager implements Runnable {
+// List of MonitorDetails for all the running monitors.
+private static final ListMonitorDetails MONITOR_DETAILS_LIST;
+private final ResultHandler resultHandler;
+private final long monitorFrequency;
+private volatile long rowCount;
+private volatile boolean shouldStop = false;
+private volatile boolean isRunning = false;
+
+static {
+// TODO Move this out to config. Possible use Guice and use IOC to
inject it in.
+MONITOR_DETAILS_LIST = Arrays.asList(MonitorDetails.values());
+}
+
+public MonitorManager() throws Exception {
+this(PherfConstants.MONITOR_FREQUENCY);
+}
+
+/**
+ *
+ * @param monitorFrequency Frequency at which monitor stats are
written to a log file.
+ * @throws Exception
+ */
+public MonitorManager(long monitorFrequency) throws Exception {
+this.monitorFrequency = monitorFrequency;
+MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
+
+// Register all the monitors to JMX
+for (MonitorDetails monitorDetails : MONITOR_DETAILS_LIST) {
+StandardMBean bean = new
StandardMBean(monitorDetails.getMonitor(), Monitor.class);
+ObjectName monitorThreadStatName = new
ObjectName(monitorDetails.toString());
+try {
+mbs.registerMBean(bean, monitorThreadStatName);
+} catch (InstanceAlreadyExistsException e) {
+mbs.unregisterMBean(monitorThreadStatName);
+mbs.registerMBean(bean, monitorThreadStatName);
+}
+}
+this.rowCount = 0;
+this.resultHandler = new
CSVResultHandler(PherfConstants.MONITOR_FILE_NAME, ResultFileDetails.CSV);
+}
+
+@Override
+public void run() {
+try {
+while (!shouldStop()) {
+isRunning = true;
+List rowValues = new ArrayListString();
+synchronized (resultHandler) {
+for (MonitorDetails monitorDetails :
MONITOR_DETAILS_LIST) {
+rowValues.clear();
+try {
+StandardMBean bean = new
StandardMBean(monitorDetails.getMonitor(), Monitor.class);
+
+Calendar calendar = new GregorianCalendar();
+rowValues.add(monitorDetails);
+
+rowValues.add
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request: https://github.com/apache/phoenix/pull/41#discussion_r26256135 --- Diff: phoenix-pherf/README.md --- @@ -0,0 +1,105 @@ +Pherf is a performance test framework that exercises HBase through Apache Phoenix, a SQL layer interface. --- End diff -- Cool. i'll create a jira and do that post commit of this pull. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/41#discussion_r26243047
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/Scenario.java
---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.pherf.configuration;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+import javax.xml.bind.annotation.XmlAttribute;
+import javax.xml.bind.annotation.XmlElement;
+import javax.xml.bind.annotation.XmlRootElement;
+
+import org.apache.phoenix.pherf.util.PhoenixUtil;
+
+@XmlRootElement(namespace =
org.apache.phoenix.pherf.configuration.DataModel)
+public class Scenario {
+ private String tableName;
+ private int rowCount;
+private MapString,String phoenixProperties;
+private DataOverride dataOverride;
+ private ListQuerySet querySet = new ArrayListQuerySet();
+ private String name;
+
+public Scenario() {
+}
+
+/**
+ * Scenarios have to have unique table names
+ * @param object
+ * @return
+ */
+@Override
+public boolean equals(Object object) {
--- End diff --
Added
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/41#discussion_r26243169
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/jmx/MonitorManager.java ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.pherf.jmx;
+
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.exception.FileLoaderRuntimeException;
+import org.apache.phoenix.pherf.jmx.monitors.Monitor;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+import org.apache.phoenix.pherf.result.impl.CSVResultHandler;
+import org.apache.phoenix.pherf.result.Result;
+import org.apache.phoenix.pherf.result.ResultHandler;
+import org.apache.phoenix.util.DateUtil;
+
+import javax.management.*;
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.util.*;
+
+/**
+ * This class starts JMX stats for the configured monitors. Monitors
should be configured in MonitorDetails Enum.
+ * Each stat implements {@link
org.apache.phoenix.pherf.jmx.monitors.Monitor}.
+ *
+ * For the duration of any Pherf run, when the configured {@link
org.apache.phoenix.pherf.PherfConstants#MONITOR_FREQUENCY}
+ * is reached a snapshot of each monitor is taken and dumped out to a log
file.
+ */
+public class MonitorManager implements Runnable {
+// List of MonitorDetails for all the running monitors.
+private static final ListMonitorDetails MONITOR_DETAILS_LIST;
+private final ResultHandler resultHandler;
+private final long monitorFrequency;
+private volatile long rowCount;
+private volatile boolean shouldStop = false;
+private volatile boolean isRunning = false;
+
+static {
+// TODO Move this out to config. Possible use Guice and use IOC to
inject it in.
+MONITOR_DETAILS_LIST = Arrays.asList(MonitorDetails.values());
+}
+
+public MonitorManager() throws Exception {
+this(PherfConstants.MONITOR_FREQUENCY);
+}
+
+/**
+ *
+ * @param monitorFrequency Frequency at which monitor stats are
written to a log file.
+ * @throws Exception
+ */
+public MonitorManager(long monitorFrequency) throws Exception {
+this.monitorFrequency = monitorFrequency;
+MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
+
+// Register all the monitors to JMX
+for (MonitorDetails monitorDetails : MONITOR_DETAILS_LIST) {
+StandardMBean bean = new
StandardMBean(monitorDetails.getMonitor(), Monitor.class);
+ObjectName monitorThreadStatName = new
ObjectName(monitorDetails.toString());
+try {
+mbs.registerMBean(bean, monitorThreadStatName);
+} catch (InstanceAlreadyExistsException e) {
+mbs.unregisterMBean(monitorThreadStatName);
+mbs.registerMBean(bean, monitorThreadStatName);
+}
+}
+this.rowCount = 0;
+this.resultHandler = new
CSVResultHandler(PherfConstants.MONITOR_FILE_NAME, ResultFileDetails.CSV);
+}
+
+@Override
+public void run() {
+try {
+while (!shouldStop()) {
+isRunning = true;
+List rowValues = new ArrayListString();
+synchronized (resultHandler) {
+for (MonitorDetails monitorDetails :
MONITOR_DETAILS_LIST) {
+rowValues.clear();
+try {
+StandardMBean bean = new
StandardMBean(monitorDetails.getMonitor(), Monitor.class);
+
+Calendar calendar = new GregorianCalendar();
+rowValues.add(monitorDetails);
+
+rowValues.add
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request: https://github.com/apache/phoenix/pull/41#discussion_r26239341 --- Diff: phoenix-pherf/config/pherf.properties --- @@ -0,0 +1,61 @@ +# --- End diff -- Fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/41#discussion_r26241095
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/XMLConfigParser.java
---
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.pherf.configuration;
+
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.exception.FileLoaderException;
+import org.apache.phoenix.pherf.util.ResourceList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.OutputStream;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+import javax.xml.bind.JAXBContext;
+import javax.xml.bind.JAXBException;
+import javax.xml.bind.Marshaller;
+import javax.xml.bind.Unmarshaller;
+
+public class XMLConfigParser {
+
+private static final Logger logger =
LoggerFactory.getLogger(XMLConfigParser.class);
+private String filePattern;
+private ListDataModel dataModels;
+private ListScenario scenarios = null;
+private ResourceList resourceList;
+private CollectionPath paths = null;// Only populate
once. It may be slow of CP is large.
+
+public XMLConfigParser(String pattern) throws Exception{
+init(pattern);
+}
+
+public ListDataModel getDataModels() {
+return dataModels;
+}
+
+public CollectionPath getPaths(String strPattern) throws Exception{
--- End diff --
I'll synchronize it. Threads may actually call it. Though I have not seen
any concurrency bugs due to it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/41#discussion_r26243302
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/jmx/MonitorManager.java ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.pherf.jmx;
+
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.exception.FileLoaderRuntimeException;
+import org.apache.phoenix.pherf.jmx.monitors.Monitor;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+import org.apache.phoenix.pherf.result.impl.CSVResultHandler;
+import org.apache.phoenix.pherf.result.Result;
+import org.apache.phoenix.pherf.result.ResultHandler;
+import org.apache.phoenix.util.DateUtil;
+
+import javax.management.*;
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.util.*;
+
+/**
+ * This class starts JMX stats for the configured monitors. Monitors
should be configured in MonitorDetails Enum.
+ * Each stat implements {@link
org.apache.phoenix.pherf.jmx.monitors.Monitor}.
+ *
+ * For the duration of any Pherf run, when the configured {@link
org.apache.phoenix.pherf.PherfConstants#MONITOR_FREQUENCY}
+ * is reached a snapshot of each monitor is taken and dumped out to a log
file.
+ */
+public class MonitorManager implements Runnable {
+// List of MonitorDetails for all the running monitors.
+private static final ListMonitorDetails MONITOR_DETAILS_LIST;
+private final ResultHandler resultHandler;
+private final long monitorFrequency;
+private volatile long rowCount;
+private volatile boolean shouldStop = false;
+private volatile boolean isRunning = false;
+
+static {
--- End diff --
done :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/41#discussion_r26240525
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/workload/MultithreadedRunner.java
---
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.pherf.workload;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.util.Calendar;
+import java.util.Date;
+
+import org.apache.phoenix.pherf.PherfConstants.RunMode;
+
+import org.apache.phoenix.pherf.result.DataModelResult;
+import org.apache.phoenix.pherf.result.ResultManager;
+import org.apache.phoenix.pherf.result.RunTime;
+import org.apache.phoenix.pherf.result.ThreadTime;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.phoenix.pherf.configuration.Query;
+import org.apache.phoenix.pherf.util.PhoenixUtil;
+
+class MultithreadedRunner implements Runnable {
+ private static final Logger logger = LoggerFactory
+ .getLogger(MultithreadedRunner.class);
+ private Thread t;
+ private Query query;
+ private ThreadTime threadTime;
+ private PhoenixUtil pUtil = new PhoenixUtil();
+ private String threadName;
+ private DataModelResult dataModelResult;
+ private long numberOfExecutions;
+ private long executionDurationInMs;
+ private static long lastResultWritten = System.currentTimeMillis() -
1000;
+private final ResultManager resultManager;
+
+/**
+ * Multithreaded runner
+ *
+ * @param threadName
+ * @param query
+ * @param dataModelResult
+ * @param threadTime
+ * @param numberOfExecutions
+ * @param executionDurationInMs
+ */
+MultithreadedRunner(String threadName,
+Query query,
+DataModelResult dataModelResult,
+ThreadTime threadTime,
+long numberOfExecutions,
+long executionDurationInMs) {
+this.query = query;
+this.threadName = threadName;
+this.threadTime = threadTime;
+this.dataModelResult = dataModelResult;
+this.numberOfExecutions = numberOfExecutions;
+this.executionDurationInMs = executionDurationInMs;
+this.resultManager = new ResultManager(dataModelResult.getName(),
RunMode.PERFORMANCE);
+}
+
+ /**
+* Executes run for a minimum of number of execution or execution
duration
+*/
+ public void run() {
+ logger.info(\n\nThread Starting + t.getName() + ; +
query.getStatement() + for
+ + numberOfExecutions + times\n\n);
+ Long start = System.currentTimeMillis();
+ for (long i = numberOfExecutions; (i 0 ((System
+ .currentTimeMillis() - start)
executionDurationInMs)); i--) {
+ try {
+synchronized (resultManager) {
+timedQuery();
+if ((System.currentTimeMillis() - lastResultWritten)
1000) {
+//
resultUtil.writeResultToFile(dataModelResult, dataModelResult.getName(),
RunMode.PERFORMANCE);
--- End diff --
Removed. The write method takes care of it now. That was from old
implementation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/41#discussion_r26241491
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/XMLConfigParser.java
---
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.pherf.configuration;
+
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.exception.FileLoaderException;
+import org.apache.phoenix.pherf.util.ResourceList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.OutputStream;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+import javax.xml.bind.JAXBContext;
+import javax.xml.bind.JAXBException;
+import javax.xml.bind.Marshaller;
+import javax.xml.bind.Unmarshaller;
+
+public class XMLConfigParser {
+
+private static final Logger logger =
LoggerFactory.getLogger(XMLConfigParser.class);
+private String filePattern;
+private ListDataModel dataModels;
+private ListScenario scenarios = null;
+private ResourceList resourceList;
+private CollectionPath paths = null;// Only populate
once. It may be slow of CP is large.
+
+public XMLConfigParser(String pattern) throws Exception{
+init(pattern);
+}
+
+public ListDataModel getDataModels() {
+return dataModels;
+}
+
+public CollectionPath getPaths(String strPattern) throws Exception{
+if (paths != null) {
+return paths;
+}
+paths = getResources(strPattern);
+return paths;
+}
+
+public ListScenario getScenarios() throws Exception{
+if (scenarios != null) {
+return scenarios;
+}
+
+scenarios = new ArrayListScenario();
+for (Path path : getPaths(getFilePattern())) {
+try {
+ListScenario scenarioList =
XMLConfigParser.readDataModel(path).getScenarios();
+for (Scenario scenario : scenarioList) {
+scenarios.add(scenario);
+}
+} catch (JAXBException e) {
+e.printStackTrace();
+}
+}
+return scenarios;
+}
+
+/**
--- End diff --
I occasionally use it for testing and debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/41#discussion_r26239994
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java
---
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.pherf;
+
+import org.apache.phoenix.pherf.configuration.XMLConfigParser;
+import org.apache.phoenix.pherf.schema.SchemaReader;
+import org.apache.phoenix.pherf.util.PhoenixUtil;
+import org.apache.phoenix.pherf.util.ResourceList;
+import org.apache.phoenix.pherf.workload.WorkloadExecutor;
+
+import org.apache.commons.cli.*;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.file.Path;
+import java.util.Collection;
+import java.util.Properties;
+
+public class Pherf {
+private static final Logger logger =
LoggerFactory.getLogger(Pherf.class);
+private static final Options options = new Options();
+
+static {
+//options.addOption(r, release, true, Required option.
Release that you want to test. 190, 192, ...);
+options.addOption(m, monitor, false, Launch the stats
profilers);
+options.addOption(monitorFrequency, true, Override for
frequency in Ms for which monitor should log stats. +
+\n See pherf.default.monitorFrequency in
pherf.properties);
+options.addOption(d, debug, false, Put tool in debug mode);
+options.addOption(z, zookeeper, true, HBase Zookeeper address
for connection. Default: localhost);
+options.addOption(l, load, false, Loads data according to
specified configuration values.);
+options.addOption(scenarioFile, true, Regex or file name for
the Test Scenario configuration .xml file to use.);
+options.addOption(drop, true, Regex drop all tables with schema
name as PHERF. +
+\nExample drop Event tables: -drop .*(EVENT).* Drop all:
-drop .* or -drop all);
+options.addOption(schemaFile, true, Regex or file name for the
Test phoenix table schema .sql to use.);
+options.addOption(rowCountOverride, true, Row count override to
use instead of one specified in scenario.);
+options.addOption(hint, true, Executes all queries with
specified hint. Example SMALL);
+options.addOption(diff, false, Run pherf in verification mode
and diff with exported results);
+options.addOption(export, false, Exports query results to CSV
files in + PherfConstants.EXPORT_DIR + directory);
+options.addOption(listFiles, false, List available resource
files);
+options.addOption(writerThreadSize, true, Override the default
number of writer threads. +
+See pherf.default.dataloader.threadpool in
Pherf.properties.);
+options.addOption(q, query, false, Executes multi-threaded
query sets);
+options.addOption(h, help, false, Get help on using this
utility.);
+}
+
+//private final String release;
+private final String zookeeper;
+private final String scenarioFile;
+private final String schemaFile;
+private final String queryHint;
+private final Properties properties;
+private final boolean loadData;
+private final String dropPherfTablesRegEx;
+private final boolean executeQuerySets;
+private final boolean exportCSV;
+private final boolean diff;
+private final boolean monitor;
+private final int rowCountOverride;
+private final boolean listFiles;
+
+public Pherf(String[] args) throws Exception {
+CommandLineParser parser = new PosixParser();
+CommandLine command = null;
+HelpFormatter hf = new HelpFormatter();
+
+try {
+command = parser.parse(options, args);
+} catch (ParseException e) {
+hf.printHelp(Pherf, options);
+System.exit(1
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/41#discussion_r26243599
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/jmx/MonitorManager.java ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.pherf.jmx;
+
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.exception.FileLoaderRuntimeException;
+import org.apache.phoenix.pherf.jmx.monitors.Monitor;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+import org.apache.phoenix.pherf.result.impl.CSVResultHandler;
+import org.apache.phoenix.pherf.result.Result;
+import org.apache.phoenix.pherf.result.ResultHandler;
+import org.apache.phoenix.util.DateUtil;
+
+import javax.management.*;
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.util.*;
+
+/**
+ * This class starts JMX stats for the configured monitors. Monitors
should be configured in MonitorDetails Enum.
+ * Each stat implements {@link
org.apache.phoenix.pherf.jmx.monitors.Monitor}.
+ *
+ * For the duration of any Pherf run, when the configured {@link
org.apache.phoenix.pherf.PherfConstants#MONITOR_FREQUENCY}
+ * is reached a snapshot of each monitor is taken and dumped out to a log
file.
+ */
+public class MonitorManager implements Runnable {
--- End diff --
Currently there is only one Monitor, but it's possible that we may want
different managers at some point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/41#discussion_r26243827
--- Diff:
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/jmx/MonitorManager.java ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.pherf.jmx;
+
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.exception.FileLoaderRuntimeException;
+import org.apache.phoenix.pherf.jmx.monitors.Monitor;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+import org.apache.phoenix.pherf.result.impl.CSVResultHandler;
+import org.apache.phoenix.pherf.result.Result;
+import org.apache.phoenix.pherf.result.ResultHandler;
+import org.apache.phoenix.util.DateUtil;
+
+import javax.management.*;
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.util.*;
+
+/**
+ * This class starts JMX stats for the configured monitors. Monitors
should be configured in MonitorDetails Enum.
+ * Each stat implements {@link
org.apache.phoenix.pherf.jmx.monitors.Monitor}.
+ *
+ * For the duration of any Pherf run, when the configured {@link
org.apache.phoenix.pherf.PherfConstants#MONITOR_FREQUENCY}
+ * is reached a snapshot of each monitor is taken and dumped out to a log
file.
+ */
+public class MonitorManager implements Runnable {
+// List of MonitorDetails for all the running monitors.
+private static final ListMonitorDetails MONITOR_DETAILS_LIST;
+private final ResultHandler resultHandler;
+private final long monitorFrequency;
+private volatile long rowCount;
+private volatile boolean shouldStop = false;
+private volatile boolean isRunning = false;
+
+static {
+// TODO Move this out to config. Possible use Guice and use IOC to
inject it in.
+MONITOR_DETAILS_LIST = Arrays.asList(MonitorDetails.values());
+}
+
+public MonitorManager() throws Exception {
+this(PherfConstants.MONITOR_FREQUENCY);
+}
+
+/**
+ *
+ * @param monitorFrequency Frequency at which monitor stats are
written to a log file.
+ * @throws Exception
+ */
+public MonitorManager(long monitorFrequency) throws Exception {
+this.monitorFrequency = monitorFrequency;
+MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
+
+// Register all the monitors to JMX
+for (MonitorDetails monitorDetails : MONITOR_DETAILS_LIST) {
+StandardMBean bean = new
StandardMBean(monitorDetails.getMonitor(), Monitor.class);
+ObjectName monitorThreadStatName = new
ObjectName(monitorDetails.toString());
+try {
+mbs.registerMBean(bean, monitorThreadStatName);
+} catch (InstanceAlreadyExistsException e) {
+mbs.unregisterMBean(monitorThreadStatName);
+mbs.registerMBean(bean, monitorThreadStatName);
+}
+}
+this.rowCount = 0;
+this.resultHandler = new
CSVResultHandler(PherfConstants.MONITOR_FILE_NAME, ResultFileDetails.CSV);
+}
+
+@Override
+public void run() {
+try {
+while (!shouldStop()) {
+isRunning = true;
+List rowValues = new ArrayListString();
+synchronized (resultHandler) {
+for (MonitorDetails monitorDetails :
MONITOR_DETAILS_LIST) {
+rowValues.clear();
+try {
+StandardMBean bean = new
StandardMBean(monitorDetails.getMonitor(), Monitor.class);
+
+Calendar calendar = new GregorianCalendar();
+rowValues.add(monitorDetails);
+
+rowValues.add
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/41#discussion_r26239779
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java
---
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.pherf;
+
+import org.apache.phoenix.pherf.configuration.XMLConfigParser;
+import org.apache.phoenix.pherf.schema.SchemaReader;
+import org.apache.phoenix.pherf.util.PhoenixUtil;
+import org.apache.phoenix.pherf.util.ResourceList;
+import org.apache.phoenix.pherf.workload.WorkloadExecutor;
+
+import org.apache.commons.cli.*;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.file.Path;
+import java.util.Collection;
+import java.util.Properties;
+
+public class Pherf {
+private static final Logger logger =
LoggerFactory.getLogger(Pherf.class);
+private static final Options options = new Options();
+
+static {
+//options.addOption(r, release, true, Required option.
Release that you want to test. 190, 192, ...);
--- End diff --
yup and done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: working additions of pherf to phoenix as a p...
GitHub user codymarcel opened a pull request: https://github.com/apache/phoenix/pull/41 working additions of pherf to phoenix as a project module mvn clean package builds everything correctly. All tests pass. You can merge this pull request into a Git repository by running: $ git pull https://github.com/codymarcel/phoenix pherf Alternatively you can review and apply these changes as the patch at: https://github.com/apache/phoenix/pull/41.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #41 commit b56edd2d3de18d37ce9a8a5220e40ab661d86f21 Author: cmarcel cmar...@salesforce.com Date: 2015-03-09T22:59:22Z working additions of pherf to phoenix as a project module --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
