[GitHub] chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger
chrismattmann commented on a change in pull request #131: Changed the map and
reduce function to work with XmlRpcWorkflowManger
URL: https://github.com/apache/drat/pull/131#discussion_r194242488
##
File path: proteus/src/main/java/backend/ProcessDratWrapper.java
##
@@ -128,17 +157,43 @@ private synchronized void solrIndex() throws
InstantiationException, SolrServerE
sIndexer.commit();
sIndexer.optimize();
idl.logInfo("Completed",null);
-
}
@Override
- public void map() throws IOException, DratWrapperException {
-simpleDratExec(MAP_CMD);
+ public void map() {
+setStatus(MAP_CMD);
+DratLog mapLog = new DratLog("MAPPING");
+WorkflowRestResource restResource = new WorkflowRestResource();
+DynamicWorkflowRequestWrapper requestBody = new
DynamicWorkflowRequestWrapper();
+requestBody.taskIds = new ArrayList<>();
+requestBody.taskIds.add("urn:drat:MimePartitioner");
+LOG.info("STARTING MAPPING");
+mapLog.logInfo("STARTING", " (dynamic workflow with task
urn:drat:MimePartitioner");
+String resp = restResource.performDynamicWorkFlow(requestBody);
+if(resp.equals("OK")) {
+mapLog.logInfo("STARTED SUCCESSFULLY, urn:drat:MimePartitioner dynamic
workflow");
+}else {
+mapLog.logSevere("FAILED", "Dynamic workflow starting failed "+resp);
+}
}
@Override
- public void reduce() throws IOException, DratWrapperException {
-simpleDratExec(REDUCE_CMD);
+ public void reduce() throws IOException {
+setStatus(REDUCE_CMD);
+DratLog mapLog = new DratLog("REDUCING");
+WorkflowRestResource restResource = new WorkflowRestResource();
+DynamicWorkflowRequestWrapper requestBody = new
DynamicWorkflowRequestWrapper();
+requestBody.taskIds = new ArrayList<>();
+requestBody.taskIds.add("urn:drat:RatAggregator");
Review comment:
this should be a static final String key like REDUCE_TASK at the top.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger
chrismattmann commented on a change in pull request #131: Changed the map and
reduce function to work with XmlRpcWorkflowManger
URL: https://github.com/apache/drat/pull/131#discussion_r194242484
##
File path: proteus/src/main/java/backend/ProcessDratWrapper.java
##
@@ -128,17 +157,43 @@ private synchronized void solrIndex() throws
InstantiationException, SolrServerE
sIndexer.commit();
sIndexer.optimize();
idl.logInfo("Completed",null);
-
}
@Override
- public void map() throws IOException, DratWrapperException {
-simpleDratExec(MAP_CMD);
+ public void map() {
+setStatus(MAP_CMD);
+DratLog mapLog = new DratLog("MAPPING");
+WorkflowRestResource restResource = new WorkflowRestResource();
+DynamicWorkflowRequestWrapper requestBody = new
DynamicWorkflowRequestWrapper();
+requestBody.taskIds = new ArrayList<>();
+requestBody.taskIds.add("urn:drat:MimePartitioner");
Review comment:
create a static final key for this at the top (like MAPPER_TASK from before).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger
chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger URL: https://github.com/apache/drat/pull/131#discussion_r193808064 ## File path: proteus/src/main/java/drat/proteus/workflow/rest/WorkflowRestResource.java ## @@ -0,0 +1,55 @@ +package drat.proteus.workflow.rest; + + +import java.io.IOException; +import java.util.logging.Logger; + + +import org.apache.oodt.cas.metadata.Metadata; +import org.apache.oodt.cas.metadata.util.PathUtils; +import org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient; +import org.apache.oodt.pcs.util.WorkflowManagerUtils; +import org.wicketstuff.rest.annotations.MethodMapping; +import org.wicketstuff.rest.annotations.parameters.RequestBody; +import org.wicketstuff.rest.contenthandling.json.webserialdeserial.GsonWebSerialDeserial; +import org.wicketstuff.rest.resource.AbstractRestResource; +import org.wicketstuff.rest.utils.http.HttpMethod; + +import backend.FileConstants; + Review comment: add simple class docs This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger
chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger URL: https://github.com/apache/drat/pull/131#discussion_r193806657 ## File path: proteus/src/main/java/backend/ProcessDratWrapper.java ## @@ -72,7 +75,8 @@ private static final String RESET_CMD = "reset"; private static final String STATUS_IDLE = "idle"; - private static final String MAPPER_TASK = "urn:drat:RatCodeAudit"; +// private static final String MAPPER_TASK = "urn:drat:RatCodeAudit"; Review comment: I don't understand this. You use this exact string on L182 and yet you comment it out here so you can change the name of this static final String to something you can log. Why not simply just leave this KEY the way it is, and then just log the simple name (RatCodeAudit)? These keys are usually reused, and I think you have more chance to reuse the key the way it is than by creating a new key. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger
chrismattmann commented on a change in pull request #131: Changed the map and
reduce function to work with XmlRpcWorkflowManger
URL: https://github.com/apache/drat/pull/131#discussion_r193807586
##
File path: proteus/src/main/java/backend/ProcessDratWrapper.java
##
@@ -310,36 +391,65 @@ private boolean mapsStillRunning() throws Exception {
return items;
}
- @VisibleForTesting
- protected boolean stillRunning(List items) {
-List mapperItems = filterMappers(items);
-LOG.info("Checking mappers: inspecting ["
-+ String.valueOf(mapperItems.size()) + "] mappers.");
-for (WorkflowItem mapperItem : mapperItems) {
- if (isRunning(mapperItem.getStatus())) {
-LOG.info("Mapper: [" + mapperItem.getId() + "] still running.");
-return true;
+ protected boolean stillRunning(List instances){
+ List mapperInstances = filterMappers(instances);
+ LOG.info("Checking mappers: inspecting ["
+ + String.valueOf(mapperInstances.size()) + "] mappers.");
+ for(WorkflowInstance mapperInstance:mapperInstances){
+ if(isRunning(mapperInstance.getState().getName())){
+ LOG.info("Mapper: [" + mapperInstance.getId() + "] still
running.");
+ return true;
+ }
}
-}
+ return false;
-return false;
}
- @VisibleForTesting
- protected List filterMappers(List items) {
-List mappers = new ArrayList();
-if (items != null && items.size() > 0) {
- for (WorkflowItem item : items) {
-if (item.getCurrentTask().equals(MAPPER_TASK)) {
- LOG.info("Adding mapper: [" + item.getCurrentTask() + "]");
- mappers.add(item);
-} else {
- LOG.info("Filtering task: [" + item.getCurrentTask() + "]");
-}
+// @VisibleForTesting
Review comment:
remove all of this if it's commented out
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger
chrismattmann commented on a change in pull request #131: Changed the map and
reduce function to work with XmlRpcWorkflowManger
URL: https://github.com/apache/drat/pull/131#discussion_r193807320
##
File path: proteus/src/main/java/backend/ProcessDratWrapper.java
##
@@ -257,16 +330,24 @@ private synchronized boolean hasAggregateRatLog() {
}
private boolean mapsStillRunning() throws Exception {
-String args[] = { FileConstants.WORKFLOW_PATH, "--url",
-"http://localhost:9001";, "--operation", "--getWorkflowInsts" };
-String cmd = Joiner.on(" ").join(args);
-LOG.info("Maps Still Running: Executing: " + cmd);
-String output = execToString(cmd);
-LOG.info("Output from maps still running: " + output);
-List items = parseWorkflows(output);
-return stillRunning(items);
+//String args[] = { FileConstants.WORKFLOW_PATH, "--url",
Review comment:
remove all of this
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger
chrismattmann commented on a change in pull request #131: Changed the map and
reduce function to work with XmlRpcWorkflowManger
URL: https://github.com/apache/drat/pull/131#discussion_r193806972
##
File path: proteus/src/main/java/backend/ProcessDratWrapper.java
##
@@ -111,15 +115,51 @@ public synchronized void setStatus(String status) {
@Override
public void crawl() throws Exception {
-simpleDratExec(CRAWL_CMD, this.path);
+//simpleDratExec(CRAWL_CMD, this.path);
+ crawlTemp();
+ }
+
+
+ public void crawlTemp() throws Exception{
+DratLog crawlLog = new DratLog("CRAWLING");
+try{
+ setStatus(CRAWL_CMD);
+
+ crawlLog.logInfo("Configuring");
+ String beanRepo =
System.getProperty("org.apache.oodt.cas.crawl.bean.repo",
+ FileConstants.CRAWLER_CONFIG);
+ String crawlerId = "MetExtractorProductCrawler";
+ System.setProperty("DRAT_EXCLUDE","");
+ FileSystemXmlApplicationContext appContext = new
FileSystemXmlApplicationContext("file:"+beanRepo);
+
+MetExtractorProductCrawler crawler = new MetExtractorProductCrawler();
+crawler.setApplicationContext(appContext);
+crawler.setId(crawlerId);
+
crawler.setMetExtractor("org.apache.oodt.cas.metadata.extractors.CopyAndRewriteExtractor");
+crawler.setMetExtractorConfig(FileConstants.MET_EXT_CONFIG_PATH);
+crawler.setFilemgrUrl(FileConstants.FILEMGR_URL);
+
crawler.setClientTransferer("org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferFactory");
+crawler.setPreCondIds(Arrays.asList("RegExExcludeComparator"));
+crawler.setProductPath(this.path);
+crawlLog.logInfo("STARTING ",null);
+crawler.crawl();
+crawlLog.logInfo("COMPLETED",null);
+ }catch (Exception ex) {
+// LOG.severe(ex.getLocalizedMessage());
Review comment:
if it's commented out, remove it
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger
chrismattmann commented on a change in pull request #131: Changed the map and
reduce function to work with XmlRpcWorkflowManger
URL: https://github.com/apache/drat/pull/131#discussion_r193807770
##
File path: proteus/src/main/java/backend/ProcessDratWrapper.java
##
@@ -446,39 +556,39 @@ private synchronized void wipeSolrCore(String coreName) {
+ e.getLocalizedMessage());
}
}
-
+
private class DratLog{
- private static final String MODULE = "DRAT_LOG";
+ private static final String MODULE = "DRAT_LOG";
long startTime =0;
private long lastActionTime=-1L;
private long timeDiff =-1L;
private ZonedDateTime zdt;
private String action;
public DratLog(String action) {
this.action = action;
-
+
}
-
+
private void logWarning(String status,String desc) {
LOG.warning(getMsg(status,desc));
}
-
+
private void logWarning(String desc) {
LOG.warning(MODULE+" : "+desc);
}
-
+
private void logInfo(String status,String desc) {
LOG.info(getMsg(status,desc));
}
-
+
private void logInfo(String desc) {
LOG.info(MODULE+" : "+desc);
}
-
+
private void logSevere(String status,String desc) {
LOG.fine(getMsg(status,desc));
}
-
Review comment:
a bunch of extraneous white space and newline changes. Don't couple these to
your actual PR. They are extraneous.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger
chrismattmann commented on a change in pull request #131: Changed the map and
reduce function to work with XmlRpcWorkflowManger
URL: https://github.com/apache/drat/pull/131#discussion_r193807228
##
File path: proteus/src/main/java/backend/ProcessDratWrapper.java
##
@@ -128,17 +168,47 @@ private synchronized void solrIndex() throws
InstantiationException, SolrServerE
sIndexer.commit();
sIndexer.optimize();
idl.logInfo("Completed",null);
-
+
}
@Override
public void map() throws IOException, DratWrapperException {
-simpleDratExec(MAP_CMD);
+//simpleDratExec(MAP_CMD);
+setStatus(MAP_CMD);
+DratLog mapLog = new DratLog("MAPPING");
+WorkflowRestResource restResource = new WorkflowRestResource();
+DynamicWorkflowRequestWrapper requestBody = new
DynamicWorkflowRequestWrapper();
+requestBody.taskIds = new ArrayList<>();
+requestBody.taskIds.add("urn:drat:MimePartitioner");
+LOG.info("STARTING MAPPING");
+mapLog.logInfo("STARTING", " (dynamic workflow with task
urn:drat:MimePartitioner");
+String resp = (String)restResource.performDynamicWorkFlow(requestBody);
+if(resp.equals("OK")) {
+mapLog.logInfo("STARTED SUCCESSFULLY, urn:drat:MimePartitioner dynamic
workflow");
+}else {
+mapLog.logSevere("FAILED", "Dynamic workflow starting failed "+resp);
+}
}
+
@Override
public void reduce() throws IOException, DratWrapperException {
-simpleDratExec(REDUCE_CMD);
+//simpleDratExec(REDUCE_CMD);
Review comment:
remove it
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger
chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger URL: https://github.com/apache/drat/pull/131#discussion_r193807960 ## File path: proteus/src/main/java/drat/proteus/workflow/rest/DynamicWorkflowRequestWrapper.java ## @@ -0,0 +1,7 @@ +package drat.proteus.workflow.rest; + +import java.util.List; + Review comment: add some simple class docs. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger
chrismattmann commented on a change in pull request #131: Changed the map and
reduce function to work with XmlRpcWorkflowManger
URL: https://github.com/apache/drat/pull/131#discussion_r193807542
##
File path: proteus/src/main/java/backend/ProcessDratWrapper.java
##
@@ -310,36 +391,65 @@ private boolean mapsStillRunning() throws Exception {
return items;
}
- @VisibleForTesting
- protected boolean stillRunning(List items) {
-List mapperItems = filterMappers(items);
-LOG.info("Checking mappers: inspecting ["
-+ String.valueOf(mapperItems.size()) + "] mappers.");
-for (WorkflowItem mapperItem : mapperItems) {
- if (isRunning(mapperItem.getStatus())) {
-LOG.info("Mapper: [" + mapperItem.getId() + "] still running.");
-return true;
+ protected boolean stillRunning(List instances){
Review comment:
I think here you are reformatting white space. Don't do this. It makes it
hard to see if this is a real change or not.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger
chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger URL: https://github.com/apache/drat/pull/131#discussion_r193807922 ## File path: proteus/src/main/java/drat/proteus/workflow/rest/DynamicWorkflowRequestWrapper.java ## @@ -0,0 +1,7 @@ +package drat.proteus.workflow.rest; Review comment: need to add Apache license header from other files here. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger
chrismattmann commented on a change in pull request #131: Changed the map and
reduce function to work with XmlRpcWorkflowManger
URL: https://github.com/apache/drat/pull/131#discussion_r193806844
##
File path: proteus/src/main/java/backend/ProcessDratWrapper.java
##
@@ -111,15 +115,51 @@ public synchronized void setStatus(String status) {
@Override
public void crawl() throws Exception {
-simpleDratExec(CRAWL_CMD, this.path);
+//simpleDratExec(CRAWL_CMD, this.path);
+ crawlTemp();
+ }
+
+
+ public void crawlTemp() throws Exception{
Review comment:
this method name doesn't make sense to me. I would name it performCrawl().
The crawl() DRAT API method on L117 is fine - and then this actually _performs_
the crawl.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger
chrismattmann commented on a change in pull request #131: Changed the map and
reduce function to work with XmlRpcWorkflowManger
URL: https://github.com/apache/drat/pull/131#discussion_r193807091
##
File path: proteus/src/main/java/backend/ProcessDratWrapper.java
##
@@ -128,17 +168,47 @@ private synchronized void solrIndex() throws
InstantiationException, SolrServerE
sIndexer.commit();
sIndexer.optimize();
idl.logInfo("Completed",null);
-
+
}
@Override
public void map() throws IOException, DratWrapperException {
-simpleDratExec(MAP_CMD);
+//simpleDratExec(MAP_CMD);
Review comment:
remove it
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger
chrismattmann commented on a change in pull request #131: Changed the map and reduce function to work with XmlRpcWorkflowManger URL: https://github.com/apache/drat/pull/131#discussion_r193808013 ## File path: proteus/src/main/java/drat/proteus/workflow/rest/WorkflowRestResource.java ## @@ -0,0 +1,55 @@ +package drat.proteus.workflow.rest; Review comment: Apache license header from other files here. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
