Hi Tom
Thanks for the input, WHIRR-325 is just what the doctor ordered. I'll
extend it with nodeId if it applies cleanly, or simply use IP address as nodeId.
Regarding the benchmark, I had already seen the Teramark stuff, which
was my initial inspiration. I wanted something a bit more generic that would
start a server cluster (e.g. S4 or HBase) and a client cluster that would feed
data to the server cluster, later on I added initial data sync to the cloud,
client bench node data download, and log and metrics generation and extraction.
It might be a nice contrib after its done, we'll see.
Again, thanks for the feedback
Cheers
-david
On Oct 10, 2011, at 5:45 PM, Tom White wrote:
> Environment variables for the cluster name and IP address are being
> added in https://issues.apache.org/jira/browse/WHIRR-325. Does that
> help? You could extend that approach to add a node ID too.
>
> There's also some benchmark code in
> org.apache.whirr.service.hadoop.integration.benchmark, which might be
> useful for what you are doing.
>
> Cheers,
> Tom
>
> On Mon, Oct 10, 2011 at 10:44 AM, David Alves <[email protected]> wrote:
>> Hi All
>>
>> I'm about to finish writing a small benchmark framework built on top
>> of whirr (very simple, uploads data to the cloud, prepares client and server
>> clusters, runs clients against servers and uploads results to the cloud
>> before destroying all the clusters).
>> In the final stage I need to extract the metrics and log output of
>> every node in the cluster and put them in cloud storage (e.g., s3) but I'd
>> like to package them with the cluster name and node name such as:
>> [bench]-[cluster]-[node]-metrics.tar.gz.
>> I was implementing the packaging and uploading of these files as
>> statements, but I'm having trouble finding the [cluster] and [node] ids.
>> Are these env variables or something I can get when I run the ssh
>> commands? orf is there some form of knowing against which node a statement
>> is being executed (if I call a statement with
>> clusterController.runScriptOnNodesMathing(… stmt)).
>>
>> Cheers
>> -david
