[
https://issues.apache.org/jira/browse/HDDS-1972?focusedWorklogId=296408&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296408
]
ASF GitHub Bot logged work on HDDS-1972:
----------------------------------------
Author: ASF GitHub Bot
Created on: 16/Aug/19 16:40
Start Date: 16/Aug/19 16:40
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on issue #1304: HDDS-1972.
Provide example ha proxy with multiple s3 servers back end.
URL: https://github.com/apache/hadoop/pull/1304#issuecomment-522073531
On my desktop I see tests are passing.
Below is the result.
```
HW13865:ozones3-haproxy bviswanadham$ ./test.sh
Removing network ozones3-haproxy_default
WARNING: Network ozones3-haproxy_default not found.
Creating network "ozones3-haproxy_default" with the default driver
Creating ozones3-haproxy_datanode_1 ... done
Creating ozones3-haproxy_datanode_2 ... done
Creating ozones3-haproxy_datanode_3 ... done
Creating ozones3-haproxy_scm_1 ... done
Creating ozones3-haproxy_om_1 ... done
Creating ozones3-haproxy_s3g_1 ... done
Creating ozones3-haproxy_s3g1_1 ... done
Creating ozones3-haproxy_s3g2_1 ... done
Creating ozones3-haproxy_s3g3_1 ... done
0 datanode is up and healthy (until now)
0 datanode is up and healthy (until now)
3 datanodes are up and registered to the scm
==============================================================================
ozones3-haproxy-basic :: Smoketest ozone cluster startup
==============================================================================
Check webui static resources | PASS
|
------------------------------------------------------------------------------
Start freon testing | PASS
|
------------------------------------------------------------------------------
ozones3-haproxy-basic :: Smoketest ozone cluster startup | PASS
|
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Output:
/tmp/smoketest/ozones3-haproxy/result/robot-ozones3-haproxy-ozones3-haproxy-basic-scm.xml
==============================================================================
ozones3-haproxy-s3
==============================================================================
ozones3-haproxy-s3.Awss3 :: S3 gateway test with aws cli
==============================================================================
File upload and directory list | PASS
|
------------------------------------------------------------------------------
ozones3-haproxy-s3.Awss3 :: S3 gateway test with aws cli | PASS
|
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Bucketcreate :: S3 gateway test with aws cli
==============================================================================
Create bucket which already exists | PASS
|
------------------------------------------------------------------------------
ozones3-haproxy-s3.Bucketcreate :: S3 gateway test with aws cli | PASS
|
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Buckethead :: S3 gateway test with aws cli
==============================================================================
Head Bucket not existent | PASS
|
------------------------------------------------------------------------------
ozones3-haproxy-s3.Buckethead :: S3 gateway test with aws cli | PASS
|
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Bucketlist :: S3 gateway test with aws cli
==============================================================================
List buckets | PASS
|
------------------------------------------------------------------------------
ozones3-haproxy-s3.Bucketlist :: S3 gateway test with aws cli | PASS
|
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.MultipartUpload :: S3 gateway test with aws cli
==============================================================================
Test Multipart Upload | PASS
|
------------------------------------------------------------------------------
Test Multipart Upload Complete | PASS
|
------------------------------------------------------------------------------
Test Multipart Upload Complete Entity too small | PASS
|
------------------------------------------------------------------------------
Test Multipart Upload Complete Invalid part | PASS
|
------------------------------------------------------------------------------
Test abort Multipart upload | PASS
|
------------------------------------------------------------------------------
Test abort Multipart upload with invalid uploadId | PASS
|
------------------------------------------------------------------------------
Upload part with Incorrect uploadID | PASS
|
------------------------------------------------------------------------------
Test list parts | PASS
|
------------------------------------------------------------------------------
Test Multipart Upload with the simplified aws s3 cp API | PASS
|
------------------------------------------------------------------------------
ozones3-haproxy-s3.MultipartUpload :: S3 gateway test with aws cli | PASS
|
9 critical tests, 9 passed, 0 failed
9 tests total, 9 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Objectcopy :: S3 gateway test with aws cli
==============================================================================
Copy Object Happy Scenario | PASS
|
------------------------------------------------------------------------------
Copy Object Where Bucket is not available | PASS
|
------------------------------------------------------------------------------
Copy Object Where both source and dest are same with change to sto... | PASS
|
------------------------------------------------------------------------------
Copy Object Where Key not available | PASS
|
------------------------------------------------------------------------------
ozones3-haproxy-s3.Objectcopy :: S3 gateway test with aws cli | PASS
|
4 critical tests, 4 passed, 0 failed
4 tests total, 4 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Objectdelete :: S3 gateway test with aws cli
==============================================================================
Delete file with s3api | PASS
|
------------------------------------------------------------------------------
Delete file with s3api, file doesn't exist | PASS
|
------------------------------------------------------------------------------
Delete dir with s3api | PASS
|
------------------------------------------------------------------------------
Delete file with s3api, file doesn't exist, prefix of a real file | PASS
|
------------------------------------------------------------------------------
Delete file with s3api, bucket doesn't exist | PASS
|
------------------------------------------------------------------------------
ozones3-haproxy-s3.Objectdelete :: S3 gateway test with aws cli | PASS
|
5 critical tests, 5 passed, 0 failed
5 tests total, 5 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Objectmultidelete :: S3 gateway test with aws cli
==============================================================================
Delete file with multi delete | PASS
|
------------------------------------------------------------------------------
ozones3-haproxy-s3.Objectmultidelete :: S3 gateway test with aws cli | PASS
|
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Objectputget :: S3 gateway test with aws cli
==============================================================================
Put object to s3 | PASS
|
------------------------------------------------------------------------------
Get object from s3 | PASS
|
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset | PASS
|
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset(start off... | PASS
|
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset(end offse... | PASS
|
------------------------------------------------------------------------------
Get Partial object from s3 with only start offset | PASS
|
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset which are... | PASS
|
------------------------------------------------------------------------------
Get Partial object from s3 to get last n bytes | PASS
|
------------------------------------------------------------------------------
Incorrect values for end and start offset | PASS
|
------------------------------------------------------------------------------
Zero byte file | PASS
|
------------------------------------------------------------------------------
ozones3-haproxy-s3.Objectputget :: S3 gateway test with aws cli | PASS
|
10 critical tests, 10 passed, 0 failed
10 tests total, 10 passed, 0 failed
==============================================================================
ozones3-haproxy-s3.Webui :: S3 gateway web ui test
==============================================================================
S3 Gateway Web UI | PASS
|
------------------------------------------------------------------------------
ozones3-haproxy-s3.Webui :: S3 gateway web ui test | PASS
|
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
ozones3-haproxy-s3 | PASS
|
34 critical tests, 34 passed, 0 failed
34 tests total, 34 passed, 0 failed
==============================================================================
Output:
/tmp/smoketest/ozones3-haproxy/result/robot-ozones3-haproxy-ozones3-haproxy-s3-scm.xml
Stopping ozones3-haproxy_datanode_2 ... done
Stopping ozones3-haproxy_s3g3_1 ... done
Stopping ozones3-haproxy_om_1 ... done
Stopping ozones3-haproxy_s3g1_1 ... done
Stopping ozones3-haproxy_s3g2_1 ... done
Stopping ozones3-haproxy_s3g_1 ... done
Stopping ozones3-haproxy_datanode_1 ... done
Stopping ozones3-haproxy_datanode_3 ... done
Stopping ozones3-haproxy_scm_1 ... done
Removing ozones3-haproxy_datanode_2 ... done
Removing ozones3-haproxy_s3g3_1 ... done
Removing ozones3-haproxy_om_1 ... done
Removing ozones3-haproxy_s3g1_1 ... done
Removing ozones3-haproxy_s3g2_1 ... done
Removing ozones3-haproxy_s3g_1 ... done
Removing ozones3-haproxy_datanode_1 ... done
Removing ozones3-haproxy_datanode_3 ... done
Removing ozones3-haproxy_scm_1 ... done
Removing network ozones3-haproxy_default
Log:
/Users/bviswanadham/workspace/myozone/hadoop/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozones3-haproxy/result/log.html
Report:
/Users/bviswanadham/workspace/myozone/hadoop/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozones3-haproxy/result/report.html
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 296408)
Time Spent: 1.5h (was: 1h 20m)
> Provide example ha proxy with multiple s3 servers back end.
> -----------------------------------------------------------
>
> Key: HDDS-1972
> URL: https://issues.apache.org/jira/browse/HDDS-1972
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
> Reporter: Bharat Viswanadham
> Assignee: Bharat Viswanadham
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> In this Jira, we shall provide docker-compose files where we start 3 s3
> gateway servers, and ha-proxy is used to load balance these S3 Gateway
> Servers.
>
> In this Jira, all are proxy configurations are hardcoded, we can make
> improvements to scale and automatically configure with environment variables
> as a future improvement. This is just a starter example.
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
