chrishanKH opened a new issue, #10654:
URL: https://github.com/apache/pinot/issues/10654
I installed Pinot 0.12.0 on my AWS EKS cluster via `helm` (command below)
and attempting to use S3 for deep storage. I ran an ingestion job via Spark,
which completes successfully with the outputs in the right locations in S3, but
Pinot reports `BAD` segments.
Upon clicking on a segment's debug details through the Pinot UI, it shows
that `externalView` is `ERROR`. I couldn't find too much information on
`externalView` in the Pinot docs:
```
{
"segmentName": "$tableName_OFFLINE_100",
"serverState": {
"Server_pinot-server-0.pinot-server-headless.$namespace.svc.cluster.local_8098":
{
"idealState": "ONLINE",
"externalView": "ERROR",
"segmentSize": "0 bytes",
"consumerInfo": null,
"errorInfo": null
}
}
}
```
Upon inspecting the logs with `kubectl logs pinot-server-0 -n $myNamespace`
I noticed the error message `java.lang.IllegalStateException: PinotFS for
scheme: s3 has not been initialized`. Full exception here:
```
Caught exception while fetching segment from:
s3://$path_to_segments/$tableName_OFFLINE_296.tar.gz to:
/var/pinot/server/data/index/$tabletName_OFFLINE/tmp/tmp-$tableName_OFFLINE_296-518442be-5f6b-47ec-a672-e70ab6bd5bd2/$tableName_OFFLINE_296.tar.gz
java.lang.IllegalStateException: PinotFS for scheme: s3 has not been
initialized
at
org.apache.pinot.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.spi.filesystem.PinotFSFactory.create(PinotFSFactory.java:78)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher.fetchSegmentToLocalWithoutRetry(PinotFSSegmentFetcher.java:31)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.lambda$fetchSegmentToLocal$0(BaseSegmentFetcher.java:75)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:58)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:73)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocalInternal(SegmentFetcherFactory.java:157)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:151)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocalInternal(SegmentFetcherFactory.java:198)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(SegmentFetcherFactory.java:186)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.core.data.manager.BaseTableDataManager.downloadAndDecrypt(BaseTableDataManager.java:565)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegmentFromDeepStore(BaseTableDataManager.java:528)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegment(BaseTableDataManager.java:512)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.core.data.manager.BaseTableDataManager.addOrReplaceSegment(BaseTableDataManager.java:472)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOrReplaceSegment(HelixInstanceDataManager.java:482)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:166)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at jdk.internal.reflect.GeneratedMethodAccessor37.invoke(Unknown
Source) ~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Attempts exceeded when downloading segment: $tableName_OFFLINE_296 for
table: $tableName from:
s3://$pathToSegments/segments/$tableName_OFFLINE_296.tar.gz to:
/var/pinot/server/data/index/$tableName_OFFLINE/tmp/tmp-$tableName_OFFLINE_296-518442be-5f6b-47ec-a672-e70ab6bd5bd2/$tableName_v2_OFFLINE_296.tar.gz
Caught exception in state transition from OFFLINE -> ONLINE for resource:
$tableName_OFFLINE, partition: $tableName_OFFLINE_296
org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed
after 3 attempts
at
org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:73)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocalInternal(SegmentFetcherFactory.java:157)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:151)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocalInternal(SegmentFetcherFactory.java:198)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(SegmentFetcherFactory.java:186)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.core.data.manager.BaseTableDataManager.downloadAndDecrypt(BaseTableDataManager.java:565)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegmentFromDeepStore(BaseTableDataManager.java:528)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegment(BaseTableDataManager.java:512)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.core.data.manager.BaseTableDataManager.addOrReplaceSegment(BaseTableDataManager.java:472)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOrReplaceSegment(HelixInstanceDataManager.java:482)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:166)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at jdk.internal.reflect.GeneratedMethodAccessor37.invoke(Unknown
Source) ~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Exception while executing a state transition task $tableName_OFFLINE_296
java.lang.reflect.InvocationTargetException: null
at jdk.internal.reflect.GeneratedMethodAccessor37.invoke(Unknown
Source) ~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49)
[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.pinot.spi.utils.retry.AttemptsExceededException:
Operation failed after 3 attempts
at
org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:73)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocalInternal(SegmentFetcherFactory.java:157)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:151)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocalInternal(SegmentFetcherFactory.java:198)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(SegmentFetcherFactory.java:186)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.core.data.manager.BaseTableDataManager.downloadAndDecrypt(BaseTableDataManager.java:565)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegmentFromDeepStore(BaseTableDataManager.java:528)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegment(BaseTableDataManager.java:512)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.core.data.manager.BaseTableDataManager.addOrReplaceSegment(BaseTableDataManager.java:472)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOrReplaceSegment(HelixInstanceDataManager.java:482)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
at
org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:166)
~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
... 11 more
Default rollback method invoked on error. Error Code: ERROR
Message execution failed. msgId: ddba0761-acd8-41d3-b00e-401b958b045c,
errorMsg: java.lang.reflect.InvocationTargetException
Delete message ddba0761-acd8-41d3-b00e-401b958b045c from zk!
message finished: ddba0761-acd8-41d3-b00e-401b958b045c, took 2428
```
I read the documentation
[here](https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot)
and included the config via the `helm install command`:
```
helm upgrade pinot pinot/pinot -n $namespace \
--set cluster.name=pinot \
--set server.replicaCount=2 \
--set zookeeper.persistence.size=500G \
--set input.fs.className=org.apache.pinot.plugin.filesystem.S3PinotFS \
--set input.fs.prop.accessKey=$aws_access_key \
--set input.fs.prop.secretKey=$aws_secret_key \
--set input.fs.prop.region=$aws_region \
--set
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
\
--set pinot.controller.storage.factory.s3.region=$aws_region \
--set controller.data.dir=$s3_path \
--set controller.local.temp.dir=/tmp/pinot-dev-output \
--set controller.enable.split.commit=true \
--set
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
\
--set pinot.server.instance.enable.split.commit=true \
--set
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
\
--set pinot.server.storage.factory.s3.region=$aws_region \
--set pinot.server.storage.factory.s3.httpclient.maxConnections=50 \
--set pinot.server.storage.factory.s3.httpclient.socketTimeout=30s \
--set pinot.server.storage.factory.s3.httpclient.connectionTimeout=2s \
--set pinot.server.storage.factory.s3.httpclient.connectionTimeToLive=0s \
--set
pinot.server.storage.factory.s3.httpclient.connectionAcquisitionTimeout=10s \
--set
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
\
--set
pinot.minion.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
\
--set pinot.minion.storage.factory.s3.region=$aws_region \
--set
pinot.minion.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
\
--set pinot.controller.segment.fetcher.protocols="file\,http\,s3" \
--set pinot.minion.segment.fetcher.protocols="file\,http\,s3" \
--set pinot.server.segment.fetcher.protocols="file\,http\,s3"
```
Any guidance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
