Re: Problem on druid-s3-extensions
I greatly appreciate your guidance. Now all the tests pass well! Thanks, Dongjin On Thu, Jul 26, 2018 at 3:27 AM Jihoon Son wrote: > Hi Dongjin, > > as Charles said, you should set the region for aws sdk. Here is a snippet > of our Travis configuration. > > > # other modules test > >- sudo: false > > env: > >- NAME="other modules test" > >- AWS_REGION=us-east-1 # set a aws region for unit tests > > install: echo "MAVEN_OPTS='-Xmx3000m'" > ~/.mavenrc && mvn install > -q -ff -DskipTests -B > > before_script: > >- unset _JAVA_OPTIONS > > script: echo "MAVEN_OPTS='-Xmx512m'" > ~/.mavenrc && mvn test -B > -Pparallel-test -Dmaven.fork.count=2 -pl '!processing,!server' > > Jihoon > > On Wed, Jul 25, 2018 at 8:29 AM Charles Allen > wrote: > > > IMHO the best way to do it would be to setup the test so that it sets the > > region just for the test, and let production systems use whichever region > > provider fits the way they typically do region configs. That way people > > running the tests don't have to worry about anything regarding how to > setup > > a region, and people deploying cloud services don't have surprise > behavior > > for Druid compared to other usages of the java sdk. > > > > On Tue, Jul 24, 2018 at 9:48 PM Dongjin Lee wrote: > > > > > Hello. I encountered a problem building druid. *In short, > > > `TestAWSCredentialsProvider` fails like the following*: > > > > > > ``` > > > mvn -pl extensions-core/s3-extensions test > > > > > > ... > > > > > > > > > > > > testWithFileSessionCredentials(io.druid.storage.s3.TestAWSCredentialsProvider) > > > Time elapsed: 6.615 sec <<< ERROR! > > > com.amazonaws.SdkClientException: Unable to find a region via the > region > > > provider chain. Must provide an explicit region in the builder or setup > > > environment to supply a region. > > > at > > > > > > > > > io.druid.storage.s3.TestAWSCredentialsProvider.testWithFileSessionCredentials(TestAWSCredentialsProvider.java:98) > > > > > > testWithFixedAWSKeys(io.druid.storage.s3.TestAWSCredentialsProvider) > > Time > > > elapsed: 4.345 sec <<< ERROR! > > > com.amazonaws.SdkClientException: Unable to find a region via the > region > > > provider chain. Must provide an explicit region in the builder or setup > > > environment to supply a region. > > > at > > > > > > > > > io.druid.storage.s3.TestAWSCredentialsProvider.testWithFixedAWSKeys(TestAWSCredentialsProvider.java:67) > > > ``` > > > > > > After digging the code, I found following: > > > > > > 1. `S3StorageDruidModule` does not provide a way to explictily set > > > `Region`, but detects the region automatically with its default > > > credential/region provider chain. > `S3StorageDruidModule#getAmazonS3Client > > > -> AmazonS3Client#builder -> AmazonS3ClientBuilder#standard;` > > > 2. The default region provider chain[^1][^2] tries to determine region > in > > > following orders:[^1][^2] > > > a. Environment Variable[^5] > > > b. JVM property[^6] > > > c. AWS Configuration[^7] > > > d. EC2 instance metadata service[^8] > > > > > > If all of the above fails, it throws an Exception. > > > > > > *In short, there is a possibility that `S3StorageDruidModule` can't > > > determine the Region. It would be better to provide a way to set Region > > > explicitly.* > > > > > > How do you think? > > > > > > Best, > > > Dongjin > > > > > > [^1]: > > > > > > > > > https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/DefaultAwsRegionProviderChain.java > > > [^2]: > > > > > > > > > https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsRegionProviderChain.java > > > [^3]: > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.aws.amazon.com_sdk-2Dfor-2Djava_v2_developer-2Dguide_java-2Ddg-2Dregion-2Dselection.html=DwIBaQ=ncDTmphkJTvjIDPh0hpF_w=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI=Ro8j9Jj9lCbFbc0aYDNjwtZuD2a_9fPaYw1lOp7ZQeQ= > > > [^4]: The example client in documentation is `AmazonEC2ClientBuilder` > but > > > it also applies to the other clients like `AmazonS3ClientBuilder`. > > > [^5]: > > > > > > > > > https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsEnvVarOverrideRegionProvider.java > > > [^6]: > > > > > > > > > https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsSystemPropertyRegionProvider.java > > > [^7]: > > > > > > > > > https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsProfileRegionProvider.java > > > [^8]: > > > > > > > > > https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/InstanceMetadataRegionProvider.java > > > > > > -- > > > *Dongjin Lee* > > > > > > *A hitchhiker in the mathematical world.* > > > > > > *github: < > > > > > >
Re: Build failure on 0.13.SNAPSHOT
Oh, it seems like apache mailing server ate the attached image as Gian said; here is the table: total usedfree shared buff/cacheavailable Mem: 7704 6147 122 161 1435 1123 Swap: 4095 16672428 Frankly, this result is not accurate; since I closed all the other applications except the build process, it shows the memory usage of `system processes + build process` - Jihoon is right, if the test process uses 8gb of memory as I said first, the test can't be run on Travis. The reason would be the other something as Jihoon said - I am working on a desktop version of Ubuntu with the different implementation of JDK (openjdk 8) to Travis configuration. Now I guess these differences caused some side effects. Since this is not a problem with the code, you don't need a fix. Thanks again for your kind guidance. Thanks, Dongjin On Thu, Jul 26, 2018 at 3:24 AM Jihoon Son wrote: > @Dongjin, thank you for looking into this. I got some follow-up questions. > > > Druid uses above 8gb of memory for testing. > > How did you get this result? Did you use some tools to get the memory usage > of only the maven process for Druid tests? When I checked last time using > JMC, I could see that running all unit tests of each Travis Job usually use > up to 512 MB of memory (you can find our Travis Job configuration here: > https://github.com/apache/incubator-druid/blob/master/.travis.yml). That's > why I set the memory limit to 512 MB for our Travis CI. Maybe we need to > check again and raise the memory limit to avoid frequent OOM failures in > groupBy query tests. > > Also, if it does use 8 GB of memory, I suspect it's a sort of bug or > something. I don't think even our integration tests don't use such a large > memory which run all Druid processes, ZooKeeper, and Kafka. So, we should > figure out which tests use such a lot of memory and fix it instead of > fixing documents. > > > Fwiw I think 8GB should be enough memory with the right settings, since > our Travis CI build environment only has about 7.5GB. > > @Gian, the memory of the Travis instance is shared by all processes > including OS. AFAIK, the OS requires about 1 GB of memory. It means that > our unit tests can't be run on Travis if it needs 8 GB of memory. > > Best, > Jihoon > > On Wed, Jul 25, 2018 at 10:01 AM Charles Allen > wrote: > > > OOME seems to be showing up in some of the Travis testing as well for > group > > by related stuff. Unsure what's going on there. > > > > On Tue, Jul 24, 2018 at 9:46 PM Dongjin Lee wrote: > > > > > After some experiments, I figured out the following: > > > > > > 1. Druid uses above 8gb of memory for testing. (building-druid.png) > > > 2. With 8gb(physical)+4gb(swap) of memory, the test succeeds regardless > > of > > > maven version (3.3.9, 3.5.2, 3.5.4) or MAVEN_OPTS. However, with > > > 8gb(physical)+2gb(swap) of memory[^1], some tests failed. The list of > > > failing tests differs between maven 3.3.9 and 3.5.2. > > > > > > In short, retaining sufficient memory solved the problem - *It seems > like > > > 12gb of memory is a recommended setting for building druid.* (I guess > > > lots of you are working with the MacBook Pro with 16gb RAM, right? In > > that > > > case, you must not have encountered this problem.) > > > > > > If you are okay, may I update the building documentation for the > newbies > > > like me? > > > > > > Thanks, > > > Dongjin > > > > > > +1. While building Druid, I found another problem. But this issue > should > > > be discussed in another thread. > > > > > > [^1]: You know, the other processes also occupy the memory. > > > > > > > > > On Tue, Jul 24, 2018 at 3:07 AM Jihoon Son > wrote: > > > > > >> I'm also using Maven 3.5.2 and not using any special configurations > for > > >> Maven, but I have never seen that error too. > > >> Most of our Travis jobs have been working with only 512 MB of direct > > >> memory. Only the 'strict compilation' Travis job requires 3 GB of > > memory. > > >> > > >> I think it's worthwhile to look into this more. Maybe we somehow use > > more > > >> memory when we run all tests by 'mvn install'. Maybe this relates to > the > > >> frequent transient failures of 'processing module test', one of our > > Travis > > >> jobs. > > >> > > >> Jihoon > > >> > > >> On Mon, Jul 23, 2018 at 9:32 AM Gian Merlino wrote: > > >> > > >> > Interesting. Fwiw, I am using Maven 3.5.2 for building Druid and it > > has > > >> > been working for for me. I don't think I"m using any special Maven > > >> > overrides (at least, I don't see anything interesting in my ~/.m2 > > >> directory > > >> > or in my environment variables). It might have to do with how much > > >> memory > > >> > our machines have? I do most of my builds on a Mac with 16GB RAM. > > Maybe > > >> try > > >> > checking .travis.yml in the druid repo. It sets -Xmx3000m for mvn > > >> install > > >> > commands, which might be needed for more low memory environments. > > >> > > > >>
CLA
Hi everyone, As I'm setting up for contribution, I have noticed that Druid's website directs contributors to sign a non-Apache CLA (See http://druid.io/community/cla.html). I believe this is outdated and contributors need instead to follow the Apache guidelines (For contributors: http://www.apache.org/licenses/#clas). Am I correct? Also, as a side note, I was wondering if anyone knows how would one present a submitted Individual CLA when opening a pull request (I couldn't find a clear answer to that on the ASF website). Thanks, Eyal Yurman. - To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
Re: Issue with documentation
Just a pull request is good enough. On Wed, Jul 25, 2018 at 4:23 PM Himanshu Pandey wrote: > Thanks Gian! Do I still need to create a new issue in GitHub for this ? or > just a pull request will suffice? > > > > *Thanks & Regards,* > *Himanshu Pandey* > > > On Wed, Jul 25, 2018 at 3:39 PM, Gian Merlino wrote: > > > Hey Himanshu, > > > > The docs are kept in the main Druid repo and released to the site when we > > do Druid releases. The source for the doc you mentioned is here: > > https://github.com/apache/incubator-druid/blob/master/ > > docs/content/tutorials/quickstart.md > > > > If you still see the error in master, please raise a pull request to fix > it > > -- and thank you! > > > > On Wed, Jul 25, 2018 at 3:30 PM Himanshu Pandey < > > himanshu.pande...@gmail.com> > > wrote: > > > > > Hi Team, > > > > > > I am new to druid and would like to contribute to the project. > > > > > > While setting it up on my local, I found one minor documentation issue > > > here: > > > > > > http://druid.io/docs/0.12.1/tutorials/quickstart.html > > > > > > How can I proceed with creating the issue and possibly fixing it? > > > > > > *Thanks & Regards,* > > > *Himanshu Pandey* > > > *Cell: +1 (408) 644 - 8765* > > > > > >
