[GitHub] drill pull request #1222: DRILL-6341: Fixed failing tests for mongodb storag...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/1222


---

[GitHub] drill pull request #1222: DRILL-6341: Fixed failing tests for mongodb storag...

[cgivre]

Github user cgivre commented on a diff in the pull request:

https://github.com/apache/drill/pull/1222#discussion_r184882305
  
--- Diff: 
contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/MongoTestSuit.java
 ---
@@ -128,42 +130,63 @@ private static void setup() throws Exception {
   createDbAndCollections(DATATYPE_DB, DATATYPE_COLLECTION, "_id");
 }
 
-private static IMongodConfig crateConfigServerConfig(int 
configServerPort,
-boolean flag) throws UnknownHostException, IOException {
-  IMongoCmdOptions cmdOptions = new 
MongoCmdOptionsBuilder().useNoJournal(false).verbose(false)
-  .build();
+private static IMongodConfig crateConfigServerConfig(int 
configServerPort) throws UnknownHostException, IOException {
+  IMongoCmdOptions cmdOptions = new MongoCmdOptionsBuilder()
+.useNoPrealloc(false)
+.useSmallFiles(false)
+.useNoJournal(false)
+.useStorageEngine(STORAGE_ENGINE)
+.verbose(false)
+.build();
+
+  Storage replication = new Storage(null, CONFIG_REPLICA_SET, 0);
 
   IMongodConfig mongodConfig = new MongodConfigBuilder()
-  .version(Version.Main.PRODUCTION)
+  .version(Version.Main.V3_4)
   .net(new Net(LOCALHOST, configServerPort, 
Network.localhostIsIPv6()))
-  .configServer(flag).cmdOptions(cmdOptions).build();
+  .replication(replication)
+  .shardServer(false)
+  .configServer(true).cmdOptions(cmdOptions).build();
--- End diff --

Successfully built Mongo storage plugin on my Mac.  LGTM +1


---

[GitHub] drill pull request #1222: DRILL-6341: Fixed failing tests for mongodb storag...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1222#discussion_r184881467
  
--- Diff: 
contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/MongoTestSuit.java
 ---
@@ -128,42 +130,63 @@ private static void setup() throws Exception {
   createDbAndCollections(DATATYPE_DB, DATATYPE_COLLECTION, "_id");
 }
 
-private static IMongodConfig crateConfigServerConfig(int 
configServerPort,
-boolean flag) throws UnknownHostException, IOException {
-  IMongoCmdOptions cmdOptions = new 
MongoCmdOptionsBuilder().useNoJournal(false).verbose(false)
-  .build();
+private static IMongodConfig crateConfigServerConfig(int 
configServerPort) throws UnknownHostException, IOException {
+  IMongoCmdOptions cmdOptions = new MongoCmdOptionsBuilder()
+.useNoPrealloc(false)
+.useSmallFiles(false)
+.useNoJournal(false)
+.useStorageEngine(STORAGE_ENGINE)
+.verbose(false)
+.build();
+
+  Storage replication = new Storage(null, CONFIG_REPLICA_SET, 0);
 
   IMongodConfig mongodConfig = new MongodConfigBuilder()
-  .version(Version.Main.PRODUCTION)
+  .version(Version.Main.V3_4)
   .net(new Net(LOCALHOST, configServerPort, 
Network.localhostIsIPv6()))
-  .configServer(flag).cmdOptions(cmdOptions).build();
+  .replication(replication)
+  .shardServer(false)
+  .configServer(true).cmdOptions(cmdOptions).build();
--- End diff --

Please move to the new lines `.cmdOptions(cmdOptions).build();`


---

[GitHub] drill pull request #1222: DRILL-6341: Fixed failing tests for mongodb storag...

[ilooner]

GitHub user ilooner opened a pull request:

https://github.com/apache/drill/pull/1222

DRILL-6341: Fixed failing tests for mongodb storage plugin

## Issue

The Mongodb storage plugin tests fail on osx 10.13.x because MongoDB 3.2 
itself has a bug on osx 10.13.x  . This is a known and documented issue here 
https://github.com/github/hub/issues/1405 . The issue occurs consistently on my 
laptop.

## Fix

 - The version of MongDB used for tests needs to be updated to 3.4. Why 3.4 
and not 3.6? There is some discussion here that people are experiencing issues 
with 3.6 on Windows and that 3.4 is the safest option. 
 - The **de.flapdoodle.embed.mongo** plugin needs to be updated to the 
latest 2.x version which supports mongo 3.4. Currently we are using 1.x which 
doesn't support Mongo 3.4.
 - In Mongo db 3.4 Config servers need to be part of a replica set. Also 
**de.flapdoodle.embed.mongo** 2.x requires a replica set for config servers. 
And the minimum number of servers in a replica set is 3, so I launch 3 config 
servers now.
 - There is a bug with **de.flapdoodle.embed.mongo** where config servers 
need to be initialized as replica sets so you need to add them to your complete 
list of replica sets to be initialized. But if you provide the configServers to 
MongosSystemForTestFactory it will try to start the config servers again! To 
work around this bug I pass an empty list of config servers to 
MongosSystemForTestFactory. This ensures each config server is only started 
once.
 - The **wiredTiger** storage engine is required to run replica sets. To 
ensure we don't revert back to **mmapv1** storage engine I explicitly configure 
**wiredTiger** on all the configs.
 - **useNoPrealloc** and ** useSmallFiles** default to true and must be set 
to false since they are configs for **mmapv1** and  
**de.flapdoodle.embed.mongo** passes these options to MongoDB if they are true. 
When MongoDB sees **mmapv1** options it throws an error. To prevent all this I 
have to explicitly set these two options to false.
 - The config server and shard server flags need to be set correctly on the 
configs, so I do that now.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ilooner/drill DRILL-6341

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1222.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1222


commit 0dde8c99c92281f8993ee598af22480afcf67a28
Author: Timothy Farkas 
Date:   2018-04-17T20:29:14Z

DRILL-6341: Fixed failing tests for mongodb storage plugin by upgrading 
MongoDB version.




---