> On July 13, 2015, 6:25 p.m., Darrel Schneider wrote: > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStore.java, > > line 19 > > <https://reviews.apache.org/r/36397/diff/1/?file=1008296#file1008296line19> > > > > How is this disk store used in later steps? > > Is it configured on the HDFS store or the Region?
DiskStore is configured on HDFS Store. Updated javadoc below: 1. [Optional] Creates a DiskStore for HDFS buffer reliability (HDFS buffers will be persisted locally till data lands on HDFS) 2. Creates a HDFS Store (connects to DiskStore created earlier) 3. Creates a Region connected to HDFS Store 4. Uses region API to create and query data > On July 13, 2015, 6:25 p.m., Darrel Schneider wrote: > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStore.java, > > line 20 > > <https://reviews.apache.org/r/36397/diff/1/?file=1008296#file1008296line20> > > > > Is "HDFS buffers..." a step the user performs? This list is supposed to > > be a list of steps done by the user HDFS buffer is automatically created for the user. Updated javadoc below: 1. [Optional] Creates a DiskStore for HDFS buffer reliability (HDFS buffers will be persisted locally till data lands on HDFS) 2. Creates a HDFS Store (connects to DiskStore created earlier) 3. Creates a Region connected to HDFS Store 4. Uses region API to create and query data > On July 13, 2015, 6:25 p.m., Darrel Schneider wrote: > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStore.java, > > line 22 > > <https://reviews.apache.org/r/36397/diff/1/?file=1008296#file1008296line22> > > > > Is it also connected to the optional disk store in step 1? > > Does a new step or sentence begin between "Store Uses"? Yes. Region uses optional disk store in step 1. I missed a line break between "Store Uses". Fixed it. > On July 13, 2015, 6:25 p.m., Darrel Schneider wrote: > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStore.java, > > line 74 > > <https://reviews.apache.org/r/36397/diff/1/?file=1008296#file1008296line74> > > > > The last sentence "... is also fine" should probably be removed. I > > think you should define the precedence of the different ways of specifying > > the name node url. Also if the explicit one is defined will it only be used > > or if is not found will it still pick up one from hdfs-site.xml and/or the > > classpath. Update javadoc: The following precedence order is applied: 1. URL configured in the store 2. URL provided in client configuration file: {@link #getHDFSClientConfigFile()} 3. URL provided in default configuration files loaded by hdfs-client HDFSStore will use the selected URL only. It will fail if the selected URL is not reachable. > On July 13, 2015, 6:25 p.m., Darrel Schneider wrote: > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStore.java, > > line 191 > > <https://reviews.apache.org/r/36397/diff/1/?file=1008296#file1008296line191> > > > > Change MaxWriteOnlyFileSize to WriteOnlyFileSizeMaximum > > > > I think in general Max should be a suffix instead of a prefix. > > > > I'm ok with Max instead of Maximum but thought it might be better to > > use the full word. I replaced the method name with WriteOnlyFileSizeLimit in Factory, Store and Mutator. Thanks for this suggestion > On July 13, 2015, 6:25 p.m., Darrel Schneider wrote: > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStore.java, > > line 202 > > <https://reviews.apache.org/r/36397/diff/1/?file=1008296#file1008296line202> > > > > This one seems related to the previous size limit. > > Perhaps better names for both are: > > WriteOnlyFileSizeLimit > > WriteOnlyFileTimeLimit > > > > which ever limit is hit first causes the file to be closed. > > > > In general seems like defaults should be doc'd (you have constants so > > put a link in these javadocs to the default constant javadocs). I think WriteOnlyFileRolloverInterval is more informative than WriteOnlyFileTimeLimit . "Interval" is used everywhere (compaction interval, batch interval, etc). Rollover is also popularly used in log configuration and its behavior is well understood. I suggest we don't change name for rollover interval. What do you think? > On July 13, 2015, 6:25 p.m., Darrel Schneider wrote: > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStoreMutator.java, > > line 14 > > <https://reviews.apache.org/r/36397/diff/1/?file=1008298#file1008298line14> > > > > Add an interface comment saying what this is, how you get and instance, > > and how to use it. * Instances of this interface are created using * {@link HDFSStore#createHdfsStoreMutator} and applied using * {@link HDFSStore#alter} > On July 13, 2015, 6:25 p.m., Darrel Schneider wrote: > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStoreMutator.java, > > line 23 > > <https://reviews.apache.org/r/36397/diff/1/?file=1008298#file1008298line23> > > > > Why the magic -1? For booleans you return Boolean and use null. Why not > > use this for ordinal's too? > > In this case it would return Integer. > > It seems unlikely that code will use the get methods on the mutator > > interface. In the current mutator design, getters are used to check which properties were explicitly altered by user. It does compute differences with HdfsStore for selective alter actions. So in case of booleans this logic depends on a third state (null) since both the values, T/F, are valid. However for integers (intervals and sizes) -1 is not a valid value. To keep it consistent, I could change this behvior. I am wondering if I should track that in a separate jira. > On July 13, 2015, 6:25 p.m., Darrel Schneider wrote: > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStoreFactory.java, > > line 28 > > <https://reviews.apache.org/r/36397/diff/1/?file=1008297#file1008297line28> > > > > Should these settor methods (also on the mutator interface) throw > > IllegalArgumentException for values out of range? For example should you be > > able to configure the number of threads to be < 0? The code validates the value and throws a IllegalArgumentException. It is not documented. I added it now. > On July 13, 2015, 6:25 p.m., Darrel Schneider wrote: > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStore.java, > > line 303 > > <https://reviews.apache.org/r/36397/diff/1/?file=1008296#file1008296line303> > > > > The first line should describe this attribute. > > The generic "This advanced..." info could be in a later sentence. > > For example: > > A file larger than this size, in megabytes, will not be considered for > > minor compaction. This option is advanced ... Fixed /** * A file larger than this size, in megabytes, will not be compacted by minor * compactor. Increasing this value will result in compaction of bigger files. * This will lower the number of files on HDFS at the cost of increased IO. * This option is for advanced users and will need tuning in special cases * only. This option is not applicable to write-only regions. * * @return size threshold (in MB) */ public int getInputFileSizeMax(); /** * A minimum number of files must exist in a bucket directory on HDFS before * minor compaction will start compaction. Keeping a higher value for this * option will reduce the frequency of minor compaction, which in turn may * result in reduced IO overhead. However it may result in increased pressure * on HDFS NameNode. This option is for advanced users and will need tuning in * special cases only. This option is not applicable to write-only regions. * * @return minimum number of files for minor compaction to get triggered */ public int getInputFileCountMin(); /** * The maximum number of files compacted by Minor compactor in a cycle. * Keeping a higher value for this option will reduce the frequency of minor * compaction, which in turn may result in reduced IO overhead. However it may * result in large number of concurrent IO operations which in-turn may * degrade the performance. This option is for advanced users and will need * tuning in special cases only. This option is not applicable to write-only * regions. * * @return maximum number of files minor compacted in one cycle */ > On July 13, 2015, 6:25 p.m., Darrel Schneider wrote: > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStore.java, > > line 175 > > <https://reviews.apache.org/r/36397/diff/1/?file=1008296#file1008296line175> > > > > What used sync disk writes? Is it only when the store writes to its > > diskStore? > > What is the default? What are the risks of setting it to false? * HDFS buffers can be persisted on local disk. Each region update record is * written to the disk synchronously if synchronous disk write is enabled. * Enable this option if the data being persisted in very and no * record should be lost in case of a crash. This high reliability mode may * increase write latency. If synchronous mode is disabled, data is persisted * in batches which usually results in better performance. > On July 13, 2015, 6:25 p.m., Darrel Schneider wrote: > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStore.java, > > line 145 > > <https://reviews.apache.org/r/36397/diff/1/?file=1008296#file1008296line145> > > > > Is this really per region or would is it per HDFS store? Each HDFS region has a linked asynchronous queue (sorted queue for RW regions), and each queue has these many dispatcher threads. Do you see any issues / differences? > On July 13, 2015, 6:25 p.m., Darrel Schneider wrote: > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStore.java, > > line 261 > > <https://reviews.apache.org/r/36397/diff/1/?file=1008296#file1008296line261> > > > > Add some info on the life cycle of these threads. > > Will they all immediately be created and live forever? Or do they only > > get created as needed. > > Do they ever timeout? Compaction threads are live forever. Each compaction thread picks up a compaction task as and when the task is created. There is no task level timeout. - Ashvin ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36397/#review91487 ----------------------------------------------------------- On July 10, 2015, 8:45 p.m., Ashvin A wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/36397/ > ----------------------------------------------------------- > > (Updated July 10, 2015, 8:45 p.m.) > > > Review request for geode, Amogh Shetkar and Dan Smith. > > > Bugs: GEODE-10 > https://issues.apache.org/jira/browse/GEODE-10 > > > Repository: geode > > > Description > ------- > > GEODE-10: Refactor HdfsStore api to match spec > > * Currently HdfsStore's configuration object is nested and a user needs to > create multiple sub objects to manage the store instance. This is less > usable > and gets confusing at times. User also gets exposed to a lot of internal > details. So replacing nested configuration with a flat structure will be > better. > * Rename members > > > Diffs > ----- > > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSEventQueueAttributes.java > ef7e863 > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSEventQueueAttributesFactory.java > fc09b7a > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStore.java > f5bd943 > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStoreFactory.java > 516d2aa > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/HDFSStoreMutator.java > 47b1708 > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/internal/HDFSEventQueueAttributesImpl.java > df89841 > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/internal/HDFSIntegrationUtil.java > 242923b > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/internal/HDFSStoreConfigHolder.java > 50ea3c6 > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/internal/HDFSStoreCreation.java > b5fbfe8 > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/internal/HDFSStoreFactoryImpl.java > e7e75dc > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/internal/HDFSStoreImpl.java > 8e7e358 > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/internal/HDFSStoreMutatorImpl.java > e4e2093 > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/internal/SizeTieredHdfsCompactionConfigHolder.java > 57d58b7 > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/internal/hoplog/HDFSCompactionManager.java > d96cd11 > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/internal/hoplog/HDFSUnsortedHoplogOrganizer.java > c9be401 > > gemfire-core/src/main/java/com/gemstone/gemfire/cache/hdfs/internal/hoplog/HdfsSortedOplogOrganizer.java > 7bb025b > > gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/GemFireCacheImpl.java > 76488dd > > gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/xmlcache/CacheCreation.java > a5fd3f1 > > gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/xmlcache/CacheXmlParser.java > e015624 > > gemfire-core/src/main/java/com/gemstone/gemfire/management/internal/cli/commands/HDFSStoreCommands.java > b044156 > > gemfire-core/src/main/java/com/gemstone/gemfire/management/internal/cli/functions/AlterHDFSStoreFunction.java > 13ba5fa > > gemfire-core/src/main/java/com/gemstone/gemfire/management/internal/cli/functions/CreateHDFSStoreFunction.java > ac9c895 > > gemfire-core/src/test/java/com/gemstone/gemfire/cache/hdfs/internal/ColocatedRegionWithHDFSDUnitTest.java > ea9dc11 > > gemfire-core/src/test/java/com/gemstone/gemfire/cache/hdfs/internal/HDFSConfigJUnitTest.java > dc3f404 > > gemfire-core/src/test/java/com/gemstone/gemfire/cache/hdfs/internal/HdfsStoreMutatorJUnitTest.java > f12078e > > gemfire-core/src/test/java/com/gemstone/gemfire/cache/hdfs/internal/hoplog/BaseHoplogTestCase.java > 06c3c4b > > gemfire-core/src/test/java/com/gemstone/gemfire/cache/hdfs/internal/hoplog/HDFSCacheLoaderJUnitTest.java > 3e4b8e2 > > gemfire-core/src/test/java/com/gemstone/gemfire/cache/hdfs/internal/hoplog/HDFSCompactionManagerJUnitTest.java > 2c32d61 > > gemfire-core/src/test/java/com/gemstone/gemfire/cache/hdfs/internal/hoplog/HDFSUnsortedHoplogOrganizerJUnitTest.java > 8d2fae3 > > gemfire-core/src/test/java/com/gemstone/gemfire/cache/hdfs/internal/hoplog/HdfsSortedOplogOrganizerJUnitTest.java > 0884a40 > > gemfire-core/src/test/java/com/gemstone/gemfire/cache/hdfs/internal/hoplog/TieredCompactionJUnitTest.java > 007e73b > > gemfire-core/src/test/java/com/gemstone/gemfire/cache/hdfs/internal/hoplog/mapreduce/HoplogUtilJUnitTest.java > 576144f > > gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/HDFSRegionOperationsJUnitTest.java > d068cb4 > > gemfire-core/src/test/java/com/gemstone/gemfire/management/internal/cli/commands/HDFSStoreCommandsJUnitTest.java > b46abc7 > > gemfire-core/src/test/java/com/gemstone/gemfire/management/internal/cli/functions/AlterHDFSStoreFunctionJUnitTest.java > f66f54b > > gemfire-core/src/test/java/com/gemstone/gemfire/management/internal/cli/functions/CreateHDFSStoreFunctionJUnitTest.java > a20297b > > gemfire-core/src/test/java/com/gemstone/gemfire/management/internal/cli/functions/DescribeHDFSStoreFunctionJUnitTest.java > fe630fe > > Diff: https://reviews.apache.org/r/36397/diff/ > > > Testing > ------- > > > Thanks, > > Ashvin A > >
