Any objections if I leave KAFKA-2114 (setting min.insync.replicas default) out?
The test code is using changes that were done after 0.8.2.x cut-off, which makes it difficult to cherry-pick. Gwen On Tue, Aug 18, 2015 at 12:16 PM, Gwen Shapira <g...@confluent.io> wrote: > Jun, > > KAFKA-2147 doesn't seem to have a commit associated with it, so I can't > cherrypick just this fix. > I suggest leaving this out since there is a 0.8.2.x workaround in the JIRA. > > Gwen > > On Mon, Aug 17, 2015 at 5:24 PM, Jun Rao <j...@confluent.io> wrote: > >> Gwen, >> >> Thanks for putting the list together. >> >> I'd recommend that we exclude the following: >> KAFKA-1702: This is for the old producer and is only a problem if there >> are >> some unexpected exceptions (e.g. UnknownClass). >> KAFKA-2336: Most people don't change offsets.topic.num.partitions. >> KAFKA-1724: The patch there is never committed since the fix is included >> in >> another jira (a much larger patch). >> KAFKA-2241: This doesn't seem be a common problem. It only happens when >> the >> fetch request blocks on the broker for an extended period of time, which >> should be rare. >> >> I'd also recommend that we include the following: >> KAFKA-2147: This impacts the memory size of the purgatory and a number of >> people have experienced that. The fix is small and has been tested in >> production usage. It hasn't been committed though since the issue is >> already fixed in trunk and we weren't planning for an 0.8.2.2 release >> then. >> >> Thanks, >> >> Jun >> >> On Mon, Aug 17, 2015 at 2:56 PM, Gwen Shapira <g...@confluent.io> wrote: >> >> > Thanks for creating a list, Grant! >> > >> > I placed it on the wiki with a quick evaluation of the content and >> whether >> > it should be in 0.8.2.2: >> > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/Proposed+patches+for+0.8.2.2 >> > >> > I'm attempting to only cherrypick fixes that are both important for >> large >> > number of users (or very critical to some users) and very safe (mostly >> > judged by the size of the change, but not only) >> > >> > If your favorite bugfix is missing from the list, or is there but marked >> > "No", please let us know (in this thread) what we are missing and why >> it is >> > both important and safe. >> > Also, if I accidentally included something you consider unsafe, speak >> up! >> > >> > Gwen >> > >> > On Mon, Aug 17, 2015 at 8:17 AM, Grant Henke <ghe...@cloudera.com> >> wrote: >> > >> > > +dev >> > > >> > > Adding dev list back in. Somehow it got dropped. >> > > >> > > >> > > On Mon, Aug 17, 2015 at 10:16 AM, Grant Henke <ghe...@cloudera.com> >> > wrote: >> > > >> > > > Below is a list of candidate bug fix jiras marked fixed for 0.8.3. I >> > > don't >> > > > suspect all of these will (or should) make it into the release but >> this >> > > > should be a relatively complete list to work from: >> > > > >> > > > - KAFKA-2114 <https://issues.apache.org/jira/browse/KAFKA-2114>: >> > > Unable >> > > > to change min.insync.replicas default >> > > > - KAFKA-1702 <https://issues.apache.org/jira/browse/KAFKA-1702>: >> > > > Messages silently Lost by producer >> > > > - KAFKA-2012 <https://issues.apache.org/jira/browse/KAFKA-2012>: >> > > > Broker should automatically handle corrupt index files >> > > > - KAFKA-2406 <https://issues.apache.org/jira/browse/KAFKA-2406>: >> > ISR >> > > > propagation should be throttled to avoid overwhelming controller. >> > > > - KAFKA-2336 <https://issues.apache.org/jira/browse/KAFKA-2336>: >> > > > Changing offsets.topic.num.partitions after the offset topic is >> > > created >> > > > breaks consumer group partition assignment >> > > > - KAFKA-2337 <https://issues.apache.org/jira/browse/KAFKA-2337>: >> > > Verify >> > > > that metric names will not collide when creating new topics >> > > > - KAFKA-2393 <https://issues.apache.org/jira/browse/KAFKA-2393>: >> > > > Correctly Handle InvalidTopicException in >> > KafkaApis.getTopicMetadata() >> > > > - KAFKA-2189 <https://issues.apache.org/jira/browse/KAFKA-2189>: >> > > Snappy >> > > > compression of message batches less efficient in 0.8.2.1 >> > > > - KAFKA-2308 <https://issues.apache.org/jira/browse/KAFKA-2308>: >> > New >> > > > producer + Snappy face un-compression errors after broker restart >> > > > - KAFKA-2042 <https://issues.apache.org/jira/browse/KAFKA-2042>: >> > New >> > > > producer metadata update always get all topics. >> > > > - KAFKA-1367 <https://issues.apache.org/jira/browse/KAFKA-1367>: >> > > Broker >> > > > topic metadata not kept in sync with ZooKeeper >> > > > - KAFKA-972 <https://issues.apache.org/jira/browse/KAFKA-972>: >> > > MetadataRequest >> > > > returns stale list of brokers >> > > > - KAFKA-1867 <https://issues.apache.org/jira/browse/KAFKA-1867>: >> > > liveBroker >> > > > list not updated on a cluster with no topics >> > > > - KAFKA-1650 <https://issues.apache.org/jira/browse/KAFKA-1650>: >> > > Mirror >> > > > Maker could lose data on unclean shutdown. >> > > > - KAFKA-2009 <https://issues.apache.org/jira/browse/KAFKA-2009>: >> > Fix >> > > > UncheckedOffset.removeOffset synchronization and trace logging >> issue >> > > in >> > > > mirror maker >> > > > - KAFKA-2407 <https://issues.apache.org/jira/browse/KAFKA-2407>: >> > Only >> > > > create a log directory when it will be used >> > > > - KAFKA-2327 <https://issues.apache.org/jira/browse/KAFKA-2327>: >> > > > broker doesn't start if config defines advertised.host but not >> > > > advertised.port >> > > > - KAFKA-1788: producer record can stay in RecordAccumulator >> forever >> > if >> > > > leader is no available >> > > > - KAFKA-2234 <https://issues.apache.org/jira/browse/KAFKA-2234>: >> > > > Partition reassignment of a nonexistent topic prevents future >> > > reassignments >> > > > - KAFKA-2096 <https://issues.apache.org/jira/browse/KAFKA-2096>: >> > > > Enable keepalive socket option for broker to prevent socket leak >> > > > - KAFKA-1057 <https://issues.apache.org/jira/browse/KAFKA-1057>: >> > Trim >> > > > whitespaces from user specified configs >> > > > - KAFKA-1641 <https://issues.apache.org/jira/browse/KAFKA-1641>: >> > Log >> > > > cleaner exits if last cleaned offset is lower than earliest >> offset >> > > > - KAFKA-1648 <https://issues.apache.org/jira/browse/KAFKA-1648>: >> > > Round >> > > > robin consumer balance throws an NPE when there are no topics >> > > > - KAFKA-1724 <https://issues.apache.org/jira/browse/KAFKA-1724>: >> > > > Errors after reboot in single node setup >> > > > - KAFKA-1758 <https://issues.apache.org/jira/browse/KAFKA-1758>: >> > > > corrupt recovery file prevents startup >> > > > - KAFKA-1866 <https://issues.apache.org/jira/browse/KAFKA-1866>: >> > > > LogStartOffset gauge throws exceptions after log.delete() >> > > > - KAFKA-1883 <https://issues.apache.org/jira/browse/KAFKA-1883>: >> > > NullPointerException >> > > > in RequestSendThread >> > > > - KAFKA-1896 <https://issues.apache.org/jira/browse/KAFKA-1896>: >> > > > Record size funcition of record in mirror maker hit NPE when the >> > > message >> > > > value is null. >> > > > - KAFKA-2101 <https://issues.apache.org/jira/browse/KAFKA-2101>: >> > > > Metric metadata-age is reset on a failed update >> > > > - KAFKA-2112 <https://issues.apache.org/jira/browse/KAFKA-2112>: >> > make >> > > > overflowWheel volatile >> > > > - KAFKA-2117 <https://issues.apache.org/jira/browse/KAFKA-2117>: >> > > > OffsetManager uses incorrect field for metadata >> > > > - KAFKA-2164 <https://issues.apache.org/jira/browse/KAFKA-2164>: >> > > > ReplicaFetcherThread: suspicious log message on reset offset >> > > > - KAFKA-1668 <https://issues.apache.org/jira/browse/KAFKA-1668>: >> > > > TopicCommand doesn't warn if --topic argument doesn't match any >> > topics >> > > > - KAFKA-2198 <https://issues.apache.org/jira/browse/KAFKA-2198>: >> > > > kafka-topics.sh exits with 0 status on failures >> > > > - KAFKA-2235 <https://issues.apache.org/jira/browse/KAFKA-2235>: >> > > > LogCleaner offset map overflow >> > > > - KAFKA-2241 <https://issues.apache.org/jira/browse/KAFKA-2241>: >> > > > AbstractFetcherThread.shutdown() should not block on >> > > > ReadableByteChannel.read(buffer) >> > > > - KAFKA-2272 <https://issues.apache.org/jira/browse/KAFKA-2272>: >> > > > listeners endpoint parsing fails if the hostname has capital >> letter >> > > > - KAFKA-2345 <https://issues.apache.org/jira/browse/KAFKA-2345>: >> > > > Attempt to delete a topic already marked for deletion throws >> > > > ZkNodeExistsException >> > > > - KAFKA-2353 <https://issues.apache.org/jira/browse/KAFKA-2353>: >> > > > SocketServer.Processor should catch exception and close the >> socket >> > > properly >> > > > in configureNewConnections. >> > > > - KAFKA-1836 <https://issues.apache.org/jira/browse/KAFKA-1836>: >> > > > metadata.fetch.timeout.ms set to zero blocks forever >> > > > - KAFKA-2317 <https://issues.apache.org/jira/browse/KAFKA-2317>: >> > > De-register >> > > > isrChangeNotificationListener on controller resignation >> > > > >> > > > Note: KAFKA-2120 <https://issues.apache.org/jira/browse/KAFKA-2120> >> & >> > > > KAFKA-2421 <https://issues.apache.org/jira/browse/KAFKA-2421> were >> > > > mentioned in previous emails, but are not in the list because they >> are >> > > not >> > > > committed yet. >> > > > >> > > > Hope that helps the effort. >> > > > >> > > > Thanks, >> > > > Grant >> > > > >> > > > On Mon, Aug 17, 2015 at 12:09 AM, Grant Henke <ghe...@cloudera.com> >> > > wrote: >> > > > >> > > >> +1 to that suggestion. Though I suspect that requires a committer >> to >> > do. >> > > >> Making it part of the standard commit process could work too. >> > > >> On Aug 16, 2015 11:01 PM, "Gwen Shapira" <g...@confluent.io> >> wrote: >> > > >> >> > > >>> BTW. I think it will be great for Apache Kafka to have a 0.8.2 >> > "release >> > > >>> manager" who's role is to cherrypick low-risk bug-fixes into the >> > 0.8.2 >> > > >>> branch and once enough bug fixes happened (or if sufficiently >> > critical >> > > >>> fixes happened) to roll out a new maintenance release (with every >> 3 >> > > month >> > > >>> as a reasonable bugfix release target). >> > > >>> >> > > >>> This will add some predictability regarding how fast we release >> fixes >> > > for >> > > >>> bugs. >> > > >>> >> > > >>> Gwen >> > > >>> >> > > >>> On Sun, Aug 16, 2015 at 8:09 PM, Jeff Holoman < >> jholo...@cloudera.com >> > > >> > > >>> wrote: >> > > >>> >> > > >>> > +1 for the release and also including >> > > >>> > >> > > >>> > https://issues.apache.org/jira/browse/KAFKA-2114 >> > > >>> > >> > > >>> > Thanks >> > > >>> > >> > > >>> > Jeff >> > > >>> > >> > > >>> > On Sun, Aug 16, 2015 at 2:51 PM, Stevo Slavić < >> ssla...@gmail.com> >> > > >>> wrote: >> > > >>> > >> > > >>> > > +1 (non-binding) for 0.8.2.2 release >> > > >>> > > >> > > >>> > > Would be nice to include in that release new producer >> resiliency >> > > bug >> > > >>> > fixes >> > > >>> > > https://issues.apache.org/jira/browse/KAFKA-1788 and >> > > >>> > > https://issues.apache.org/jira/browse/KAFKA-2120 >> > > >>> > > >> > > >>> > > On Fri, Aug 14, 2015 at 4:03 PM, Gwen Shapira < >> g...@confluent.io >> > > >> > > >>> wrote: >> > > >>> > > >> > > >>> > > > Will be nice to include Kafka-2308 and fix two critical >> snappy >> > > >>> issues >> > > >>> > in >> > > >>> > > > the maintenance release. >> > > >>> > > > >> > > >>> > > > Gwen >> > > >>> > > > On Aug 14, 2015 6:16 AM, "Grant Henke" <ghe...@cloudera.com >> > >> > > >>> wrote: >> > > >>> > > > >> > > >>> > > > > Just to clarify. Will KAFKA-2189 be the only patch in the >> > > >>> release? >> > > >>> > > > > >> > > >>> > > > > On Fri, Aug 14, 2015 at 7:35 AM, Manikumar Reddy < >> > > >>> > ku...@nmsworks.co.in >> > > >>> > > > >> > > >>> > > > > wrote: >> > > >>> > > > > >> > > >>> > > > > > +1 for 0.8.2.2 release >> > > >>> > > > > > >> > > >>> > > > > > On Fri, Aug 14, 2015 at 5:49 PM, Ismael Juma < >> > > >>> ism...@juma.me.uk> >> > > >>> > > > wrote: >> > > >>> > > > > > >> > > >>> > > > > > > I think this is a good idea as the change is minimal >> on >> > our >> > > >>> side >> > > >>> > > and >> > > >>> > > > it >> > > >>> > > > > > has >> > > >>> > > > > > > been tested in production for some time by the >> reporter. >> > > >>> > > > > > > >> > > >>> > > > > > > Best, >> > > >>> > > > > > > Ismael >> > > >>> > > > > > > >> > > >>> > > > > > > On Fri, Aug 14, 2015 at 1:15 PM, Jun Rao < >> > j...@confluent.io >> > > > >> > > >>> > wrote: >> > > >>> > > > > > > >> > > >>> > > > > > > > Hi, Everyone, >> > > >>> > > > > > > > >> > > >>> > > > > > > > Since the release of Kafka 0.8.2.1, a number of >> people >> > > have >> > > >>> > > > reported >> > > >>> > > > > an >> > > >>> > > > > > > > issue with snappy compression ( >> > > >>> > > > > > > > https://issues.apache.org/jira/browse/KAFKA-2189). >> > > >>> Basically, >> > > >>> > if >> > > >>> > > > > they >> > > >>> > > > > > > use >> > > >>> > > > > > > > snappy in 0.8.2.1, they will experience a 2-3X space >> > > >>> increase. >> > > >>> > > The >> > > >>> > > > > > issue >> > > >>> > > > > > > > has since been fixed in trunk (just a snappy jar >> > > upgrade). >> > > >>> > Since >> > > >>> > > > > 0.8.3 >> > > >>> > > > > > is >> > > >>> > > > > > > > still a few months away, it may make sense to do an >> > > 0.8.2.2 >> > > >>> > > release >> > > >>> > > > > > just >> > > >>> > > > > > > to >> > > >>> > > > > > > > fix this issue. Any objections? >> > > >>> > > > > > > > >> > > >>> > > > > > > > Thanks, >> > > >>> > > > > > > > >> > > >>> > > > > > > > Jun >> > > >>> > > > > > > > >> > > >>> > > > > > > >> > > >>> > > > > > >> > > >>> > > > > >> > > >>> > > > > >> > > >>> > > > > >> > > >>> > > > > -- >> > > >>> > > > > Grant Henke >> > > >>> > > > > Software Engineer | Cloudera >> > > >>> > > > > gr...@cloudera.com | twitter.com/gchenke | >> > > >>> > linkedin.com/in/granthenke >> > > >>> > > > > >> > > >>> > > > >> > > >>> > > >> > > >>> > >> > > >>> > >> > > >>> > >> > > >>> > -- >> > > >>> > Jeff Holoman >> > > >>> > Systems Engineer >> > > >>> > >> > > >>> >> > > >> >> > > > >> > > > >> > > > -- >> > > > Grant Henke >> > > > Software Engineer | Cloudera >> > > > gr...@cloudera.com | twitter.com/gchenke | >> linkedin.com/in/granthenke >> > > > >> > > >> > > >> > > >> > > -- >> > > Grant Henke >> > > Software Engineer | Cloudera >> > > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke >> > > >> > >> > >
