Jay, This makes perfect sense and it is very helpful.
thanks, Jason On Thu, Apr 18, 2013 at 12:20 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > I think there is a misunderstanding here. The log file division is > completely transparent to the consumer. The consumer consumes by > offset (0,1,2,3,...) and these offsets span log files (i.e. there is a > single, monotonically increasing set of offsets over all log files in > a partition). In other words consumers ALWAYS consume across log files > based on the position (offset) in the set of all logs in the > partition. > > If you want fewer log files change the settings so that the log files > are bigger. However the only impact of this is fewer file handles, it > won't impact the consumer in any way at all. > > Not sure if that helps... > -Jay > > On Thu, Apr 18, 2013 at 8:36 AM, Jason Huang <jason.hu...@icare.com> > wrote: > > Number of open file handles is one thing. But I am thinking more of a use > > case where you may want to fetch information from two separate log files > of > > same topic and same partition. Could we use one high level consumer API > > call to achieve that? One naive idea is that if we can merge those two > log > > files (since they are of same topic and in partition I thought we might > be > > able to merge them) into one, then using one high level consumer API will > > do the job. > > > > > > On Thu, Apr 18, 2013 at 11:26 AM, Neha Narkhede <neha.narkh...@gmail.com > >wrote: > > > >> There is no way to merge those log files in 0.8. But why would you want > to > >> do that ? To reduce the number of open file handles ? > >> > >> Thanks, > >> Neha > >> > >> On Thursday, April 18, 2013, Jason Huang wrote: > >> > >> > Just curious - given the situation above, if you have two sets of .log > >> and > >> > .index files under the same topic and same partition, is there any > way to > >> > merge these two sets of log files and index files? > >> > > >> > > >> > On Thu, Apr 18, 2013 at 5:42 AM, Swapnil Ghike <sgh...@linkedin.com> > >> > wrote: > >> > > >> > > Otherwise if you are using the high level consumer > >> > > (ZookeeperConsumerConnector) in code, you will have to start a new > >> > > consumer with a different group id. > >> > > > >> > > Thanks, > >> > > Swapnil > >> > > > >> > > On 4/18/13 2:09 AM, "Swapnil Ghike" <sgh...@linkedin.com> wrote: > >> > > > >> > > >Snehalata, > >> > > > > >> > > >Are you using the console consumer? You can pass a --from-beginning > >> flag > >> > > >to the console consumer to get all the messages posted to that > topic. > >> > > > > >> > > >Thanks, > >> > > >Swapnil > >> > > > > >> > > >On 4/18/13 1:32 AM, "Snehalata Nagaje" > >> > > ><snehalata.nag...@harbingergroup.com> wrote: > >> > > > > >> > > >>Hi Neha, > >> > > >> > >> > > >>I was able to figure this out, this is happening due to time based > >> > > >>rolling. > >> > > >> > >> > > >>But now problem, when I fetch the messages for particular topic, > >> server > >> > > >>is > >> > > >>reading only one log file, returns those messages only. > >> > > >> > >> > > >>Is there any way I can combine all log files which are already > >> produced > >> > > >>into > >> > > >>one, so that I can get all messages posted in that topic. > >> > > >> > >> > > >>Thanks, > >> > > >>Snehalata > >> > > >> > >> > > >> > >> > > >>-----Original Message----- > >> > > >>From: Snehalata Nagaje [mailto: > snehalata.nag...@harbingergroup.com] > >> > > >>Sent: Thursday, April 18, 2013 11:41 AM > >> > > >>To: users@kafka.apache.org > >> > > >>Subject: RE: producer creating multiple log files after message > >> sending > >> > > >> > >> > > >>Thanks Neha, > >> > > >> > >> > > >>-----Original Message----- > >> > > >>From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > >> > > >>Sent: Wednesday, April 17, 2013 9:08 PM > >> > > >>To: users@kafka.apache.org > >> > > >>Subject: Re: producer creating multiple log files after message > >> sending > >> > > >> > >> > > >>Hi Snehalata, > >> > > >> > >> > > >>Would you grep through the log4j files for the topic under > >> > > >>consideration and look for "Rolling.*due to". This will tell us > why > >> > > >>the log is getting rolled over. > >> > > >> > >> > > >>Thanks, > >> > > >>Neha > >> > > >> > >> > > >>On Wed, Apr 17, 2013 at 12:05 AM, Snehalata Nagaje > >> > > >><snehalata.nag...@harbingergroup.com> wrote: > >> > > >>> Hi, > >> > > >>> > >> > > >>> > >> > > >>> > >> > > >>> I am using kafak 0.8 version and sending messages to kafka > server > >> > using > >> > > >>> below code, > >> > > >>> > >> > > >>> > >> > > >>> > >> > > >>> Producer<Integer, String> producer = new Producer<Integer, > >> > String>(new > >> > > >>> ProducerConfig(props)); > >> > > >>> > >> > > >>> producer.send(new > KeyedMessage(topicName,partitionKey,jsonString)); > >> > > >>> > >> > > >>> producer.close(); > >> > > >>> > >> > > >>> > >> > > >>> > >> > > >>> I am able to post messages with, but after certain days, kafka > >> > creating > >> > > >>> multiple log files, messages are posted in new log file, > although > >> old > >> > > >>>file > >> > > >>> size is very less. > >> > > >>> > >> > > >>> > >> > > >>> > >> > > >>> What can be the reason for creating multiple log files? Does > >> message > >> > > >>format > >> > > >>> matters while sending message to kafka? > >> > > >>> > >> > > >>> > >> > > >>> > >> > > >>> Thanks, > >> > > >>> > >> > > >>> Snehalata > >> > > >>> > >> > > >>> > >> > > >>> Disclaimer: > >> > > >>> This e-mail may contain Privileged/Confidential information and > is > >> > > >>intended on > >> >
