Re: AccessDeniedException in Kafka on Windows
Hi Martijn de Bruijn, I think I've reponsed your other mail. But you're right: > On several places FileChannel.open(path, StandardOpenOption.READ) is called which will always fail on Windows when path is a directory. Yes, we tried to flush(fsync) data on directory to make sure it's successfully stored in disk, but didn't make sure if it works in Windows. The fix will be in v3.1.0 (which should be released soon), and v3.0.1. Thank you. Luke On Thu, Dec 9, 2021 at 3:27 PM de Bruijn, M. (Martijn) wrote: > We are upgrading our Spring Boot applications to Spring Boot 2.6.1. > Spring Boot 2.6.1 upgrade our Kafka dependency from 2.7.1 to 3.0.0. > After upgrading I'm getting a AccessDeniedException on all my tests using > @EmbeddedKafka. > > Caused by: java.nio.file.AccessDeniedException: > C:\Temp\spring.kafka.ae934565-33bc-4073-ab06-59ce265490369531329615041832669 > at > java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89) > at > java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103) > at > java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108) > at > java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121) > at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298) > at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357) > at org.apache.kafka.common.utils.Utils.flushDir(Utils.java:953) > at > org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:941) > at > kafka.server.BrokerMetadataCheckpoint.liftedTree1$1(BrokerMetadataCheckpoint.scala:214) > at > kafka.server.BrokerMetadataCheckpoint.write(BrokerMetadataCheckpoint.scala:204) > > I've been searching in the Kafka source code for the cause of the error > and it seems the Utils.flushDir is causing troubles on Windows. > On several places FileChannel.open(path, StandardOpenOption.READ) is > called which will always fail on Windows when path is a directory. See: > https://mail.openjdk.java.net/pipermail/nio-dev/2013-February/002123.html > > Another example of failing code on Windows is the > kafka.log.LogManager.createAndValidateLogDirs method which calls the > Utils.flushDir(dir.toPath.toAbsolutePath.normalize.getParent) with a > directory as parm. This will always fail on Windows. > > How can this be solved? (I can't find a related Kafka Jira issue) > > > More related reports: > > https://stackoverflow.com/questions/70182684/failed-to-start-kafka-server-in-window/70183556 > > > Regards, > Martijn de Bruijn > Software engineer >
AccessDeniedException in Kafka on Windows
We are upgrading our Spring Boot applications to Spring Boot 2.6.1. Spring Boot 2.6.1 upgrade our Kafka dependency from 2.7.1 to 3.0.0. After upgrading I'm getting a AccessDeniedException on all my tests using @EmbeddedKafka. Caused by: java.nio.file.AccessDeniedException: C:\Temp\spring.kafka.ae934565-33bc-4073-ab06-59ce265490369531329615041832669 at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89) at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103) at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108) at java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121) at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298) at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357) at org.apache.kafka.common.utils.Utils.flushDir(Utils.java:953) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:941) at kafka.server.BrokerMetadataCheckpoint.liftedTree1$1(BrokerMetadataCheckpoint.scala:214) at kafka.server.BrokerMetadataCheckpoint.write(BrokerMetadataCheckpoint.scala:204) I've been searching in the Kafka source code for the cause of the error and it seems the Utils.flushDir is causing troubles on Windows. On several places FileChannel.open(path, StandardOpenOption.READ) is called which will always fail on Windows when path is a directory. See: https://mail.openjdk.java.net/pipermail/nio-dev/2013-February/002123.html Another example of failing code on Windows is the kafka.log.LogManager.createAndValidateLogDirs method which calls the Utils.flushDir(dir.toPath.toAbsolutePath.normalize.getParent) with a directory as parm. This will always fail on Windows. How can this be solved? (I can't find a related Kafka Jira issue) More related reports: https://stackoverflow.com/questions/70182684/failed-to-start-kafka-server-in-window/70183556 Regards, Martijn de Bruijn Software engineer
Re: Using Kafka on Windows (Production environment)
On 2019/06/04 12:58:10, "M. Manna" wrote: > Kafka cannot be run on Windows in production. There are problems with > memory map allocation/releases which results into fatal shutdown. On Linux > it’s allowed, but on Windows it’s prevented. > > You can reproduce this by setting a small log retention period on your > windows and test with a QuickStart. > > It’s advisable to run it on Docker or Linux VM. > > Thanks, > > > > > > On Tue, 4 Jun 2019 at 14:32, Thiago Sebestyen Varea > wrote: > > > Hi everyone! > > > > I know it is not a common scenario, but I would like to know whether Kafka > > is ready for running on Windows in a production environment? > > > > If yes, does someone have any experience with this scenario to tell me if > > it is good or not? > > > > Thanks in advance! > > > > Best Regards, > > Thiago Sebestyen Varea > > ' > > > > > > Esta mensagem pode conter informação confidencial e/ou privilegiada e é > > destinada exclusivamente ao indivíduo ou à entidade para a qual é > > endereçada. O remetente utiliza o correio eletrônico no exercício do seu > > trabalho ou em razão dele, eximindo a Companhia e as demais instituições > > integrantes de seu grupo econômico de qualquer responsabilidade por sua > > utilização indevida. Se você não for o destinatário ou a pessoa autorizada > > a receber esta mensagem, não deverá utilizar, armazenar, copiar, alterar > > e/ou divulgar a mensagem ou qualquer informação nela contida, ou tomar > > qualquer ação baseada nessas informações. Se você recebeu esta mensagem por > > engano, por favor, avise imediatamente o remetente, apagando a mensagem em > > seguida. Agradecemos sua cooperação. > > > > This message may contain confidential and/or privileged information and is > > intended to be received solely by the person or entity it is addressed to. > > The sender uses its e-mail box as a working tool, being the Company and the > > other entities that are part of its corporate group not liable for any > > misuse. If you are not the addressee or authorized to receive this for the > > addressee, you must not use, retain, copy, disclose, change and/or take any > > action based on this message or any information herein. If you have > > received this message in error, please inform the sender immediately and > > delete this message. Thank you for your cooperation. > > ' > > ' > > > > > > Esta mensagem pode conter informação confidencial e/ou privilegiada e é > > destinada exclusivamente ao indivíduo ou à entidade para a qual é > > endereçada. O remetente utiliza o correio eletrônico no exercício do seu > > trabalho ou em razão dele, eximindo a Companhia e as demais instituições > > integrantes de seu grupo econômico de qualquer responsabilidade por sua > > utilização indevida. Se você não for o destinatário ou a pessoa autorizada > > a receber esta mensagem, não deverá utilizar, armazenar, copiar, alterar > > e/ou divulgar a mensagem ou qualquer informação nela contida, ou tomar > > qualquer ação baseada nessas informações. Se você recebeu esta mensagem por > > engano, por favor, avise imediatamente o remetente, apagando a mensagem em > > seguida. Agradecemos sua cooperação. > > > > This message may contain confidential and/or privileged information and is > > intended to be received solely by the person or entity it is addressed to. > > The sender uses its e-mail box as a working tool, being the Company and the > > other entities that are part of its corporate group not liable for any > > misuse. If you are not the addressee or authorized to receive this for the > > addressee, you must not use, retain, copy, disclose, change and/or take any > > action based on this message or any information herein. If you have > > received this message in error, please inform the sender immediately and > > delete this message. Thank you for your cooperation. > > ' > > > Hi, In regard to the issue on Windows, are you referring to K-6188 issue (https://jira.apache.org/jira/browse/KAFKA-6188)? Or if it is a different issue, can you please elaborate on the issue? What are the limitations on Kafka usage? In the main Kafka Apache site it is presented as Kafka is supported on Windows. Actually, when downloading Kafka official releases, it includes the Windows Startup scripts for Kafka and ZK... and the Quickstart guide is discussing Windows without specifying any limitations. Unfortunately, Linux is not an option In many organizations and products. In regard to K-6188 (https://jira.apache.org/jira/browse/KAFKA-6188), this issue still exists in latest version of Kafka although the issue is shown as "Resolved" in Jira. Thanks.
Re: Using Kafka on Windows (Production environment)
Kafka cannot be run on Windows in production. There are problems with memory map allocation/releases which results into fatal shutdown. On Linux it’s allowed, but on Windows it’s prevented. You can reproduce this by setting a small log retention period on your windows and test with a QuickStart. It’s advisable to run it on Docker or Linux VM. Thanks, On Tue, 4 Jun 2019 at 14:32, Thiago Sebestyen Varea wrote: > Hi everyone! > > I know it is not a common scenario, but I would like to know whether Kafka > is ready for running on Windows in a production environment? > > If yes, does someone have any experience with this scenario to tell me if > it is good or not? > > Thanks in advance! > > Best Regards, > Thiago Sebestyen Varea > ' > > > Esta mensagem pode conter informação confidencial e/ou privilegiada e é > destinada exclusivamente ao indivíduo ou à entidade para a qual é > endereçada. O remetente utiliza o correio eletrônico no exercício do seu > trabalho ou em razão dele, eximindo a Companhia e as demais instituições > integrantes de seu grupo econômico de qualquer responsabilidade por sua > utilização indevida. Se você não for o destinatário ou a pessoa autorizada > a receber esta mensagem, não deverá utilizar, armazenar, copiar, alterar > e/ou divulgar a mensagem ou qualquer informação nela contida, ou tomar > qualquer ação baseada nessas informações. Se você recebeu esta mensagem por > engano, por favor, avise imediatamente o remetente, apagando a mensagem em > seguida. Agradecemos sua cooperação. > > This message may contain confidential and/or privileged information and is > intended to be received solely by the person or entity it is addressed to. > The sender uses its e-mail box as a working tool, being the Company and the > other entities that are part of its corporate group not liable for any > misuse. If you are not the addressee or authorized to receive this for the > addressee, you must not use, retain, copy, disclose, change and/or take any > action based on this message or any information herein. If you have > received this message in error, please inform the sender immediately and > delete this message. Thank you for your cooperation. > ' > ' > > > Esta mensagem pode conter informação confidencial e/ou privilegiada e é > destinada exclusivamente ao indivíduo ou à entidade para a qual é > endereçada. O remetente utiliza o correio eletrônico no exercício do seu > trabalho ou em razão dele, eximindo a Companhia e as demais instituições > integrantes de seu grupo econômico de qualquer responsabilidade por sua > utilização indevida. Se você não for o destinatário ou a pessoa autorizada > a receber esta mensagem, não deverá utilizar, armazenar, copiar, alterar > e/ou divulgar a mensagem ou qualquer informação nela contida, ou tomar > qualquer ação baseada nessas informações. Se você recebeu esta mensagem por > engano, por favor, avise imediatamente o remetente, apagando a mensagem em > seguida. Agradecemos sua cooperação. > > This message may contain confidential and/or privileged information and is > intended to be received solely by the person or entity it is addressed to. > The sender uses its e-mail box as a working tool, being the Company and the > other entities that are part of its corporate group not liable for any > misuse. If you are not the addressee or authorized to receive this for the > addressee, you must not use, retain, copy, disclose, change and/or take any > action based on this message or any information herein. If you have > received this message in error, please inform the sender immediately and > delete this message. Thank you for your cooperation. > ' >
Using Kafka on Windows (Production environment)
Hi everyone! I know it is not a common scenario, but I would like to know whether Kafka is ready for running on Windows in a production environment? If yes, does someone have any experience with this scenario to tell me if it is good or not? Thanks in advance! Best Regards, Thiago Sebestyen Varea ' Esta mensagem pode conter informação confidencial e/ou privilegiada e é destinada exclusivamente ao indivíduo ou à entidade para a qual é endereçada. O remetente utiliza o correio eletrônico no exercício do seu trabalho ou em razão dele, eximindo a Companhia e as demais instituições integrantes de seu grupo econômico de qualquer responsabilidade por sua utilização indevida. Se você não for o destinatário ou a pessoa autorizada a receber esta mensagem, não deverá utilizar, armazenar, copiar, alterar e/ou divulgar a mensagem ou qualquer informação nela contida, ou tomar qualquer ação baseada nessas informações. Se você recebeu esta mensagem por engano, por favor, avise imediatamente o remetente, apagando a mensagem em seguida. Agradecemos sua cooperação. This message may contain confidential and/or privileged information and is intended to be received solely by the person or entity it is addressed to. The sender uses its e-mail box as a working tool, being the Company and the other entities that are part of its corporate group not liable for any misuse. If you are not the addressee or authorized to receive this for the addressee, you must not use, retain, copy, disclose, change and/or take any action based on this message or any information herein. If you have received this message in error, please inform the sender immediately and delete this message. Thank you for your cooperation. ' ' Esta mensagem pode conter informação confidencial e/ou privilegiada e é destinada exclusivamente ao indivíduo ou à entidade para a qual é endereçada. O remetente utiliza o correio eletrônico no exercício do seu trabalho ou em razão dele, eximindo a Companhia e as demais instituições integrantes de seu grupo econômico de qualquer responsabilidade por sua utilização indevida. Se você não for o destinatário ou a pessoa autorizada a receber esta mensagem, não deverá utilizar, armazenar, copiar, alterar e/ou divulgar a mensagem ou qualquer informação nela contida, ou tomar qualquer ação baseada nessas informações. Se você recebeu esta mensagem por engano, por favor, avise imediatamente o remetente, apagando a mensagem em seguida. Agradecemos sua cooperação. This message may contain confidential and/or privileged information and is intended to be received solely by the person or entity it is addressed to. The sender uses its e-mail box as a working tool, being the Company and the other entities that are part of its corporate group not liable for any misuse. If you are not the addressee or authorized to receive this for the addressee, you must not use, retain, copy, disclose, change and/or take any action based on this message or any information herein. If you have received this message in error, please inform the sender immediately and delete this message. Thank you for your cooperation. '
Re: Kafka on Windows
@M.Manna - thanks for your reply. I did read that document but clearly didn't digest that bit properly. @Liam Clarke I don't expect code's users (either maintainers or end-users) to need to deduce *anything*. If it needs knowing, I documented it. I'll send you an actual example so you know it's not all talk. >your 3 days you expended on learning the hard way that Windows is the minority >market share in servers I had been accepted for a job which was to be a major career change. For this I had to learn some basic linux (no, I've still never written a bash script), scala (in depth) and kafka, and spark as well if I could fit it in. IIRC I had less than 4 weeks to do it. To lose those 3 days mattered. > you're resenting a free open source project? Hardly, just the pointless loss of time. I also try to add back to the community rather than just take. I had been working on ANTLR stuff (which I'll return to when bits of me stop hurting) and currently am trying to suss if permutation encoding can be done from L1 cache for large permutations in less than a single DRAM access time. Look up cuckoo filters to see why. jan On 05/09/2018, Liam Clarke wrote: > Hi Jan, > > .sh is not .bat and easily googleable. > > Nothing is terrifying about updating the documentation, I just took umbrage > at you "resenting" your 3 days you expended on learning the hard way that > Windows is the minority market share in servers - you're resenting a free > open source project? Because you don't know what a .sh file signifies? > > I'm sorry mate, but that's on you. I code for *nix but I know what .bat or > .ps1 means. > > And given that 99.99% of FOSS is not built for Windows, you should always > assume the worst unless you downloaded it from Codeplex or via NuGet. > > Consider those three days valuable learning instead. > > Kind regards, > > Liam Clarke > > On Wed, 5 Sep. 2018, 9:22 pm jan, wrote: > >> Hi Liam, >> as a DB guy that does MSSQL (on windows, obviously) I literally have >> no idea what a .sh file is,or what that would imply. I guess it's bash >> so yeah. >> But what's so terrifying about actually stating in the FAQ that you >> should not run it as production on windows? >> Why should I conclude that kafka on windows, with java (which is >> supported on windows), and a filesystem (which windows has), and >> reliable networking (ditto), doesn't work? >> When is "just read the code" ever an acceptable answer? >> Why should I lose 3 days just because someone couldn't put "don't do >> it" in the FAQ (which I did read), and also have other people need to >> ask here whether windows is supported? >> Why? >> This is just nuts. >> >> cheers >> >> jan >> >> >> On 05/09/2018, Liam Clarke wrote: >> > Hi Jan, >> > >> > I'd presume that downloading an archive and seeing a bunch of .sh files >> > would imply that Kafka wasn't built to run on Windows. >> > >> > Given that it's a cluster based technology, I'd assume that it wouldn't >> be >> > supported for Windows given that most server based stuff isn't unless >> > Microsoft built it. >> > >> > Kind regards, >> > >> > Liam Clarke >> > >> > On Wed, 8 Aug. 2018, 2:42 am jan, >> wrote: >> > >> >> This is an excellent suggestion and I intend to do so henceforth >> >> (thanks!), but it would be an adjunct to my request rather than the >> >> answer; it still needs to be made clear in the docs/faq that you >> >> *can't* use windows directly. >> >> >> >> jan >> >> >> >> On 07/08/2018, Rahul Singh wrote: >> >> > I would recommend using Docker — it would end up being run on a >> >> > Linux >> >> kernel >> >> > VM on windows and is easier to get started on with a bit of learning >> >> curve >> >> > for Docker. Less time wasted overall and at least at that point you >> >> > would >> >> > know Docker. >> >> > >> >> > Rahul >> >> > On Aug 7, 2018, 4:50 AM -0400, jan , >> >> wrote: >> >> >> I tried using it just for learning a while back and wasted 3 days >> >> >> because it's not supported on windows. Even basic stuff didn't >> >> >> work. >> I >> >> >> did read the docs first! >> >> >> >> >> >> I think I
Re: Kafka on Windows
Hi Jan, .sh is not .bat and easily googleable. Nothing is terrifying about updating the documentation, I just took umbrage at you "resenting" your 3 days you expended on learning the hard way that Windows is the minority market share in servers - you're resenting a free open source project? Because you don't know what a .sh file signifies? I'm sorry mate, but that's on you. I code for *nix but I know what .bat or .ps1 means. And given that 99.99% of FOSS is not built for Windows, you should always assume the worst unless you downloaded it from Codeplex or via NuGet. Consider those three days valuable learning instead. Kind regards, Liam Clarke On Wed, 5 Sep. 2018, 9:22 pm jan, wrote: > Hi Liam, > as a DB guy that does MSSQL (on windows, obviously) I literally have > no idea what a .sh file is,or what that would imply. I guess it's bash > so yeah. > But what's so terrifying about actually stating in the FAQ that you > should not run it as production on windows? > Why should I conclude that kafka on windows, with java (which is > supported on windows), and a filesystem (which windows has), and > reliable networking (ditto), doesn't work? > When is "just read the code" ever an acceptable answer? > Why should I lose 3 days just because someone couldn't put "don't do > it" in the FAQ (which I did read), and also have other people need to > ask here whether windows is supported? > Why? > This is just nuts. > > cheers > > jan > > > On 05/09/2018, Liam Clarke wrote: > > Hi Jan, > > > > I'd presume that downloading an archive and seeing a bunch of .sh files > > would imply that Kafka wasn't built to run on Windows. > > > > Given that it's a cluster based technology, I'd assume that it wouldn't > be > > supported for Windows given that most server based stuff isn't unless > > Microsoft built it. > > > > Kind regards, > > > > Liam Clarke > > > > On Wed, 8 Aug. 2018, 2:42 am jan, > wrote: > > > >> This is an excellent suggestion and I intend to do so henceforth > >> (thanks!), but it would be an adjunct to my request rather than the > >> answer; it still needs to be made clear in the docs/faq that you > >> *can't* use windows directly. > >> > >> jan > >> > >> On 07/08/2018, Rahul Singh wrote: > >> > I would recommend using Docker — it would end up being run on a Linux > >> kernel > >> > VM on windows and is easier to get started on with a bit of learning > >> curve > >> > for Docker. Less time wasted overall and at least at that point you > >> > would > >> > know Docker. > >> > > >> > Rahul > >> > On Aug 7, 2018, 4:50 AM -0400, jan , > >> wrote: > >> >> I tried using it just for learning a while back and wasted 3 days > >> >> because it's not supported on windows. Even basic stuff didn't work. > I > >> >> did read the docs first! > >> >> > >> >> I think I've seen other people on this list have questions > >> >> about/problems for exactly the same reason, and that could be a lot > of > >> >> time saved if it was in the docs - it needs to be. So how do I ask > the > >> >> maintainers to put 'No, Not Windows" in there? > >> >> Serious question. > >> >> > >> >> I resent losing 3 days of work because of essential missing info. It > >> >> sounds like (compared to @M. Manna) that I got off lightly. > >> >> > >> >> So can we put a clear caveat in the documentation, please, right at > >> >> the > >> >> top? > >> >> > >> >> jan > >> >> > >> >> On 07/08/2018, M. Manna wrote: > >> >> > The answer is - Absolutely not. If you don’t have Linux rack, or > >> >> > Kubernetes > >> >> > deployment -it will not work on Windows as guaranteed. > >> >> > > >> >> > I know this because I have tried to make it work for the past 1 > >> >> > year. > >> >> > File > >> >> > handling always fails and crashes the cluster on Windows. > >> >> > > >> >> > Thanks, > >> >> > > >> >> > > >> >> > > >> >> > On Tue, 7 Aug 2018 at 01:54, Alew wrote: > >> >> > > >> >> > > Hi! > >> >> > > > >> >> > > Is it recommended to use production Kafka cluster on Windows? > >> >> > > > >> >> > > Can't get it from the documentation. It is possible to start > Kafka > >> on > >> >> > > Windows, but maybe it's for development purposes only. > >> >> > > > >> >> > > Thanks. > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > >> > > >> > > >
Re: Kafka on Windows
I believe this needs to be articulated a bit more, which is not a big issue. The common practice for a large-scale distributed system is to use Linux servers, and it's always been a popular choice. That doesn't mean the Windows is ignored. The issue at hand is that file deletion/renaming seems to have a different behaviour at OS core level when it comes to Windows. Different users including myself have tried different approaches to resolve this, but the issue still prevails. And unless people appreciates this already, Kafka is one of the most complex distributed systems today, with a very low-friction entry barrier (except streams !). So, fixing this platform-specific issue is a certainly a pain to deal with. And as a fair clarification - https://kafka.apache.org/documentation/#os can be referenced. Reading the docs is also a skill which I always struggle with :) I would request humbly that the disclaimer is made slightly more explicit, but since most of the users have been adopting K8s/bare-metal linux for deployment, I suspect we won't know whether Windows will be supported in the future at all. Regards, On Wed, 5 Sep 2018 at 10:22, jan wrote: > Hi Liam, > as a DB guy that does MSSQL (on windows, obviously) I literally have > no idea what a .sh file is,or what that would imply. I guess it's bash > so yeah. > But what's so terrifying about actually stating in the FAQ that you > should not run it as production on windows? > Why should I conclude that kafka on windows, with java (which is > supported on windows), and a filesystem (which windows has), and > reliable networking (ditto), doesn't work? > When is "just read the code" ever an acceptable answer? > Why should I lose 3 days just because someone couldn't put "don't do > it" in the FAQ (which I did read), and also have other people need to > ask here whether windows is supported? > Why? > This is just nuts. > > cheers > > jan > > > On 05/09/2018, Liam Clarke wrote: > > Hi Jan, > > > > I'd presume that downloading an archive and seeing a bunch of .sh files > > would imply that Kafka wasn't built to run on Windows. > > > > Given that it's a cluster based technology, I'd assume that it wouldn't > be > > supported for Windows given that most server based stuff isn't unless > > Microsoft built it. > > > > Kind regards, > > > > Liam Clarke > > > > On Wed, 8 Aug. 2018, 2:42 am jan, > wrote: > > > >> This is an excellent suggestion and I intend to do so henceforth > >> (thanks!), but it would be an adjunct to my request rather than the > >> answer; it still needs to be made clear in the docs/faq that you > >> *can't* use windows directly. > >> > >> jan > >> > >> On 07/08/2018, Rahul Singh wrote: > >> > I would recommend using Docker — it would end up being run on a Linux > >> kernel > >> > VM on windows and is easier to get started on with a bit of learning > >> curve > >> > for Docker. Less time wasted overall and at least at that point you > >> > would > >> > know Docker. > >> > > >> > Rahul > >> > On Aug 7, 2018, 4:50 AM -0400, jan , > >> wrote: > >> >> I tried using it just for learning a while back and wasted 3 days > >> >> because it's not supported on windows. Even basic stuff didn't work. > I > >> >> did read the docs first! > >> >> > >> >> I think I've seen other people on this list have questions > >> >> about/problems for exactly the same reason, and that could be a lot > of > >> >> time saved if it was in the docs - it needs to be. So how do I ask > the > >> >> maintainers to put 'No, Not Windows" in there? > >> >> Serious question. > >> >> > >> >> I resent losing 3 days of work because of essential missing info. It > >> >> sounds like (compared to @M. Manna) that I got off lightly. > >> >> > >> >> So can we put a clear caveat in the documentation, please, right at > >> >> the > >> >> top? > >> >> > >> >> jan > >> >> > >> >> On 07/08/2018, M. Manna wrote: > >> >> > The answer is - Absolutely not. If you don’t have Linux rack, or > >> >> > Kubernetes > >> >> > deployment -it will not work on Windows as guaranteed. > >> >> > > >> >> > I know this because I have tried to make it work for the past 1 > >> >> > year. > >> >> > File > >> >> > handling always fails and crashes the cluster on Windows. > >> >> > > >> >> > Thanks, > >> >> > > >> >> > > >> >> > > >> >> > On Tue, 7 Aug 2018 at 01:54, Alew wrote: > >> >> > > >> >> > > Hi! > >> >> > > > >> >> > > Is it recommended to use production Kafka cluster on Windows? > >> >> > > > >> >> > > Can't get it from the documentation. It is possible to start > Kafka > >> on > >> >> > > Windows, but maybe it's for development purposes only. > >> >> > > > >> >> > > Thanks. > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > >> > > >> > > >
Re: Kafka on Windows
Hi Liam, as a DB guy that does MSSQL (on windows, obviously) I literally have no idea what a .sh file is,or what that would imply. I guess it's bash so yeah. But what's so terrifying about actually stating in the FAQ that you should not run it as production on windows? Why should I conclude that kafka on windows, with java (which is supported on windows), and a filesystem (which windows has), and reliable networking (ditto), doesn't work? When is "just read the code" ever an acceptable answer? Why should I lose 3 days just because someone couldn't put "don't do it" in the FAQ (which I did read), and also have other people need to ask here whether windows is supported? Why? This is just nuts. cheers jan On 05/09/2018, Liam Clarke wrote: > Hi Jan, > > I'd presume that downloading an archive and seeing a bunch of .sh files > would imply that Kafka wasn't built to run on Windows. > > Given that it's a cluster based technology, I'd assume that it wouldn't be > supported for Windows given that most server based stuff isn't unless > Microsoft built it. > > Kind regards, > > Liam Clarke > > On Wed, 8 Aug. 2018, 2:42 am jan, wrote: > >> This is an excellent suggestion and I intend to do so henceforth >> (thanks!), but it would be an adjunct to my request rather than the >> answer; it still needs to be made clear in the docs/faq that you >> *can't* use windows directly. >> >> jan >> >> On 07/08/2018, Rahul Singh wrote: >> > I would recommend using Docker — it would end up being run on a Linux >> kernel >> > VM on windows and is easier to get started on with a bit of learning >> curve >> > for Docker. Less time wasted overall and at least at that point you >> > would >> > know Docker. >> > >> > Rahul >> > On Aug 7, 2018, 4:50 AM -0400, jan , >> wrote: >> >> I tried using it just for learning a while back and wasted 3 days >> >> because it's not supported on windows. Even basic stuff didn't work. I >> >> did read the docs first! >> >> >> >> I think I've seen other people on this list have questions >> >> about/problems for exactly the same reason, and that could be a lot of >> >> time saved if it was in the docs - it needs to be. So how do I ask the >> >> maintainers to put 'No, Not Windows" in there? >> >> Serious question. >> >> >> >> I resent losing 3 days of work because of essential missing info. It >> >> sounds like (compared to @M. Manna) that I got off lightly. >> >> >> >> So can we put a clear caveat in the documentation, please, right at >> >> the >> >> top? >> >> >> >> jan >> >> >> >> On 07/08/2018, M. Manna wrote: >> >> > The answer is - Absolutely not. If you don’t have Linux rack, or >> >> > Kubernetes >> >> > deployment -it will not work on Windows as guaranteed. >> >> > >> >> > I know this because I have tried to make it work for the past 1 >> >> > year. >> >> > File >> >> > handling always fails and crashes the cluster on Windows. >> >> > >> >> > Thanks, >> >> > >> >> > >> >> > >> >> > On Tue, 7 Aug 2018 at 01:54, Alew wrote: >> >> > >> >> > > Hi! >> >> > > >> >> > > Is it recommended to use production Kafka cluster on Windows? >> >> > > >> >> > > Can't get it from the documentation. It is possible to start Kafka >> on >> >> > > Windows, but maybe it's for development purposes only. >> >> > > >> >> > > Thanks. >> >> > > >> >> > > >> >> > > >> >> > > >> >> > >> > >> >
Re: Kafka on Windows
Hi Jan, I'd presume that downloading an archive and seeing a bunch of .sh files would imply that Kafka wasn't built to run on Windows. Given that it's a cluster based technology, I'd assume that it wouldn't be supported for Windows given that most server based stuff isn't unless Microsoft built it. Kind regards, Liam Clarke On Wed, 8 Aug. 2018, 2:42 am jan, wrote: > This is an excellent suggestion and I intend to do so henceforth > (thanks!), but it would be an adjunct to my request rather than the > answer; it still needs to be made clear in the docs/faq that you > *can't* use windows directly. > > jan > > On 07/08/2018, Rahul Singh wrote: > > I would recommend using Docker — it would end up being run on a Linux > kernel > > VM on windows and is easier to get started on with a bit of learning > curve > > for Docker. Less time wasted overall and at least at that point you would > > know Docker. > > > > Rahul > > On Aug 7, 2018, 4:50 AM -0400, jan , > wrote: > >> I tried using it just for learning a while back and wasted 3 days > >> because it's not supported on windows. Even basic stuff didn't work. I > >> did read the docs first! > >> > >> I think I've seen other people on this list have questions > >> about/problems for exactly the same reason, and that could be a lot of > >> time saved if it was in the docs - it needs to be. So how do I ask the > >> maintainers to put 'No, Not Windows" in there? > >> Serious question. > >> > >> I resent losing 3 days of work because of essential missing info. It > >> sounds like (compared to @M. Manna) that I got off lightly. > >> > >> So can we put a clear caveat in the documentation, please, right at the > >> top? > >> > >> jan > >> > >> On 07/08/2018, M. Manna wrote: > >> > The answer is - Absolutely not. If you don’t have Linux rack, or > >> > Kubernetes > >> > deployment -it will not work on Windows as guaranteed. > >> > > >> > I know this because I have tried to make it work for the past 1 year. > >> > File > >> > handling always fails and crashes the cluster on Windows. > >> > > >> > Thanks, > >> > > >> > > >> > > >> > On Tue, 7 Aug 2018 at 01:54, Alew wrote: > >> > > >> > > Hi! > >> > > > >> > > Is it recommended to use production Kafka cluster on Windows? > >> > > > >> > > Can't get it from the documentation. It is possible to start Kafka > on > >> > > Windows, but maybe it's for development purposes only. > >> > > > >> > > Thanks. > >> > > > >> > > > >> > > > >> > > > >> > > > >
Re: Kafka on Windows
I've been using Kafka 0.10.x version on windows in production from past 1 year and plan to move to higher version soon . @M. Manna<mailto:manme...@gmail.com> By Log cleaner functionality - Did you mean compaction in topics? I've not seen this issue. Did you file a bug on this? Can you share the link? Thanks, Koushik From: M. Manna Sent: Wednesday, August 8, 2018 4:49 AM To: Kafka Users Subject: Re: Kafka on Windows Using docker is probably the best option. The one thing I cannot confirm (and I haven't tried yet) is how the volumes will behave when you are transferring the topic log offsets to your windows drive. Hopefully, this will be fine but do let us know if someone has tried. Thanks, On 7 August 2018 at 23:33, jan wrote: > @Jacob Sheck: It was 18 months ago so I don't recall but I was told > clearly windows ain't supported when I reported problems. > > I don't know if my problems were down to my inexperience or my use of > windows, but I basically assuming I was controlling for one variable > (n00bish inexperience) but was informed I now had two (n00bness, and > an unsupported platform). > > tl;dr if it doesn't work on X, we need to say so clearly. It's just... > good manners, surely? > > cheers > > jan > > > On 07/08/2018, M. Manna wrote: > > By fully broken, i mean not designed and tested to work on Windows. > > > > On Tue, 7 Aug 2018, 16:34 M. Manna, wrote: > > > >> Log Cleaner functionality is fully broken... If you haven't tried that > >> already. > >> > >> > >> > >> On 7 Aug 2018 4:24 pm, "Jacob Sheck" wrote: > >> > >> While I agree that it is less frustrating to run Kafka on Linux, I am > >> interested to hear what specific issues you are running into on windows? > >> > >> > >> On Tue, Aug 7, 2018 at 9:42 AM jan > >> wrote: > >> > >> > This is an excellent suggestion and I intend to do so henceforth > >> > (thanks!), but it would be an adjunct to my request rather than the > >> > answer; it still needs to be made clear in the docs/faq that you > >> > *can't* use windows directly. > >> > > >> > jan > >> > > >> > On 07/08/2018, Rahul Singh wrote: > >> > > I would recommend using Docker — it would end up being run on a > Linux > >> > kernel > >> > > VM on windows and is easier to get started on with a bit of learning > >> > curve > >> > > for Docker. Less time wasted overall and at least at that point you > >> would > >> > > know Docker. > >> > > > >> > > Rahul > >> > > On Aug 7, 2018, 4:50 AM -0400, jan invalid>, > >> > wrote: > >> > >> I tried using it just for learning a while back and wasted 3 days > >> > >> because it's not supported on windows. Even basic stuff didn't > work. > >> > >> I > >> > >> did read the docs first! > >> > >> > >> > >> I think I've seen other people on this list have questions > >> > >> about/problems for exactly the same reason, and that could be a lot > >> > >> of > >> > >> time saved if it was in the docs - it needs to be. So how do I ask > >> > >> the > >> > >> maintainers to put 'No, Not Windows" in there? > >> > >> Serious question. > >> > >> > >> > >> I resent losing 3 days of work because of essential missing info. > It > >> > >> sounds like (compared to @M. Manna) that I got off lightly. > >> > >> > >> > >> So can we put a clear caveat in the documentation, please, right at > >> the > >> > >> top? > >> > >> > >> > >> jan > >> > >> > >> > >> On 07/08/2018, M. Manna wrote: > >> > >> > The answer is - Absolutely not. If you don’t have Linux rack, or > >> > >> > Kubernetes > >> > >> > deployment -it will not work on Windows as guaranteed. > >> > >> > > >> > >> > I know this because I have tried to make it work for the past 1 > >> year. > >> > >> > File > >> > >> > handling always fails and crashes the cluster on Windows. > >> > >> > > >> > >> > Thanks, > >> > >> > > >> > >> > > >> > >> > > >> > >> > On Tue, 7 Aug 2018 at 01:54, Alew wrote: > >> > >> > > >> > >> > > Hi! > >> > >> > > > >> > >> > > Is it recommended to use production Kafka cluster on Windows? > >> > >> > > > >> > >> > > Can't get it from the documentation. It is possible to start > >> > >> > > Kafka > >> > on > >> > >> > > Windows, but maybe it's for development purposes only. > >> > >> > > > >> > >> > > Thanks. > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > >> > > > >> > > >> > >> > >> > > >
Re: Kafka on Windows
Using docker is probably the best option. The one thing I cannot confirm (and I haven't tried yet) is how the volumes will behave when you are transferring the topic log offsets to your windows drive. Hopefully, this will be fine but do let us know if someone has tried. Thanks, On 7 August 2018 at 23:33, jan wrote: > @Jacob Sheck: It was 18 months ago so I don't recall but I was told > clearly windows ain't supported when I reported problems. > > I don't know if my problems were down to my inexperience or my use of > windows, but I basically assuming I was controlling for one variable > (n00bish inexperience) but was informed I now had two (n00bness, and > an unsupported platform). > > tl;dr if it doesn't work on X, we need to say so clearly. It's just... > good manners, surely? > > cheers > > jan > > > On 07/08/2018, M. Manna wrote: > > By fully broken, i mean not designed and tested to work on Windows. > > > > On Tue, 7 Aug 2018, 16:34 M. Manna, wrote: > > > >> Log Cleaner functionality is fully broken... If you haven't tried that > >> already. > >> > >> > >> > >> On 7 Aug 2018 4:24 pm, "Jacob Sheck" wrote: > >> > >> While I agree that it is less frustrating to run Kafka on Linux, I am > >> interested to hear what specific issues you are running into on windows? > >> > >> > >> On Tue, Aug 7, 2018 at 9:42 AM jan > >> wrote: > >> > >> > This is an excellent suggestion and I intend to do so henceforth > >> > (thanks!), but it would be an adjunct to my request rather than the > >> > answer; it still needs to be made clear in the docs/faq that you > >> > *can't* use windows directly. > >> > > >> > jan > >> > > >> > On 07/08/2018, Rahul Singh wrote: > >> > > I would recommend using Docker — it would end up being run on a > Linux > >> > kernel > >> > > VM on windows and is easier to get started on with a bit of learning > >> > curve > >> > > for Docker. Less time wasted overall and at least at that point you > >> would > >> > > know Docker. > >> > > > >> > > Rahul > >> > > On Aug 7, 2018, 4:50 AM -0400, jan invalid>, > >> > wrote: > >> > >> I tried using it just for learning a while back and wasted 3 days > >> > >> because it's not supported on windows. Even basic stuff didn't > work. > >> > >> I > >> > >> did read the docs first! > >> > >> > >> > >> I think I've seen other people on this list have questions > >> > >> about/problems for exactly the same reason, and that could be a lot > >> > >> of > >> > >> time saved if it was in the docs - it needs to be. So how do I ask > >> > >> the > >> > >> maintainers to put 'No, Not Windows" in there? > >> > >> Serious question. > >> > >> > >> > >> I resent losing 3 days of work because of essential missing info. > It > >> > >> sounds like (compared to @M. Manna) that I got off lightly. > >> > >> > >> > >> So can we put a clear caveat in the documentation, please, right at > >> the > >> > >> top? > >> > >> > >> > >> jan > >> > >> > >> > >> On 07/08/2018, M. Manna wrote: > >> > >> > The answer is - Absolutely not. If you don’t have Linux rack, or > >> > >> > Kubernetes > >> > >> > deployment -it will not work on Windows as guaranteed. > >> > >> > > >> > >> > I know this because I have tried to make it work for the past 1 > >> year. > >> > >> > File > >> > >> > handling always fails and crashes the cluster on Windows. > >> > >> > > >> > >> > Thanks, > >> > >> > > >> > >> > > >> > >> > > >> > >> > On Tue, 7 Aug 2018 at 01:54, Alew wrote: > >> > >> > > >> > >> > > Hi! > >> > >> > > > >> > >> > > Is it recommended to use production Kafka cluster on Windows? > >> > >> > > > >> > >> > > Can't get it from the documentation. It is possible to start > >> > >> > > Kafka > >> > on > >> > >> > > Windows, but maybe it's for development purposes only. > >> > >> > > > >> > >> > > Thanks. > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > >> > > > >> > > >> > >> > >> > > >
Re: Kafka on Windows
@Jacob Sheck: It was 18 months ago so I don't recall but I was told clearly windows ain't supported when I reported problems. I don't know if my problems were down to my inexperience or my use of windows, but I basically assuming I was controlling for one variable (n00bish inexperience) but was informed I now had two (n00bness, and an unsupported platform). tl;dr if it doesn't work on X, we need to say so clearly. It's just... good manners, surely? cheers jan On 07/08/2018, M. Manna wrote: > By fully broken, i mean not designed and tested to work on Windows. > > On Tue, 7 Aug 2018, 16:34 M. Manna, wrote: > >> Log Cleaner functionality is fully broken... If you haven't tried that >> already. >> >> >> >> On 7 Aug 2018 4:24 pm, "Jacob Sheck" wrote: >> >> While I agree that it is less frustrating to run Kafka on Linux, I am >> interested to hear what specific issues you are running into on windows? >> >> >> On Tue, Aug 7, 2018 at 9:42 AM jan >> wrote: >> >> > This is an excellent suggestion and I intend to do so henceforth >> > (thanks!), but it would be an adjunct to my request rather than the >> > answer; it still needs to be made clear in the docs/faq that you >> > *can't* use windows directly. >> > >> > jan >> > >> > On 07/08/2018, Rahul Singh wrote: >> > > I would recommend using Docker — it would end up being run on a Linux >> > kernel >> > > VM on windows and is easier to get started on with a bit of learning >> > curve >> > > for Docker. Less time wasted overall and at least at that point you >> would >> > > know Docker. >> > > >> > > Rahul >> > > On Aug 7, 2018, 4:50 AM -0400, jan , >> > wrote: >> > >> I tried using it just for learning a while back and wasted 3 days >> > >> because it's not supported on windows. Even basic stuff didn't work. >> > >> I >> > >> did read the docs first! >> > >> >> > >> I think I've seen other people on this list have questions >> > >> about/problems for exactly the same reason, and that could be a lot >> > >> of >> > >> time saved if it was in the docs - it needs to be. So how do I ask >> > >> the >> > >> maintainers to put 'No, Not Windows" in there? >> > >> Serious question. >> > >> >> > >> I resent losing 3 days of work because of essential missing info. It >> > >> sounds like (compared to @M. Manna) that I got off lightly. >> > >> >> > >> So can we put a clear caveat in the documentation, please, right at >> the >> > >> top? >> > >> >> > >> jan >> > >> >> > >> On 07/08/2018, M. Manna wrote: >> > >> > The answer is - Absolutely not. If you don’t have Linux rack, or >> > >> > Kubernetes >> > >> > deployment -it will not work on Windows as guaranteed. >> > >> > >> > >> > I know this because I have tried to make it work for the past 1 >> year. >> > >> > File >> > >> > handling always fails and crashes the cluster on Windows. >> > >> > >> > >> > Thanks, >> > >> > >> > >> > >> > >> > >> > >> > On Tue, 7 Aug 2018 at 01:54, Alew wrote: >> > >> > >> > >> > > Hi! >> > >> > > >> > >> > > Is it recommended to use production Kafka cluster on Windows? >> > >> > > >> > >> > > Can't get it from the documentation. It is possible to start >> > >> > > Kafka >> > on >> > >> > > Windows, but maybe it's for development purposes only. >> > >> > > >> > >> > > Thanks. >> > >> > > >> > >> > > >> > >> > > >> > >> > > >> > >> > >> > > >> > >> >> >> >
Re: Kafka on Windows
By fully broken, i mean not designed and tested to work on Windows. On Tue, 7 Aug 2018, 16:34 M. Manna, wrote: > Log Cleaner functionality is fully broken... If you haven't tried that > already. > > > > On 7 Aug 2018 4:24 pm, "Jacob Sheck" wrote: > > While I agree that it is less frustrating to run Kafka on Linux, I am > interested to hear what specific issues you are running into on windows? > > > On Tue, Aug 7, 2018 at 9:42 AM jan wrote: > > > This is an excellent suggestion and I intend to do so henceforth > > (thanks!), but it would be an adjunct to my request rather than the > > answer; it still needs to be made clear in the docs/faq that you > > *can't* use windows directly. > > > > jan > > > > On 07/08/2018, Rahul Singh wrote: > > > I would recommend using Docker — it would end up being run on a Linux > > kernel > > > VM on windows and is easier to get started on with a bit of learning > > curve > > > for Docker. Less time wasted overall and at least at that point you > would > > > know Docker. > > > > > > Rahul > > > On Aug 7, 2018, 4:50 AM -0400, jan , > > wrote: > > >> I tried using it just for learning a while back and wasted 3 days > > >> because it's not supported on windows. Even basic stuff didn't work. I > > >> did read the docs first! > > >> > > >> I think I've seen other people on this list have questions > > >> about/problems for exactly the same reason, and that could be a lot of > > >> time saved if it was in the docs - it needs to be. So how do I ask the > > >> maintainers to put 'No, Not Windows" in there? > > >> Serious question. > > >> > > >> I resent losing 3 days of work because of essential missing info. It > > >> sounds like (compared to @M. Manna) that I got off lightly. > > >> > > >> So can we put a clear caveat in the documentation, please, right at > the > > >> top? > > >> > > >> jan > > >> > > >> On 07/08/2018, M. Manna wrote: > > >> > The answer is - Absolutely not. If you don’t have Linux rack, or > > >> > Kubernetes > > >> > deployment -it will not work on Windows as guaranteed. > > >> > > > >> > I know this because I have tried to make it work for the past 1 > year. > > >> > File > > >> > handling always fails and crashes the cluster on Windows. > > >> > > > >> > Thanks, > > >> > > > >> > > > >> > > > >> > On Tue, 7 Aug 2018 at 01:54, Alew wrote: > > >> > > > >> > > Hi! > > >> > > > > >> > > Is it recommended to use production Kafka cluster on Windows? > > >> > > > > >> > > Can't get it from the documentation. It is possible to start Kafka > > on > > >> > > Windows, but maybe it's for development purposes only. > > >> > > > > >> > > Thanks. > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > > > > > >
Re: Kafka on Windows
Log Cleaner functionality is fully broken... If you haven't tried that already. On 7 Aug 2018 4:24 pm, "Jacob Sheck" wrote: While I agree that it is less frustrating to run Kafka on Linux, I am interested to hear what specific issues you are running into on windows? On Tue, Aug 7, 2018 at 9:42 AM jan wrote: > This is an excellent suggestion and I intend to do so henceforth > (thanks!), but it would be an adjunct to my request rather than the > answer; it still needs to be made clear in the docs/faq that you > *can't* use windows directly. > > jan > > On 07/08/2018, Rahul Singh wrote: > > I would recommend using Docker — it would end up being run on a Linux > kernel > > VM on windows and is easier to get started on with a bit of learning > curve > > for Docker. Less time wasted overall and at least at that point you would > > know Docker. > > > > Rahul > > On Aug 7, 2018, 4:50 AM -0400, jan , > wrote: > >> I tried using it just for learning a while back and wasted 3 days > >> because it's not supported on windows. Even basic stuff didn't work. I > >> did read the docs first! > >> > >> I think I've seen other people on this list have questions > >> about/problems for exactly the same reason, and that could be a lot of > >> time saved if it was in the docs - it needs to be. So how do I ask the > >> maintainers to put 'No, Not Windows" in there? > >> Serious question. > >> > >> I resent losing 3 days of work because of essential missing info. It > >> sounds like (compared to @M. Manna) that I got off lightly. > >> > >> So can we put a clear caveat in the documentation, please, right at the > >> top? > >> > >> jan > >> > >> On 07/08/2018, M. Manna wrote: > >> > The answer is - Absolutely not. If you don’t have Linux rack, or > >> > Kubernetes > >> > deployment -it will not work on Windows as guaranteed. > >> > > >> > I know this because I have tried to make it work for the past 1 year. > >> > File > >> > handling always fails and crashes the cluster on Windows. > >> > > >> > Thanks, > >> > > >> > > >> > > >> > On Tue, 7 Aug 2018 at 01:54, Alew wrote: > >> > > >> > > Hi! > >> > > > >> > > Is it recommended to use production Kafka cluster on Windows? > >> > > > >> > > Can't get it from the documentation. It is possible to start Kafka > on > >> > > Windows, but maybe it's for development purposes only. > >> > > > >> > > Thanks. > >> > > > >> > > > >> > > > >> > > > >> > > > >
Re: Kafka on Windows
While I agree that it is less frustrating to run Kafka on Linux, I am interested to hear what specific issues you are running into on windows? On Tue, Aug 7, 2018 at 9:42 AM jan wrote: > This is an excellent suggestion and I intend to do so henceforth > (thanks!), but it would be an adjunct to my request rather than the > answer; it still needs to be made clear in the docs/faq that you > *can't* use windows directly. > > jan > > On 07/08/2018, Rahul Singh wrote: > > I would recommend using Docker — it would end up being run on a Linux > kernel > > VM on windows and is easier to get started on with a bit of learning > curve > > for Docker. Less time wasted overall and at least at that point you would > > know Docker. > > > > Rahul > > On Aug 7, 2018, 4:50 AM -0400, jan , > wrote: > >> I tried using it just for learning a while back and wasted 3 days > >> because it's not supported on windows. Even basic stuff didn't work. I > >> did read the docs first! > >> > >> I think I've seen other people on this list have questions > >> about/problems for exactly the same reason, and that could be a lot of > >> time saved if it was in the docs - it needs to be. So how do I ask the > >> maintainers to put 'No, Not Windows" in there? > >> Serious question. > >> > >> I resent losing 3 days of work because of essential missing info. It > >> sounds like (compared to @M. Manna) that I got off lightly. > >> > >> So can we put a clear caveat in the documentation, please, right at the > >> top? > >> > >> jan > >> > >> On 07/08/2018, M. Manna wrote: > >> > The answer is - Absolutely not. If you don’t have Linux rack, or > >> > Kubernetes > >> > deployment -it will not work on Windows as guaranteed. > >> > > >> > I know this because I have tried to make it work for the past 1 year. > >> > File > >> > handling always fails and crashes the cluster on Windows. > >> > > >> > Thanks, > >> > > >> > > >> > > >> > On Tue, 7 Aug 2018 at 01:54, Alew wrote: > >> > > >> > > Hi! > >> > > > >> > > Is it recommended to use production Kafka cluster on Windows? > >> > > > >> > > Can't get it from the documentation. It is possible to start Kafka > on > >> > > Windows, but maybe it's for development purposes only. > >> > > > >> > > Thanks. > >> > > > >> > > > >> > > > >> > > > >> > > > >
Re: Kafka on Windows
This is an excellent suggestion and I intend to do so henceforth (thanks!), but it would be an adjunct to my request rather than the answer; it still needs to be made clear in the docs/faq that you *can't* use windows directly. jan On 07/08/2018, Rahul Singh wrote: > I would recommend using Docker — it would end up being run on a Linux kernel > VM on windows and is easier to get started on with a bit of learning curve > for Docker. Less time wasted overall and at least at that point you would > know Docker. > > Rahul > On Aug 7, 2018, 4:50 AM -0400, jan , wrote: >> I tried using it just for learning a while back and wasted 3 days >> because it's not supported on windows. Even basic stuff didn't work. I >> did read the docs first! >> >> I think I've seen other people on this list have questions >> about/problems for exactly the same reason, and that could be a lot of >> time saved if it was in the docs - it needs to be. So how do I ask the >> maintainers to put 'No, Not Windows" in there? >> Serious question. >> >> I resent losing 3 days of work because of essential missing info. It >> sounds like (compared to @M. Manna) that I got off lightly. >> >> So can we put a clear caveat in the documentation, please, right at the >> top? >> >> jan >> >> On 07/08/2018, M. Manna wrote: >> > The answer is - Absolutely not. If you don’t have Linux rack, or >> > Kubernetes >> > deployment -it will not work on Windows as guaranteed. >> > >> > I know this because I have tried to make it work for the past 1 year. >> > File >> > handling always fails and crashes the cluster on Windows. >> > >> > Thanks, >> > >> > >> > >> > On Tue, 7 Aug 2018 at 01:54, Alew wrote: >> > >> > > Hi! >> > > >> > > Is it recommended to use production Kafka cluster on Windows? >> > > >> > > Can't get it from the documentation. It is possible to start Kafka on >> > > Windows, but maybe it's for development purposes only. >> > > >> > > Thanks. >> > > >> > > >> > > >> > > >> > >
Re: Kafka on Windows
I would recommend using Docker — it would end up being run on a Linux kernel VM on windows and is easier to get started on with a bit of learning curve for Docker. Less time wasted overall and at least at that point you would know Docker. Rahul On Aug 7, 2018, 4:50 AM -0400, jan , wrote: > I tried using it just for learning a while back and wasted 3 days > because it's not supported on windows. Even basic stuff didn't work. I > did read the docs first! > > I think I've seen other people on this list have questions > about/problems for exactly the same reason, and that could be a lot of > time saved if it was in the docs - it needs to be. So how do I ask the > maintainers to put 'No, Not Windows" in there? > Serious question. > > I resent losing 3 days of work because of essential missing info. It > sounds like (compared to @M. Manna) that I got off lightly. > > So can we put a clear caveat in the documentation, please, right at the top? > > jan > > On 07/08/2018, M. Manna wrote: > > The answer is - Absolutely not. If you don’t have Linux rack, or Kubernetes > > deployment -it will not work on Windows as guaranteed. > > > > I know this because I have tried to make it work for the past 1 year. File > > handling always fails and crashes the cluster on Windows. > > > > Thanks, > > > > > > > > On Tue, 7 Aug 2018 at 01:54, Alew wrote: > > > > > Hi! > > > > > > Is it recommended to use production Kafka cluster on Windows? > > > > > > Can't get it from the documentation. It is possible to start Kafka on > > > Windows, but maybe it's for development purposes only. > > > > > > Thanks. > > > > > > > > > > > > > >
Re: Kafka on Windows
I tried using it just for learning a while back and wasted 3 days because it's not supported on windows. Even basic stuff didn't work. I did read the docs first! I think I've seen other people on this list have questions about/problems for exactly the same reason, and that could be a lot of time saved if it was in the docs - it needs to be. So how do I ask the maintainers to put 'No, Not Windows" in there? Serious question. I resent losing 3 days of work because of essential missing info. It sounds like (compared to @M. Manna) that I got off lightly. So can we put a clear caveat in the documentation, please, right at the top? jan On 07/08/2018, M. Manna wrote: > The answer is - Absolutely not. If you don’t have Linux rack, or Kubernetes > deployment -it will not work on Windows as guaranteed. > > I know this because I have tried to make it work for the past 1 year. File > handling always fails and crashes the cluster on Windows. > > Thanks, > > > > On Tue, 7 Aug 2018 at 01:54, Alew wrote: > >> Hi! >> >> Is it recommended to use production Kafka cluster on Windows? >> >> Can't get it from the documentation. It is possible to start Kafka on >> Windows, but maybe it's for development purposes only. >> >> Thanks. >> >> >> >> >
Re: Kafka on Windows
The answer is - Absolutely not. If you don’t have Linux rack, or Kubernetes deployment -it will not work on Windows as guaranteed. I know this because I have tried to make it work for the past 1 year. File handling always fails and crashes the cluster on Windows. Thanks, On Tue, 7 Aug 2018 at 01:54, Alew wrote: > Hi! > > Is it recommended to use production Kafka cluster on Windows? > > Can't get it from the documentation. It is possible to start Kafka on > Windows, but maybe it's for development purposes only. > > Thanks. > > > >
Kafka on Windows
Hi! Is it recommended to use production Kafka cluster on Windows? Can't get it from the documentation. It is possible to start Kafka on Windows, but maybe it's for development purposes only. Thanks.
Re: Kafka on windows
Kafka uses Linux OS page cache and flushing techniques as well as in-sync replicas to bring down latency since storing to the hard drive is likely the slowest part. It seems it would be defeating the point if you were to install it on Windows, just like everyone here mentioned. ;) Just wanted to add some links for your reference, the second one is pretty good: https://kafka.apache.org/08/documentation.html#linuxflush https://www.quora.com/Kafka-writes-every-message-to-broker-disk-Still-performance-wise-it-is-better-than-some-of-the-in-memory-message-storing-message-queues-Why-is-that On Fri, Apr 14, 2017 at 10:13 AM, David Garcia wrote: > If you want reduce maintenance headaches, I highly recommend not running > brokers on windows (or in vm’s for that matter). It really isn’t > supported. Kafka needs highly performant OS primitives. For example, XFS, > and ext4 are recommended filesystems for reasons specific to Kafka. If > this is just for experimentation/curiosity, then go all out…but production > systems should stick with the documentation’s recommendations. > > On 4/12/17, 12:13 AM, "David Luu" wrote: > > I'm curious as well. That doc blurb doesn't give specifics. How is > kafka > run (or tested) on Windows? Natively via the command line shell, etc. > or > via cygwin, within a *nix VM on Windows, or via Windows 10's Ubuntu > Linux > Bash shell? Would be interesting to see how each method I listed > performs, > maybe the Windows 10 bash shell method might be most optimal among the > list? > > On Tue, Apr 11, 2017 at 9:41 PM, David Garcia > wrote: > > > One issue is that Kafka leverage some very specific features of the > linux > > kernel that are probably far different from Windows, so I imagine the > > performance profile is likewise much different. > > > > On 4/11/17, 8:52 AM, "Tomasz Rojek" > wrote: > > > > Hi All, > > > > We want to choose provider of messaging system in our company, > one of > > possible choices is Apache Kafka. One of operating system that > will > > host > > brokers is windows, according to documentation: > > > > https://kafka.apache.org/documentation.html#os > > "*We have seen a few issues running on Windows and Windows is not > > currently > > a well supported platform though we would be happy to change > that.*" > > > > Can you please elaborate more on this. What exactly potential > issues > > are we > > talking about? What functionalities of kafka are influenced by > this? > > Maybe > > it occurs only on specific version of windows? > > > > Thank you in advance for any information. > > > > With Regards > > Tomasz Rojek > > Java Engineer > > > > > > > > > -- > David Luu > Member of Technical Staff > Mist Systems, Inc. > 1601 S. De Anza Blvd. #248 > Cupertino, CA 95014 > > >
Re: Kafka on windows
If you want reduce maintenance headaches, I highly recommend not running brokers on windows (or in vm’s for that matter). It really isn’t supported. Kafka needs highly performant OS primitives. For example, XFS, and ext4 are recommended filesystems for reasons specific to Kafka. If this is just for experimentation/curiosity, then go all out…but production systems should stick with the documentation’s recommendations. On 4/12/17, 12:13 AM, "David Luu" wrote: I'm curious as well. That doc blurb doesn't give specifics. How is kafka run (or tested) on Windows? Natively via the command line shell, etc. or via cygwin, within a *nix VM on Windows, or via Windows 10's Ubuntu Linux Bash shell? Would be interesting to see how each method I listed performs, maybe the Windows 10 bash shell method might be most optimal among the list? On Tue, Apr 11, 2017 at 9:41 PM, David Garcia wrote: > One issue is that Kafka leverage some very specific features of the linux > kernel that are probably far different from Windows, so I imagine the > performance profile is likewise much different. > > On 4/11/17, 8:52 AM, "Tomasz Rojek" wrote: > > Hi All, > > We want to choose provider of messaging system in our company, one of > possible choices is Apache Kafka. One of operating system that will > host > brokers is windows, according to documentation: > > https://kafka.apache.org/documentation.html#os > "*We have seen a few issues running on Windows and Windows is not > currently > a well supported platform though we would be happy to change that.*" > > Can you please elaborate more on this. What exactly potential issues > are we > talking about? What functionalities of kafka are influenced by this? > Maybe > it occurs only on specific version of windows? > > Thank you in advance for any information. > > With Regards > Tomasz Rojek > Java Engineer > > > -- David Luu Member of Technical Staff Mist Systems, Inc. 1601 S. De Anza Blvd. #248 Cupertino, CA 95014
Re: Kafka on windows
I'm curious as well. That doc blurb doesn't give specifics. How is kafka run (or tested) on Windows? Natively via the command line shell, etc. or via cygwin, within a *nix VM on Windows, or via Windows 10's Ubuntu Linux Bash shell? Would be interesting to see how each method I listed performs, maybe the Windows 10 bash shell method might be most optimal among the list? On Tue, Apr 11, 2017 at 9:41 PM, David Garcia wrote: > One issue is that Kafka leverage some very specific features of the linux > kernel that are probably far different from Windows, so I imagine the > performance profile is likewise much different. > > On 4/11/17, 8:52 AM, "Tomasz Rojek" wrote: > > Hi All, > > We want to choose provider of messaging system in our company, one of > possible choices is Apache Kafka. One of operating system that will > host > brokers is windows, according to documentation: > > https://kafka.apache.org/documentation.html#os > "*We have seen a few issues running on Windows and Windows is not > currently > a well supported platform though we would be happy to change that.*" > > Can you please elaborate more on this. What exactly potential issues > are we > talking about? What functionalities of kafka are influenced by this? > Maybe > it occurs only on specific version of windows? > > Thank you in advance for any information. > > With Regards > Tomasz Rojek > Java Engineer > > > -- David Luu Member of Technical Staff Mist Systems, Inc. 1601 S. De Anza Blvd. #248 Cupertino, CA 95014
Re: Kafka on windows
One issue is that Kafka leverage some very specific features of the linux kernel that are probably far different from Windows, so I imagine the performance profile is likewise much different. On 4/11/17, 8:52 AM, "Tomasz Rojek" wrote: Hi All, We want to choose provider of messaging system in our company, one of possible choices is Apache Kafka. One of operating system that will host brokers is windows, according to documentation: https://kafka.apache.org/documentation.html#os "*We have seen a few issues running on Windows and Windows is not currently a well supported platform though we would be happy to change that.*" Can you please elaborate more on this. What exactly potential issues are we talking about? What functionalities of kafka are influenced by this? Maybe it occurs only on specific version of windows? Thank you in advance for any information. With Regards Tomasz Rojek Java Engineer
Kafka on windows
Hi All, We want to choose provider of messaging system in our company, one of possible choices is Apache Kafka. One of operating system that will host brokers is windows, according to documentation: https://kafka.apache.org/documentation.html#os "*We have seen a few issues running on Windows and Windows is not currently a well supported platform though we would be happy to change that.*" Can you please elaborate more on this. What exactly potential issues are we talking about? What functionalities of kafka are influenced by this? Maybe it occurs only on specific version of windows? Thank you in advance for any information. With Regards Tomasz Rojek Java Engineer
RE: Using Kafka on Windows - file path problems
Ugh, stupid me, I could see what was the problem from mentioned logged lines. One should escape the path if using \ char, since it interprets the \tmp as tab followed by "mp" so using: \\tmp\\kafka-logs work, and I guess that its normal behavior for java.util.Properties.load() method. -Vjeran -Original Message- From: Vjeran Marcinko [mailto:vjeran.marci...@email.t-com.hr] Sent: Saturday, January 25, 2014 6:10 PM To: users@kafka.apache.org Subject: RE: Using Kafka on Windows - file path problems Even if in that one case the thing would work, it would be best if any correct windows file path would work. I checked again, and it behaves as I've written. In case when one specifies /tmp/kafka-logs, it works, and during startup one log line says: [2014-01-25 18:04:08,335] INFO Property log.dirs is overridden to /tmp/kafka-logs (kafka.utils.VerifiableProperties) , but as I said during topic creation it fails. But when I specify the path as \tmp\kafka-logs, then the startup fails, and the same logged line says: [2014-01-25 18:07:51,395] INFO Property log.dirs is overridden to mpkafka-logs (kafka.utils.VerifiableProperties) And it fails with mentioned: java.io.IOException: The filename, directory name, or volume label syntax is incorrect at java.io.WinNTFileSystem.canonicalize0(Native Method) at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:396) at java.io.File.getCanonicalPath(File.java:559) at kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc ala:70) at kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc ala:70) The same startup failure occurs, when I specify full windows file path as D:\tmp\kafka-logs, and then the logged line is: [2014-01-25 18:08:44,377] INFO Property log.dirs is overridden to D: mpkafka-logs (kafka.utils.VerifiableProperties) I hope this helps somewhat. -Vjeran -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Saturday, January 25, 2014 5:25 PM To: users@kafka.apache.org Subject: Re: Using Kafka on Windows - file path problems It sounds like \tmp\kafka-logs should work since that's what File.getParent() returns. Not sure why you can't use that to create the file. Thanks, Jun On Sat, Jan 25, 2014 at 12:43 AM, Vjeran Marcinko < vjeran.marci...@email.t-com.hr> wrote: > Hi, > > I have a problem going through start guide on Windows. Basically what > I found are file path problems it seems: > > If I try to set Kafka log dir path in server.properties using Windows > file separator, such as D:\tmp\kafka-logs, or even \tmp\kafka-logs , > then kafka cannot be started, since it throws > > [2014-01-25 08:04:00,495] FATAL Fatal error during KafkaServerStable > startup. Prepare to shutdown (kafka.server.KafkaServerStartable) > java.io.IOException: The filename, directory name, or volume label > syntax is incorrect > at java.io.WinNTFileSystem.canonicalize0(Native Method) > at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:396) > at java.io.File.getCanonicalPath(File.java:559) > at > > kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogMana > ger.sc > ala:70) > at > > kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogMana > ger.sc > ala:70) > at > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: > 206) > at > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: > 206) > at > > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized > .scala > :34) > at > scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:206) > at scala.collection.mutable.WrappedArray.map(WrappedArray.scala:32) > at > kafka.log.LogManager.createAndValidateLogDirs(LogManager.scala:70) > at kafka.log.LogManager.(LogManager.scala:60) > at kafka.server.KafkaServer.startup(KafkaServer.scala:63) > at > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34) > at kafka.Kafka$.main(Kafka.scala:46) > > If I use Unix file separator in settings, such as default > /tmp/kafka-logs, (and for those who don't use Windows, java on Windows > accepts paths that have unix path separator such as D:/tmp/some.file), > then Kafka can be started, but during topic creation I get following > error which I assume is due to trying to find the key using windows > file separator as given in exception log: > > [2014-01-25 08:05:53,681] ERROR Error on broker 0 while processing > LeaderAndIsr request correlationId 6 received from co
RE: Using Kafka on Windows - file path problems
Even if in that one case the thing would work, it would be best if any correct windows file path would work. I checked again, and it behaves as I've written. In case when one specifies /tmp/kafka-logs, it works, and during startup one log line says: [2014-01-25 18:04:08,335] INFO Property log.dirs is overridden to /tmp/kafka-logs (kafka.utils.VerifiableProperties) , but as I said during topic creation it fails. But when I specify the path as \tmp\kafka-logs, then the startup fails, and the same logged line says: [2014-01-25 18:07:51,395] INFO Property log.dirs is overridden to mpkafka-logs (kafka.utils.VerifiableProperties) And it fails with mentioned: java.io.IOException: The filename, directory name, or volume label syntax is incorrect at java.io.WinNTFileSystem.canonicalize0(Native Method) at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:396) at java.io.File.getCanonicalPath(File.java:559) at kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc ala:70) at kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc ala:70) The same startup failure occurs, when I specify full windows file path as D:\tmp\kafka-logs, and then the logged line is: [2014-01-25 18:08:44,377] INFO Property log.dirs is overridden to D: mpkafka-logs (kafka.utils.VerifiableProperties) I hope this helps somewhat. -Vjeran -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Saturday, January 25, 2014 5:25 PM To: users@kafka.apache.org Subject: Re: Using Kafka on Windows - file path problems It sounds like \tmp\kafka-logs should work since that's what File.getParent() returns. Not sure why you can't use that to create the file. Thanks, Jun On Sat, Jan 25, 2014 at 12:43 AM, Vjeran Marcinko < vjeran.marci...@email.t-com.hr> wrote: > Hi, > > I have a problem going through start guide on Windows. Basically what > I found are file path problems it seems: > > If I try to set Kafka log dir path in server.properties using Windows > file separator, such as D:\tmp\kafka-logs, or even \tmp\kafka-logs , > then kafka cannot be started, since it throws > > [2014-01-25 08:04:00,495] FATAL Fatal error during KafkaServerStable > startup. Prepare to shutdown (kafka.server.KafkaServerStartable) > java.io.IOException: The filename, directory name, or volume label > syntax is incorrect > at java.io.WinNTFileSystem.canonicalize0(Native Method) > at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:396) > at java.io.File.getCanonicalPath(File.java:559) > at > > kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogMana > ger.sc > ala:70) > at > > kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogMana > ger.sc > ala:70) > at > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: > 206) > at > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: > 206) > at > > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized > .scala > :34) > at > scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:206) > at scala.collection.mutable.WrappedArray.map(WrappedArray.scala:32) > at > kafka.log.LogManager.createAndValidateLogDirs(LogManager.scala:70) > at kafka.log.LogManager.(LogManager.scala:60) > at kafka.server.KafkaServer.startup(KafkaServer.scala:63) > at > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34) > at kafka.Kafka$.main(Kafka.scala:46) > > If I use Unix file separator in settings, such as default > /tmp/kafka-logs, (and for those who don't use Windows, java on Windows > accepts paths that have unix path separator such as D:/tmp/some.file), > then Kafka can be started, but during topic creation I get following > error which I assume is due to trying to find the key using windows > file separator as given in exception log: > > [2014-01-25 08:05:53,681] ERROR Error on broker 0 while processing > LeaderAndIsr request correlationId 6 received from controller 0 epoch > 1 for partition (mytopic,0) (state.change.logger) > java.util.NoSuchElementException: key not found: \tmp\kafka-logs > at scala.collection.MapLike$class.default(MapLike.scala:223) > at scala.collection.immutable.Map$Map1.default(Map.scala:93) > at scala.collection.MapLike$class.apply(MapLike.scala:134) > at scala.collection.immutable.Map$Map1.apply(Map.scala:93) > at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:88) > at kafka.cluster.Par
Re: Using Kafka on Windows - file path problems
It sounds like \tmp\kafka-logs should work since that's what File.getParent() returns. Not sure why you can't use that to create the file. Thanks, Jun On Sat, Jan 25, 2014 at 12:43 AM, Vjeran Marcinko < vjeran.marci...@email.t-com.hr> wrote: > Hi, > > I have a problem going through start guide on Windows. Basically what I > found are file path problems it seems: > > If I try to set Kafka log dir path in server.properties using Windows file > separator, such as > D:\tmp\kafka-logs, or even > \tmp\kafka-logs > , then kafka cannot be started, since it throws > > [2014-01-25 08:04:00,495] FATAL Fatal error during KafkaServerStable > startup. Prepare to shutdown (kafka.server.KafkaServerStartable) > java.io.IOException: The filename, directory name, or volume label syntax > is > incorrect > at java.io.WinNTFileSystem.canonicalize0(Native Method) > at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:396) > at java.io.File.getCanonicalPath(File.java:559) > at > > kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc > ala:70) > at > > kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc > ala:70) > at > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: > 206) > at > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: > 206) > at > > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala > :34) > at > scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:206) > at scala.collection.mutable.WrappedArray.map(WrappedArray.scala:32) > at > kafka.log.LogManager.createAndValidateLogDirs(LogManager.scala:70) > at kafka.log.LogManager.(LogManager.scala:60) > at kafka.server.KafkaServer.startup(KafkaServer.scala:63) > at > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34) > at kafka.Kafka$.main(Kafka.scala:46) > > If I use Unix file separator in settings, such as default /tmp/kafka-logs, > (and for those who don't use Windows, java on Windows accepts paths that > have unix path separator such as D:/tmp/some.file), then Kafka can be > started, but during topic creation I get following error which I assume is > due to trying to find the key using windows file separator as given in > exception log: > > [2014-01-25 08:05:53,681] ERROR Error on broker 0 while processing > LeaderAndIsr request correlationId 6 received from controller 0 epoch 1 for > partition (mytopic,0) (state.change.logger) > java.util.NoSuchElementException: key not found: \tmp\kafka-logs > at scala.collection.MapLike$class.default(MapLike.scala:223) > at scala.collection.immutable.Map$Map1.default(Map.scala:93) > at scala.collection.MapLike$class.apply(MapLike.scala:134) > at scala.collection.immutable.Map$Map1.apply(Map.scala:93) > at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:88) > at kafka.cluster.Partition$$anonfun$1.apply(Partition.scala:154) > at kafka.cluster.Partition$$anonfun$1.apply(Partition.scala:154) > at > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: > 206) > at > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: > 206) > at > > scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:6 > 1) > at scala.collection.immutable.List.foreach(List.scala:45) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:206) > at scala.collection.immutable.List.map(List.scala:45) > at kafka.cluster.Partition.makeLeader(Partition.scala:154) > at > > kafka.server.ReplicaManager.kafka$server$ReplicaManager$$makeLeader(ReplicaM > anager.scala:257) > at > > kafka.server.ReplicaManager$$anonfun$becomeLeaderOrFollower$3.apply(ReplicaM > anager.scala:221) > at > > kafka.server.ReplicaManager$$anonfun$becomeLeaderOrFollower$3.apply(ReplicaM > anager.scala:213) > at scala.collection.immutable.Map$Map1.foreach(Map.scala:105) > at > > kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:213) > at > kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:86) > at kafka.server.KafkaApis.handle(KafkaApis.scala:69) > at > kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42) > at java.lang.Thread.run(Thread.java:662) > > Regards, > Vjeran > > >
Using Kafka on Windows - file path problems
Hi, I have a problem going through start guide on Windows. Basically what I found are file path problems it seems: If I try to set Kafka log dir path in server.properties using Windows file separator, such as D:\tmp\kafka-logs, or even \tmp\kafka-logs , then kafka cannot be started, since it throws [2014-01-25 08:04:00,495] FATAL Fatal error during KafkaServerStable startup. Prepare to shutdown (kafka.server.KafkaServerStartable) java.io.IOException: The filename, directory name, or volume label syntax is incorrect at java.io.WinNTFileSystem.canonicalize0(Native Method) at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:396) at java.io.File.getCanonicalPath(File.java:559) at kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc ala:70) at kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc ala:70) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: 206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: 206) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala :34) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.mutable.WrappedArray.map(WrappedArray.scala:32) at kafka.log.LogManager.createAndValidateLogDirs(LogManager.scala:70) at kafka.log.LogManager.(LogManager.scala:60) at kafka.server.KafkaServer.startup(KafkaServer.scala:63) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34) at kafka.Kafka$.main(Kafka.scala:46) If I use Unix file separator in settings, such as default /tmp/kafka-logs, (and for those who don't use Windows, java on Windows accepts paths that have unix path separator such as D:/tmp/some.file), then Kafka can be started, but during topic creation I get following error which I assume is due to trying to find the key using windows file separator as given in exception log: [2014-01-25 08:05:53,681] ERROR Error on broker 0 while processing LeaderAndIsr request correlationId 6 received from controller 0 epoch 1 for partition (mytopic,0) (state.change.logger) java.util.NoSuchElementException: key not found: \tmp\kafka-logs at scala.collection.MapLike$class.default(MapLike.scala:223) at scala.collection.immutable.Map$Map1.default(Map.scala:93) at scala.collection.MapLike$class.apply(MapLike.scala:134) at scala.collection.immutable.Map$Map1.apply(Map.scala:93) at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:88) at kafka.cluster.Partition$$anonfun$1.apply(Partition.scala:154) at kafka.cluster.Partition$$anonfun$1.apply(Partition.scala:154) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: 206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: 206) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:6 1) at scala.collection.immutable.List.foreach(List.scala:45) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.List.map(List.scala:45) at kafka.cluster.Partition.makeLeader(Partition.scala:154) at kafka.server.ReplicaManager.kafka$server$ReplicaManager$$makeLeader(ReplicaM anager.scala:257) at kafka.server.ReplicaManager$$anonfun$becomeLeaderOrFollower$3.apply(ReplicaM anager.scala:221) at kafka.server.ReplicaManager$$anonfun$becomeLeaderOrFollower$3.apply(ReplicaM anager.scala:213) at scala.collection.immutable.Map$Map1.foreach(Map.scala:105) at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:213) at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:86) at kafka.server.KafkaApis.handle(KafkaApis.scala:69) at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42) at java.lang.Thread.run(Thread.java:662) Regards, Vjeran
Re: Need help for kafka on windows
Hi Sripada, Unfortunately I can't provide a code fix, but it's an easy fix actually. Basically the path that is trying to look for kafka-run-class.bat is wrong as it expects that file to be from the current window's folder. You can either cd into the parent and run it or fix the script. Tim On Tue, Jan 14, 2014 at 10:46 AM, Francois Langelier < francois.langel...@mate1inc.com> wrote: > I think your bug was reported here, but there is not enough details so > maybe i'm wrong > > https://issues.apache.org/jira/browse/KAFKA-1195 > > If this is your bug, the good new is you are not alone > on the other hand, the bug priority is minor and there is no fix yet... > > > > On Tue, Jan 14, 2014 at 6:15 AM, sripada.ngg >wrote: > > > Hi All, > > > > > > > > I am new to apache kafka. Can you please any one help me to setup > apache > > kafka (kafka_2.8.0-0.8.0). > > > > I have download and try to run batch file under bin/windows folder I am > > getting below error. > > > > "'kafka-run-class.bat' is not recognized as an internal or external > > command,operable program or batch file." > > > > Can you please guide me how to resolve this issue and steps to set up > > kafaka > > on window local system. > > > > > > > > Thanks and Regards, > > > > Shreepada. > > > > >
Re: Need help for kafka on windows
I think your bug was reported here, but there is not enough details so maybe i'm wrong https://issues.apache.org/jira/browse/KAFKA-1195 If this is your bug, the good new is you are not alone on the other hand, the bug priority is minor and there is no fix yet... On Tue, Jan 14, 2014 at 6:15 AM, sripada.ngg wrote: > Hi All, > > > > I am new to apache kafka. Can you please any one help me to setup apache > kafka (kafka_2.8.0-0.8.0). > > I have download and try to run batch file under bin/windows folder I am > getting below error. > > "'kafka-run-class.bat' is not recognized as an internal or external > command,operable program or batch file." > > Can you please guide me how to resolve this issue and steps to set up > kafaka > on window local system. > > > > Thanks and Regards, > > Shreepada. > >
Need help for kafka on windows
Hi All, I am new to apache kafka. Can you please any one help me to setup apache kafka (kafka_2.8.0-0.8.0). I have download and try to run batch file under bin/windows folder I am getting below error. "'kafka-run-class.bat' is not recognized as an internal or external command,operable program or batch file." Can you please guide me how to resolve this issue and steps to set up kafaka on window local system. Thanks and Regards, Shreepada.
RE: Running Kafka on Windows?
I am happy to report after I download from the 7.2 trunk. I was able to run kafka following the quickstart on Windows without any major problem. Kudos to you all. Good job!! A minor issue, if you run the commands from Kafka top level directory like this: bin\kafka-console-producer.bat --zookeeper localost:2181 --topic test You'll get below error complaining it can't find kafka-run-class.bat. 'kafka-run-class.bat' is not recognized as an internal or external command, operable program or batch file. Instead go to the bin directory and run the commands or modify the .bat file to include the path to the run-class .bat. Jamie -Original Message- From: Jamie Wang [mailto:jamie.w...@actuate.com] Sent: Wednesday, December 05, 2012 12:33 PM To: users@kafka.apache.org Subject: RE: Running Kafka on Windows? Thanks for all the replies. I downloaded the 0.72 source as you mentioned below. My question is what does the patch do? Can I download the patch and apply to 0.72 source? Jamie -Original Message- From: Antoine Vianey [mailto:antoine.via...@gmail.com] Sent: Tuesday, December 04, 2012 11:01 PM To: users@kafka.apache.org Subject: Re: Running Kafka on Windows? I ported them for both 7.2 and 8.0 but the patch was only commited for the 8.0 branch. You can find the 7.2 scripts here : https://github.com/avianey/kafka/tree/0.7.2 2012/12/5 Johan Lundahl > I've been testing 0.72 successfully on windows but converted the start > scripts myself. The only trap I ran into was forgetting to convert the > .properties files to use unix style line breaks after editing them in a win > editor. > > Johan > > > On Wed, Dec 5, 2012 at 6:21 AM, Jun Rao wrote: > > > The windows scripts are only in the 0.8 branch. Not sure if they work for > > the 0.72 release, but you can give it a try. > > > > Thanks, > > > > Jun > > > > On Tue, Dec 4, 2012 at 4:59 PM, Jamie Wang > wrote: > > > > > Hi, > > > > > > I read the literature somewhere that the windows platform is supported > > for > > > running kafka. But I didn't find any *.bat command files in the 0.72 > > > release I downloaded. I searched the net and found one entry: > > > > > > https://github.com/apache/kafka/commit/ddbedd5214a3e43afedb51ace71b783751ec8434for"; > provides windows batch script for starting Kafka/Zookeeper; patched > > > b...". Is this the one I should download and use to run Kafka on > > Windows? > > > Can someone please point me to the right direction? Thanks for your > > help. > > > > > > Jamie > > > > > >
RE: Running Kafka on Windows?
Thanks for all the replies. I downloaded the 0.72 source as you mentioned below. My question is what does the patch do? Can I download the patch and apply to 0.72 source? Jamie -Original Message- From: Antoine Vianey [mailto:antoine.via...@gmail.com] Sent: Tuesday, December 04, 2012 11:01 PM To: users@kafka.apache.org Subject: Re: Running Kafka on Windows? I ported them for both 7.2 and 8.0 but the patch was only commited for the 8.0 branch. You can find the 7.2 scripts here : https://github.com/avianey/kafka/tree/0.7.2 2012/12/5 Johan Lundahl > I've been testing 0.72 successfully on windows but converted the start > scripts myself. The only trap I ran into was forgetting to convert the > .properties files to use unix style line breaks after editing them in a win > editor. > > Johan > > > On Wed, Dec 5, 2012 at 6:21 AM, Jun Rao wrote: > > > The windows scripts are only in the 0.8 branch. Not sure if they work for > > the 0.72 release, but you can give it a try. > > > > Thanks, > > > > Jun > > > > On Tue, Dec 4, 2012 at 4:59 PM, Jamie Wang > wrote: > > > > > Hi, > > > > > > I read the literature somewhere that the windows platform is supported > > for > > > running kafka. But I didn't find any *.bat command files in the 0.72 > > > release I downloaded. I searched the net and found one entry: > > > > > > https://github.com/apache/kafka/commit/ddbedd5214a3e43afedb51ace71b783751ec8434for"; > provides windows batch script for starting Kafka/Zookeeper; patched > > > b...". Is this the one I should download and use to run Kafka on > > Windows? > > > Can someone please point me to the right direction? Thanks for your > > help. > > > > > > Jamie > > > > > >
Re: Running Kafka on Windows?
I ported them for both 7.2 and 8.0 but the patch was only commited for the 8.0 branch. You can find the 7.2 scripts here : https://github.com/avianey/kafka/tree/0.7.2 2012/12/5 Johan Lundahl > I've been testing 0.72 successfully on windows but converted the start > scripts myself. The only trap I ran into was forgetting to convert the > .properties files to use unix style line breaks after editing them in a win > editor. > > Johan > > > On Wed, Dec 5, 2012 at 6:21 AM, Jun Rao wrote: > > > The windows scripts are only in the 0.8 branch. Not sure if they work for > > the 0.72 release, but you can give it a try. > > > > Thanks, > > > > Jun > > > > On Tue, Dec 4, 2012 at 4:59 PM, Jamie Wang > wrote: > > > > > Hi, > > > > > > I read the literature somewhere that the windows platform is supported > > for > > > running kafka. But I didn't find any *.bat command files in the 0.72 > > > release I downloaded. I searched the net and found one entry: > > > > > > https://github.com/apache/kafka/commit/ddbedd5214a3e43afedb51ace71b783751ec8434for"; > provides windows batch script for starting Kafka/Zookeeper; patched > > > b...". Is this the one I should download and use to run Kafka on > > Windows? > > > Can someone please point me to the right direction? Thanks for your > > help. > > > > > > Jamie > > > > > >
Re: Running Kafka on Windows?
I've been testing 0.72 successfully on windows but converted the start scripts myself. The only trap I ran into was forgetting to convert the .properties files to use unix style line breaks after editing them in a win editor. Johan On Wed, Dec 5, 2012 at 6:21 AM, Jun Rao wrote: > The windows scripts are only in the 0.8 branch. Not sure if they work for > the 0.72 release, but you can give it a try. > > Thanks, > > Jun > > On Tue, Dec 4, 2012 at 4:59 PM, Jamie Wang wrote: > > > Hi, > > > > I read the literature somewhere that the windows platform is supported > for > > running kafka. But I didn't find any *.bat command files in the 0.72 > > release I downloaded. I searched the net and found one entry: > > > https://github.com/apache/kafka/commit/ddbedd5214a3e43afedb51ace71b783751ec8434for > " provides windows batch script for starting Kafka/Zookeeper; patched > > b...". Is this the one I should download and use to run Kafka on > Windows? > > Can someone please point me to the right direction? Thanks for your > help. > > > > Jamie > > >
Re: Running Kafka on Windows?
The windows scripts are only in the 0.8 branch. Not sure if they work for the 0.72 release, but you can give it a try. Thanks, Jun On Tue, Dec 4, 2012 at 4:59 PM, Jamie Wang wrote: > Hi, > > I read the literature somewhere that the windows platform is supported for > running kafka. But I didn't find any *.bat command files in the 0.72 > release I downloaded. I searched the net and found one entry: > https://github.com/apache/kafka/commit/ddbedd5214a3e43afedb51ace71b783751ec8434 > for " provides windows batch script for starting Kafka/Zookeeper; patched > b...". Is this the one I should download and use to run Kafka on Windows? > Can someone please point me to the right direction? Thanks for your help. > > Jamie >
Running Kafka on Windows?
Hi, I read the literature somewhere that the windows platform is supported for running kafka. But I didn't find any *.bat command files in the 0.72 release I downloaded. I searched the net and found one entry: https://github.com/apache/kafka/commit/ddbedd5214a3e43afedb51ace71b783751ec8434 for " provides windows batch script for starting Kafka/Zookeeper; patched b...". Is this the one I should download and use to run Kafka on Windows? Can someone please point me to the right direction? Thanks for your help. Jamie
