Re: [DISCUSS] KIP-116 - Add State Store Checkpoint Interval Configuration

[Dhwani Katagade]

May be for fine tuning the performance.
Say we don't need the checkpointing and would like to gain the lil bit 
of performance improvement by turning it off.
The trade off is between giving people control knobs vs complicating the 
complete set of knobs.


-dk

On Friday 10 February 2017 04:05 PM, Eno Thereska wrote:

I can't see why users would care to turn it off.

Eno

On 10 Feb 2017, at 10:29, Damian Guy  wrote:

Hi Eno,

Sounds good to me. The only reason i can think of is if we want to be able
to turn it off.
Gouzhang - thoughts?

On Fri, 10 Feb 2017 at 10:28 Eno Thereska  wrote:


Question: if checkpointing is so cheap why not do it every commit
interval? That way we can get rid of this extra config variable and just
use the existing commit interval.

Less tuning knobs.

Eno


On 10 Feb 2017, at 09:27, Damian Guy  wrote:

Gouzhang,

You've confused me. The failure scenarios you have described are the same
as they are today. With the checkpoint files in place less data will be
replayed, so there will be fewer duplicates.

Are you saying you'd like the option to turn checkpointing off?

Thanks,
Damian

On Thu, 9 Feb 2017 at 21:55 Guozhang Wang  wrote:


Eno,

You are right, it is not a new scenario.

Thinking a bit more on how we could incorporate KIP-98 in Streams, I

feel

that if EOS is turned on inside Streams, then we probably cannot always
resume from the checkpointed offsets as it is not guaranteed to be
"consistent"; but since EOS may not be turned on by default this is

still

worthwhile to add this feature I guess.

About the default config values: I think the default value of 5 min is

OK

to me, since restoration is usually faster than normal processing

(unless

your traffic was really high), about allowing it to be "turned off"

with a

non-positive value: I feel there are still values to keep this door

open as

in the future if EOS is turned on, people may just want to turn off
checkpointing anyways, or there maybe other scenarios that we have not
realized yet. On the other hand, I would argue that it is less likely

users

mistakenly set it to a non-positive value.

Guozhang

On Thu, Feb 9, 2017 at 1:03 PM, Eno Thereska 
wrote:


Hi Guozhang,

It seems to me we have the same semantics today. Are you saying there

is

a

new failure scenario?

Thanks,
Eno


On 9 Feb 2017, at 19:42, Guozhang Wang  wrote:

More specifically, here is my reasoning of failure cases, and would

like

to

get your feedbacks:

*StreamTask*

For stream-task, the committing order is 1) flush state (may send more
records to changelog in producer), 2) flush producer, 3) commit

upstream

offsets. My understanding is that the writing of the checkpoint file

will

between 2) and 3). So thatt he new order will be 1) flush state, 2)

flush

producer, 3) write checkpoint file (when necessary), 4) commit

upstream

offsets.

And we have a bunch of "changelog offsets" regarding the state: a)

offset

corresponding to the image of the persistent file, name it point A, b)

log

end offset, name it offset B, c) checkpoint file recorded offset, name

it

offset C, d) offset corresponding to the current committed upstream

offset,

name it offset D.

Now let's talk about the failure cases:

If there is a crash between 1) and 2), then A > B = C = D. In this

case,

if

we restore, we will replay no logs at all since B = C while the

persistent

state file is actually "ahead of time", and we will start reprocessing
since from the input offset corresponding to D = B < A and hence have

some

duplicated, *which will be incorrect* if the update logic involve

reading

the state store values as well (i.e. not a blind write), e.g.

aggregations.

If there is a crash between 2) and 3), then A = B > C = D. When we

restore,

we will replay from C -> B = A, and then start reprocessing from input
offset corresponding to D < A, and same issue applies as above.

If there is a crash between 3) and 4), then A = B = C > D. When we

restore,

we will not replay, and then start reprocessing from input offset
corresponding to D < A, and same issue applies as above.


*StandbyTask*

We only do one operation today, which is 1) flush state, I think we

will

add the writing of the checkpoint file after it as step 2).

Failure cases again: offset A -> correspond to the image of the file,
offset B -> changelog end offset, offset C -> written as in the

checkpoint

file.

If there is a crash between 1) and 2), then B >= A > C (B >= A because

we

are reading from changelog topic so A will never be greater than B),

1) and if this task resumes as a standby task, we will resume

restoration

from offset C, and a few duplicates from C -> A will be applied again

to

local state files, then continue from A -> B, *this is OK* since they

do

not incur any computations hence no side effects and are all

idempotent.

2) and 

Re: General question about kafka to use in a correct way

[Dhwani Katagade]

Hi Vishnu,

Assuming that the POS application will generate POS events, we could 
accumulate these events in Kafka and use it as a data feed for live 
dashboards. We could do some common events stream processing within 
Kafka itself using the streams API. Alternatively the POS events stream 
could be fed to Spark jobs with aggregates stored to required 
granularity on some relational store.


In general there can be a role for Kafka to play, but what solution 
would be best suited depends on the details of the analytics you need. 
The following link very crisply states "What is it good for" :-) - 
https://kafka.apache.org/uses


thanks
-dhwani

On Thursday 19 January 2017 02:44 PM, Vishnu Krishnan wrote:

I read about kafka, and still i am thinking about some scenarios, what is
the possibilities to use kafka in a cloud POS application? is there any
sense to do a live dashboard?

Thanks
Amtest




DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

[jira] [Commented] (KAFKA-4617) gradle-generated core eclipse project has incorrect source folder structure

[Dhwani Katagade (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827453#comment-15827453
 ] 

Dhwani Katagade commented on KAFKA-4617:


[~ecomar] Thanks for the review and LGTM :-)

> gradle-generated core eclipse project has incorrect source folder structure
> ---
>
> Key: KAFKA-4617
> URL: https://issues.apache.org/jira/browse/KAFKA-4617
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Reporter: Edoardo Comar
>    Assignee: Dhwani Katagade
>Priority: Minor
>  Labels: build
>
> The gradle-generated Eclipse Scala project for Kafka core has a 
> classpath defined as :
> {code:xml}
>   
>   
>   
> {code}
> because of how the source files are for tests are structured, code navigation 
> / running unit tests fails. The correct structure should be instead :
> {code:xml}
>   
>path="src/test/scala"/>
>   
>   
>   
>   
> {code}
> Moreover, the classpath included as libraries core/build/test and 
> core/build/resources
> which should not be there as the eclipse classes are not generated under build



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4617) gradle-generated core eclipse project has incorrect source folder structure

[Dhwani Katagade (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15826017#comment-15826017
 ] 

Dhwani Katagade commented on KAFKA-4617:


[~ecomar] thanks for your comments.
I tried to remove the following explicit dependency from build.gradle but it is 
required by the scala plugin for building test classes. Without it the 
:core:compileTestScala task fails.

testCompile project(':clients').sourceSets.test.output

I had to fix the generated .classpath file in the whenMerged closure itself. I 
have pushed an additional commit with changes to handle this.
Please review.

> gradle-generated core eclipse project has incorrect source folder structure
> ---
>
> Key: KAFKA-4617
> URL: https://issues.apache.org/jira/browse/KAFKA-4617
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Reporter: Edoardo Comar
>    Assignee: Dhwani Katagade
>Priority: Minor
>  Labels: build
>
> The gradle-generated Eclipse Scala project for Kafka core has a 
> classpath defined as :
> {code:xml}
>   
>   
>   
> {code}
> because of how the source files are for tests are structured, code navigation 
> / running unit tests fails. The correct structure should be instead :
> {code:xml}
>   
>path="src/test/scala"/>
>   
>   
>   
>   
> {code}
> Moreover, the classpath included as libraries core/build/test and 
> core/build/resources
> which should not be there as the eclipse classes are not generated under build



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work stopped] (KAFKA-4604) Gradle Eclipse plugin creates projects for non project subfolders

[Dhwani Katagade (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-4604 stopped by Dhwani Katagade.
--
> Gradle Eclipse plugin creates projects for non project subfolders
> -
>
> Key: KAFKA-4604
> URL: https://issues.apache.org/jira/browse/KAFKA-4604
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.11.0.0
> Environment: Tried with Gradle 3.2.1 and Eclipse Neon
>Reporter: Dhwani Katagade
>    Assignee: Dhwani Katagade
>Priority: Minor
>  Labels: build, easyfix, patch
>
> Running the command *./gradlew eclipse* generates .project and .classpath 
> files for all projects. But it also generates these files for the root 
> project folder and the connect subfolder that holds the 4 connector projects 
> even though these folders are not actual project folders.
> The unnecessary connect project is benign, but the unnecessary kafka project 
> created for the root folder has a side effect. The root folder has a bin 
> directory that holds some scripts. When a _Clean all projects_ is done in 
> Eclipse, it cleans up the scripts in the bin directory. These have to be 
> restored by running *git checkout \-\- *. This same could become a 
> problem for the connect project as well if tomorrow we place some files under 
> connect/bin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4617) gradle-generated core eclipse project has incorrect source folder structure

[Dhwani Katagade (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dhwani Katagade updated KAFKA-4617:
---
  Labels: build  (was: )
Reviewer: Edoardo Comar
  Status: Patch Available  (was: In Progress)

Please review the linked PR. 
The core project .classpath file gets generated properly. The 6 folders are 
listed properly under Project Properties-> Java Build Path -> Source.

But on my eclipse 4.6.1 it does not render all 6 source folders in Package 
Explorer view. I have to remove the core project and re-import it, and then it 
renders the src folders properly under Package Explorer view as well.

> gradle-generated core eclipse project has incorrect source folder structure
> ---
>
> Key: KAFKA-4617
> URL: https://issues.apache.org/jira/browse/KAFKA-4617
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Reporter: Edoardo Comar
>    Assignee: Dhwani Katagade
>Priority: Minor
>  Labels: build
>
> The gradle-generated Eclipse Scala project for Kafka core has a 
> classpath defined as :
> {code:xml}
>   
>   
>   
> {code}
> because of how the source files are for tests are structured, code navigation 
> / running unit tests fails. The correct structure should be instead :
> {code:xml}
>   
>path="src/test/scala"/>
>   
>   
>   
>   
> {code}
> Moreover, the classpath included as libraries core/build/test and 
> core/build/resources
> which should not be there as the eclipse classes are not generated under build



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-4617) gradle-generated core eclipse project has incorrect source folder structure

[Dhwani Katagade (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-4617 started by Dhwani Katagade.
--
> gradle-generated core eclipse project has incorrect source folder structure
> ---
>
> Key: KAFKA-4617
> URL: https://issues.apache.org/jira/browse/KAFKA-4617
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Reporter: Edoardo Comar
>    Assignee: Dhwani Katagade
>Priority: Minor
>
> The gradle-generated Eclipse Scala project for Kafka core has a 
> classpath defined as :
> {code:xml}
>   
>   
>   
> {code}
> because of how the source files are for tests are structured, code navigation 
> / running unit tests fails. The correct structure should be instead :
> {code:xml}
>   
>path="src/test/scala"/>
>   
>   
>   
>   
> {code}
> Moreover, the classpath included as libraries core/build/test and 
> core/build/resources
> which should not be there as the eclipse classes are not generated under build



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-4617) gradle-generated core eclipse project has incorrect source folder structure

[Dhwani Katagade (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dhwani Katagade reassigned KAFKA-4617:
--

Assignee: Dhwani Katagade

> gradle-generated core eclipse project has incorrect source folder structure
> ---
>
> Key: KAFKA-4617
> URL: https://issues.apache.org/jira/browse/KAFKA-4617
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Reporter: Edoardo Comar
>    Assignee: Dhwani Katagade
>Priority: Minor
>
> The gradle-generated Eclipse Scala project for Kafka core has a 
> classpath defined as :
> {code:xml}
>   
>   
>   
> {code}
> because of how the source files are for tests are structured, code navigation 
> / running unit tests fails. The correct structure should be instead :
> {code:xml}
>   
>path="src/test/scala"/>
>   
>   
>   
>   
> {code}
> Moreover, the classpath included as libraries core/build/test and 
> core/build/resources
> which should not be there as the eclipse classes are not generated under build



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-4604) Gradle Eclipse plugin creates projects for non project subfolders

[Dhwani Katagade (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-4604 started by Dhwani Katagade.
--
> Gradle Eclipse plugin creates projects for non project subfolders
> -
>
> Key: KAFKA-4604
> URL: https://issues.apache.org/jira/browse/KAFKA-4604
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.11.0.0
> Environment: Tried with Gradle 3.2.1 and Eclipse Neon
>Reporter: Dhwani Katagade
>    Assignee: Dhwani Katagade
>Priority: Minor
>  Labels: build, easyfix, patch
>
> Running the command *./gradlew eclipse* generates .project and .classpath 
> files for all projects. But it also generates these files for the root 
> project folder and the connect subfolder that holds the 4 connector projects 
> even though these folders are not actual project folders.
> The unnecessary connect project is benign, but the unnecessary kafka project 
> created for the root folder has a side effect. The root folder has a bin 
> directory that holds some scripts. When a _Clean all projects_ is done in 
> Eclipse, it cleans up the scripts in the bin directory. These have to be 
> restored by running *git checkout \-\- *. This same could become a 
> problem for the connect project as well if tomorrow we place some files under 
> connect/bin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Subscribe to mailing list

[Dhwani Katagade]

Hi Abhishek,

Please follow the instructions here https://kafka.apache.org/contact. 
For your purpose the users list would be more appropriate I feel.


-dhwani

On Wednesday 11 January 2017 03:15 PM, Barot, Abhishek wrote:

Hi Team,

I would like to subscribe for the mailing list ( 
dev@kafka.apache.org ). I would like to post few 
queries we've around setting up KAFKA cluster in our production
Environment with respect to ideal configuration for 6 node cluster across 3 
data centers.

Thanks,
Abhishek Barot

--
This message w/attachments (message) is intended solely for the use of the 
intended recipient(s) and may contain information that is privileged, 
confidential or proprietary.  If you are not an intended recipient, please 
notify the sender, and then please delete and destroy all copies and 
attachments, and be advised that any review or dissemination of, or the taking 
of any action in reliance on, the information contained in or attached to this 
message is prohibited.
Unless specifically indicated, this message is not an offer to sell or a 
solicitation of any investment products or other financial product or service, 
an official confirmation of any transaction, or an official statement of 
Sender.  Subject to applicable law, Sender may intercept, monitor, review and 
retain e-communications (EC) traveling through its networks/systems and may 
produce any such EC to regulators, law enforcement, in litigation and as 
required by law.
The laws of the country of each sender/recipient may impact the handling of EC, 
and EC may be archived, supervised and produced in countries other than the 
country in which you are located. This message cannot be guaranteed to be 
secure or free of errors or viruses.  Attachments that are part of this EC may 
have additional important disclosures and disclaimers, which you should read.   
By messaging with Sender you consent to the foregoing.




DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

Re: Build problems in Eclipse dev environment setup

[Dhwani Katagade]

Hi Edoardo,

These kind of inconsistencies do not pose problems in most cases. For 
instance I was able to get things working in Eclipse by doing a full 
build on CLI and then importing the gradle eclipse plugin generated 
.project and .classpath files in eclipse.


But as a counter example, the generated project files for core look for 
test classes and resources from clients project under the gradle output 
location. If something is modified in clients test or resources under 
eclipse, the core project will not see the change unless the clients 
project is re-built from CLI.


I agree this is not a big pain, but just an inconvenient setup 
inconsistency.


-dhwani

On Wednesday 11 January 2017 05:02 PM, Edoardo Comar wrote:

Just my personal preference
the different output folders have not been annoying to me.
I know that the eclipse compile output is different from the output
generated running gradle on the CLI and am happy with that.
--
Edoardo Comar
IBM MessageHub
eco...@uk.ibm.com
IBM UK Ltd, Hursley Park, SO21 2JN

IBM United Kingdom Limited Registered in England and Wales with number
741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
3AU



From:   Dhwani Katagade <dhwani_katag...@persistent.co.in>
To: <dev@kafka.apache.org>
Date:   08/01/2017 13:52
Subject:Re: Build problems in Eclipse dev environment setup



Hi Ismael,

Thanks for your response. I agree this is not a Kafka issue, its about
gradle and its eclipse plugin. I was able to resolve the compile errors
in Eclipse by manually setting the Default output folder for core
project to core/build/classes/main.

I understand Eclipse is not the popular choice for an IDE but, for me
and my other Eclipse brethren, I looked into the problems I faced and
noticed that the output folders for gradle and those for eclipse get
configured differently. The output folders in eclipse go under
/bin where as in gradle they are under /build. Due to
this the projects in Eclipse don't work fine unless the command line
build is completed at least once. Will it be fine if we could align
these so that inter project dependencies fall well in place in either
build? *I was looking into the possibilities of fixing that and I have a
change available that I am testing out.*

There is another discussion on the pros and cons of this here
https://discuss.gradle.org/t/eclipse-generated-files-should-be-put-in-the-same-place-as-the-gradle-generated-files/6986


People, please let me know your opinion and I will provide a PR for this
if I get this working for myself.

thanks
-dhwani

On 01/07/2017 07:02 PM, Ismael Juma wrote:

Hi Dhwani,

This is more about Gradle and Eclipse than Kafka and many of us use
IntelliJ IDEA so won't be able to help you. I can confirm that IntelliJ

is

able to import the project fine out of the box.

Ismael

On 3 Jan 2017 6:45 pm, "Dhwani Katagade"

<dhwani_katag...@persistent.co.in>

wrote:


Hi,

I am trying to setup the eclipse dev environment on trunk. I have

followed

the instructions given here
https://github.com/apache/kafka/blob/trunk/README.md
https://cwiki.apache.org/confluence/display/KAFKA/Developer+
Setup#DeveloperSetup-Eclipsesetup

After I run ./gradlew eclipse in the command line followed by "Import
existing projects into workspace" in Eclipse, I am able to see the
projects. But I see the following build errors in Eclipse.

Description: The import kafka.utils.ShutdownableThread cannot be

resolved

Resource: Consumer.java
Path: /kafka-examples/src/main/java/kafka/examples
Location: line 19
Type: Java Problem

There are other errors as well but they are of a similar nature and

others

dependent on them. I can see the class ShutdownableThread in core and

core

is compiled properly. The examples project has core setup as dependency
project which seems correct. Also the command line build works just

fine.

Also I don't see any .class files generated under core/bin. They are
present under core/build/classes/main/ though. Is this due to some kind

of

mismatch between the command line build and Eclipse build?

Could you please suggest what am I missing here.

Environment:
- Eclipse Neon
- Java 1.7
- Scala 2.10.6
- CentOS 6.8

thanks
-dhwani


DISCLAIMER
==
This e-mail may contain privileged and confidential information which

is

the property of Persistent Systems Ltd. It is intended only for the use

of

the individual or entity to which it is addressed. If you are not the
intended recipient, you are not authorized to read, retain, copy,

print,

distribute or use this message. If you have received this communication

in

error, please notify the sender and delete all copies of this message.
Persistent Systems Ltd. does not accept any liability for virus

infected

mails.




DISCLAIMER
==
This e-mail may contain privileged and confidential information which is
the property of Persistent System

[jira] [Commented] (KAFKA-4617) gradle-generated core eclipse project has incorrect source folder structure

[Dhwani Katagade (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818024#comment-15818024
 ] 

Dhwani Katagade commented on KAFKA-4617:


The gradle eclipse plugin seem to have other problems as well, like the output 
folders for src and test class files and resources. More details discussed here 
http://mail-archives.apache.org/mod_mbox/kafka-dev/201701.mbox/%3Cd98febbd-ceb8-a995-56bb-2294d60beb5b%40persistent.co.in%3E


> gradle-generated core eclipse project has incorrect source folder structure
> ---
>
> Key: KAFKA-4617
> URL: https://issues.apache.org/jira/browse/KAFKA-4617
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Reporter: Edoardo Comar
>Priority: Minor
>
> The gradle-generated Eclipse Scala project for Kafka core has a 
> classpath defined as :
> {code:xml}
>   
>   
>   
> {code}
> because of how the source files are for tests are structured, code navigation 
> / running unit tests fails. The correct structure should be instead :
> {code:xml}
>   
>path="src/test/scala"/>
>   
>   
>   
>   
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4604) Gradle Eclipse plugin creates projects for non project subfolders

[Dhwani Katagade (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816852#comment-15816852
 ] 

Dhwani Katagade commented on KAFKA-4604:


Hi

I have tried it with Eclipse since I use Eclipse for an IDE. On a dev list 
thread Ismael had confirmed that the Intellij setup works just fine. I can 
believe that since the Intellij model is slightly different.
http://mail-archives.apache.org/mod_mbox/kafka-dev/201701.mbox/%3CCAD5tkZa12yGShXbbDXGUgMK7Upo-qQQx4tMn88xjYP%2B5nMfMnA%40mail.gmail.com%3E


> Gradle Eclipse plugin creates projects for non project subfolders
> -
>
> Key: KAFKA-4604
> URL: https://issues.apache.org/jira/browse/KAFKA-4604
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.11.0.0
> Environment: Tried with Gradle 3.2.1 and Eclipse Neon
>Reporter: Dhwani Katagade
>    Assignee: Dhwani Katagade
>Priority: Minor
>  Labels: build, easyfix, patch
>
> Running the command *./gradlew eclipse* generates .project and .classpath 
> files for all projects. But it also generates these files for the root 
> project folder and the connect subfolder that holds the 4 connector projects 
> even though these folders are not actual project folders.
> The unnecessary connect project is benign, but the unnecessary kafka project 
> created for the root folder has a side effect. The root folder has a bin 
> directory that holds some scripts. When a _Clean all projects_ is done in 
> Eclipse, it cleans up the scripts in the bin directory. These have to be 
> restored by running *git checkout \-\- *. This same could become a 
> problem for the connect project as well if tomorrow we place some files under 
> connect/bin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Build problems in Eclipse dev environment setup

[Dhwani Katagade]

Hi Ismael,

Thanks for your response. I agree this is not a Kafka issue, its about 
gradle and its eclipse plugin. I was able to resolve the compile errors 
in Eclipse by manually setting the Default output folder for core 
project to core/build/classes/main.


I understand Eclipse is not the popular choice for an IDE but, for me 
and my other Eclipse brethren, I looked into the problems I faced and 
noticed that the output folders for gradle and those for eclipse get 
configured differently. The output folders in eclipse go under 
/bin where as in gradle they are under /build. Due to 
this the projects in Eclipse don't work fine unless the command line 
build is completed at least once. Will it be fine if we could align 
these so that inter project dependencies fall well in place in either 
build? *I was looking into the possibilities of fixing that and I have a 
change available that I am testing out.*


There is another discussion on the pros and cons of this here 
https://discuss.gradle.org/t/eclipse-generated-files-should-be-put-in-the-same-place-as-the-gradle-generated-files/6986


People, please let me know your opinion and I will provide a PR for this 
if I get this working for myself.


thanks
-dhwani

On 01/07/2017 07:02 PM, Ismael Juma wrote:

Hi Dhwani,

This is more about Gradle and Eclipse than Kafka and many of us use
IntelliJ IDEA so won't be able to help you. I can confirm that IntelliJ is
able to import the project fine out of the box.

Ismael

On 3 Jan 2017 6:45 pm, "Dhwani Katagade" <dhwani_katag...@persistent.co.in>
wrote:


Hi,

I am trying to setup the eclipse dev environment on trunk. I have followed
the instructions given here
https://github.com/apache/kafka/blob/trunk/README.md
https://cwiki.apache.org/confluence/display/KAFKA/Developer+
Setup#DeveloperSetup-Eclipsesetup

After I run ./gradlew eclipse in the command line followed by "Import
existing projects into workspace" in Eclipse, I am able to see the
projects. But I see the following build errors in Eclipse.

Description: The import kafka.utils.ShutdownableThread cannot be resolved
Resource: Consumer.java
Path: /kafka-examples/src/main/java/kafka/examples
Location: line 19
Type: Java Problem

There are other errors as well but they are of a similar nature and others
dependent on them. I can see the class ShutdownableThread in core and core
is compiled properly. The examples project has core setup as  dependency
project which seems correct. Also the command line build works just fine.

Also I don't see any .class files generated under core/bin. They are
present under core/build/classes/main/ though. Is this due to some kind of
mismatch between the command line build and Eclipse build?

Could you please suggest what am I missing here.

Environment:
- Eclipse Neon
- Java 1.7
- Scala 2.10.6
- CentOS 6.8

thanks
-dhwani


DISCLAIMER
==
This e-mail may contain privileged and confidential information which is
the property of Persistent Systems Ltd. It is intended only for the use of
the individual or entity to which it is addressed. If you are not the
intended recipient, you are not authorized to read, retain, copy, print,
distribute or use this message. If you have received this communication in
error, please notify the sender and delete all copies of this message.
Persistent Systems Ltd. does not accept any liability for virus infected
mails.





DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

Re: Build problems in Eclipse dev environment setup

[Dhwani Katagade]

Hi,

I was able to resolve the compile errors in Eclipse by manually setting 
the Default output folder for core project to core/build/classes/main.


Looking further into this issue I see that the output folders for gradle 
and those for eclipse get configured differently. The output folders in 
eclipse go under /bin where as in gradle they are under 
/build. Will it be fine if we could align these so that inter 
project dependencies fall well in place in either build? I was looking 
into the possibilities of fixing that.


There is another discussion on the pros and cons of this here 
https://discuss.gradle.org/t/eclipse-generated-files-should-be-put-in-the-same-place-as-the-gradle-generated-files/6986


thanks
-dhwani

On 01/04/2017 12:15 AM, Dhwani Katagade wrote:

Hi,

I am trying to setup the eclipse dev environment on trunk. I have 
followed the instructions given here

https://github.com/apache/kafka/blob/trunk/README.md
https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup#DeveloperSetup-Eclipsesetup 



After I run ./gradlew eclipse in the command line followed by "Import 
existing projects into workspace" in Eclipse, I am able to see the 
projects. But I see the following build errors in Eclipse.


Description: The import kafka.utils.ShutdownableThread cannot be resolved
Resource: Consumer.java
Path: /kafka-examples/src/main/java/kafka/examples
Location: line 19
Type: Java Problem

There are other errors as well but they are of a similar nature and 
others dependent on them. I can see the class ShutdownableThread in 
core and core is compiled properly. The examples project has core 
setup as  dependency project which seems correct. Also the command 
line build works just fine.


Also I don't see any .class files generated under core/bin. They are 
present under core/build/classes/main/ though. Is this due to some 
kind of mismatch between the command line build and Eclipse build?


Could you please suggest what am I missing here.

Environment:
- Eclipse Neon
- Java 1.7
- Scala 2.10.6
- CentOS 6.8

thanks
-dhwani


DISCLAIMER
==
This e-mail may contain privileged and confidential information which 
is the property of Persistent Systems Ltd. It is intended only for the 
use of the individual or entity to which it is addressed. If you are 
not the intended recipient, you are not authorized to read, retain, 
copy, print, distribute or use this message. If you have received this 
communication in error, please notify the sender and delete all copies 
of this message. Persistent Systems Ltd. does not accept any liability 
for virus infected mails.





DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

[jira] [Commented] (KAFKA-4604) Gradle Eclipse plugin creates projects for non project subfolders

[Dhwani Katagade (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15804227#comment-15804227
 ] 

Dhwani Katagade commented on KAFKA-4604:


I would like to assign this to myself. I am not able to find a way.

> Gradle Eclipse plugin creates projects for non project subfolders
> -
>
> Key: KAFKA-4604
> URL: https://issues.apache.org/jira/browse/KAFKA-4604
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.11.0.0
> Environment: Tried with Gradle 3.2.1 and Eclipse Neon
>Reporter: Dhwani Katagade
>Priority: Minor
>  Labels: build, easyfix, patch
>
> Running the command *./gradlew eclipse* generates .project and .classpath 
> files for all projects. But it also generates these files for the root 
> project folder and the connect subfolder that holds the 4 connector projects 
> even though these folders are not actual project folders.
> The unnecessary connect project is benign, but the unnecessary kafka project 
> created for the root folder has a side effect. The root folder has a bin 
> directory that holds some scripts. When a _Clean all projects_ is done in 
> Eclipse, it cleans up the scripts in the bin directory. These have to be 
> restored by running *git checkout \-\- *. This same could become a 
> problem for the connect project as well if tomorrow we place some files under 
> connect/bin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4604) Gradle Eclipse plugin creates projects for non project subfolders

[Dhwani Katagade (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dhwani Katagade updated KAFKA-4604:
---
Summary: Gradle Eclipse plugin creates projects for non project subfolders  
(was: Gradle Eclipse plugin creates blank projects for subfolders)

> Gradle Eclipse plugin creates projects for non project subfolders
> -
>
> Key: KAFKA-4604
> URL: https://issues.apache.org/jira/browse/KAFKA-4604
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.11.0.0
> Environment: Tried with Gradle 3.2.1 and Eclipse Neon
>Reporter: Dhwani Katagade
>Priority: Minor
>  Labels: build, easyfix, patch
>
> Running the command *./gradlew eclipse* generates .project and .classpath 
> files for all projects. But it also generates these files for the root 
> project folder and the connect subfolder that holds the 4 connector projects 
> even though these folders are not actual project folders.
> The unnecessary connect project is benign, but the unnecessary kafka project 
> created for the root folder has a side effect. The root folder has a bin 
> directory that holds some scripts. When a _Clean all projects_ is done in 
> Eclipse, it cleans up the scripts in the bin directory. These have to be 
> restored by running *git checkout \-\- *. This same could become a 
> problem for the connect project as well if tomorrow we place some files under 
> connect/bin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4604) Gradle Eclipse plugin creates blank projects for subfolders

[Dhwani Katagade (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dhwani Katagade updated KAFKA-4604:
---
Summary: Gradle Eclipse plugin creates blank projects for subfolders  (was: 
Gradle Eclipse plugins creates blank projects for subfolders)

> Gradle Eclipse plugin creates blank projects for subfolders
> ---
>
> Key: KAFKA-4604
> URL: https://issues.apache.org/jira/browse/KAFKA-4604
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.11.0.0
> Environment: Tried with Gradle 3.2.1 and Eclipse Neon
>Reporter: Dhwani Katagade
>Priority: Minor
>  Labels: build, easyfix, patch
>
> Running the command *./gradlew eclipse* generates .project and .classpath 
> files for all projects. But it also generates these files for the root 
> project folder and the connect subfolder that holds the 4 connector projects 
> even though these folders are not actual project folders.
> The unnecessary connect project is benign, but the unnecessary kafka project 
> created for the root folder has a side effect. The root folder has a bin 
> directory that holds some scripts. When a _Clean all projects_ is done in 
> Eclipse, it cleans up the scripts in the bin directory. These have to be 
> restored by running *git checkout \-\- *. This same could become a 
> problem for the connect project as well if tomorrow we place some files under 
> connect/bin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4604) Gradle Eclipse plugins creates blank projects for subfolders

[Dhwani Katagade (JIRA)]

Dhwani Katagade created KAFKA-4604:
--

 Summary: Gradle Eclipse plugins creates blank projects for 
subfolders
 Key: KAFKA-4604
 URL: https://issues.apache.org/jira/browse/KAFKA-4604
 Project: Kafka
  Issue Type: Bug
  Components: build
Affects Versions: 0.11.0.0
 Environment: Tried with Gradle 3.2.1 and Eclipse Neon
Reporter: Dhwani Katagade
Priority: Minor


Running the command *./gradlew eclipse* generates .project and .classpath files 
for all projects. But it also generates these files for the root project folder 
and the connect subfolder that holds the 4 connector projects even though these 
folders are not actual project folders.

The unnecessary connect project is benign, but the unnecessary kafka project 
created for the root folder has a side effect. The root folder has a bin 
directory that holds some scripts. When a _Clean all projects_ is done in 
Eclipse, it cleans up the scripts in the bin directory. These have to be 
restored by running *git checkout \-\- *. This same could become a 
problem for the connect project as well if tomorrow we place some files under 
connect/bin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build problems in Eclipse dev environment setup

[Dhwani Katagade]

Hi,

I am trying to setup the eclipse dev environment on trunk. I have 
followed the instructions given here

https://github.com/apache/kafka/blob/trunk/README.md
https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup#DeveloperSetup-Eclipsesetup

After I run ./gradlew eclipse in the command line followed by "Import 
existing projects into workspace" in Eclipse, I am able to see the 
projects. But I see the following build errors in Eclipse.


Description: The import kafka.utils.ShutdownableThread cannot be resolved
Resource: Consumer.java
Path: /kafka-examples/src/main/java/kafka/examples
Location: line 19
Type: Java Problem

There are other errors as well but they are of a similar nature and 
others dependent on them. I can see the class ShutdownableThread in core 
and core is compiled properly. The examples project has core setup as  
dependency project which seems correct. Also the command line build 
works just fine.


Also I don't see any .class files generated under core/bin. They are 
present under core/build/classes/main/ though. Is this due to some kind 
of mismatch between the command line build and Eclipse build?


Could you please suggest what am I missing here.

Environment:
- Eclipse Neon
- Java 1.7
- Scala 2.10.6
- CentOS 6.8

thanks
-dhwani


DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

Re: Some questions about providing corrections to documentation.

[Dhwani Katagade]

Happy New Year Team :-),

Could you please look at https://github.com/apache/kafka-site/pull/38 
and let me know if this looks good. This is my first PR and I am just 
checking to see if I got the process right.


thanks
-dhwani

On 12/31/2016 01:59 PM, Dhwani Katagade wrote:

Thanks Gouzhang,

I have created a PR here https://github.com/apache/kafka-site/pull/38

Also I believe my other assumption about the site docs process is also 
correct. That when the next release happens a new directory will be 
added under kafka-site:asf-site.


thanks
-dhwani

On 12/30/2016 11:19 AM, Guozhang Wang wrote:

Hello Dhwani,

Yes since it is a hotfix on the website, you need to submit a patch to
kafka-site on all the error directories instead of on kafka repo's docs
folder.

Details about web docs contribution can be found here:

https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Website+Documentation+Changes 




Guozhang


On Thu, Dec 29, 2016 at 9:46 AM, Dhwani Katagade <
dhwani_katag...@persistent.co.in> wrote:


Hi,

_Observations:_

While going through the design documentation I noticed a minor error of
duplication in points 4 and 5 here https://kafka.apache.org/docum
entation/#design_compactionguarantees. When I looked up the files in 
git,

I see the duplication in all of the below files in kafka-site repo

https://github.com/apache/kafka-site/blob/asf-site/081/design.html
https://github.com/apache/kafka-site/blob/asf-site/082/design.html
https://github.com/apache/kafka-site/blob/asf-site/090/design.html
https://github.com/apache/kafka-site/blob/asf-site/0100/design.html
https://github.com/apache/kafka-site/blob/asf-site/0101/design.html

And also in the following branches under kafka repo
https://github.com/apache/kafka/blob/0.9.0/docs/design.html
https://github.com/apache/kafka/blob/0.10.0/docs/design.html
https://github.com/apache/kafka/blob/0.10.1/docs/design.html

But the same is corrected under trunk here 
https://github.com/apache/kafk

a/blob/trunk/docs/design.html

_Questions:_

1. If I have to provide a patch/PR to cleanup this documentation error,
should I provide the fix in files corresponding to all the versions
under kafka-site?
2. When the next release happens, as I understand, a new directory will
be added under kafka-site:asf-site. Since this would come from
kafka:trunk it will have the correction. Is my understanding 
correct?

3. As I understand, older branches under kafka repo are release
branches, and hence we should not make any new changes under the
docs directory on these branches. Is my understanding correct?
4. As I understand, this fix does not require a JIRA issue to be
logged. Is my understanding correct?

Thanks in advance for the clarifications.

-dhwani


DISCLAIMER
==
This e-mail may contain privileged and confidential information 
which is
the property of Persistent Systems Ltd. It is intended only for the 
use of

the individual or entity to which it is addressed. If you are not the
intended recipient, you are not authorized to read, retain, copy, 
print,
distribute or use this message. If you have received this 
communication in

error, please notify the sender and delete all copies of this message.
Persistent Systems Ltd. does not accept any liability for virus 
infected

mails.







DISCLAIMER
==
This e-mail may contain privileged and confidential information which 
is the property of Persistent Systems Ltd. It is intended only for the 
use of the individual or entity to which it is addressed. If you are 
not the intended recipient, you are not authorized to read, retain, 
copy, print, distribute or use this message. If you have received this 
communication in error, please notify the sender and delete all copies 
of this message. Persistent Systems Ltd. does not accept any liability 
for virus infected mails.





DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

Re: Some questions about providing corrections to documentation.

[Dhwani Katagade]

Thanks Gouzhang,

I have created a PR here https://github.com/apache/kafka-site/pull/38

Also I believe my other assumption about the site docs process is also 
correct. That when the next release happens a new directory will be 
added under kafka-site:asf-site.


thanks
-dhwani

On 12/30/2016 11:19 AM, Guozhang Wang wrote:

Hello Dhwani,

Yes since it is a hotfix on the website, you need to submit a patch to
kafka-site on all the error directories instead of on kafka repo's docs
folder.

Details about web docs contribution can be found here:

https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Website+Documentation+Changes


Guozhang


On Thu, Dec 29, 2016 at 9:46 AM, Dhwani Katagade <
dhwani_katag...@persistent.co.in> wrote:


Hi,

_Observations:_

While going through the design documentation I noticed a minor error of
duplication in points 4 and 5 here https://kafka.apache.org/docum
entation/#design_compactionguarantees. When I looked up the files in git,
I see the duplication in all of the below files in kafka-site repo

https://github.com/apache/kafka-site/blob/asf-site/081/design.html
https://github.com/apache/kafka-site/blob/asf-site/082/design.html
https://github.com/apache/kafka-site/blob/asf-site/090/design.html
https://github.com/apache/kafka-site/blob/asf-site/0100/design.html
https://github.com/apache/kafka-site/blob/asf-site/0101/design.html

And also in the following branches under kafka repo
https://github.com/apache/kafka/blob/0.9.0/docs/design.html
https://github.com/apache/kafka/blob/0.10.0/docs/design.html
https://github.com/apache/kafka/blob/0.10.1/docs/design.html

But the same is corrected under trunk here https://github.com/apache/kafk
a/blob/trunk/docs/design.html

_Questions:_

1. If I have to provide a patch/PR to cleanup this documentation error,
should I provide the fix in files corresponding to all the versions
under kafka-site?
2. When the next release happens, as I understand, a new directory will
be added under kafka-site:asf-site. Since this would come from
kafka:trunk it will have the correction. Is my understanding correct?
3. As I understand, older branches under kafka repo are release
branches, and hence we should not make any new changes under the
docs directory on these branches. Is my understanding correct?
4. As I understand, this fix does not require a JIRA issue to be
logged. Is my understanding correct?

Thanks in advance for the clarifications.

-dhwani


DISCLAIMER
==
This e-mail may contain privileged and confidential information which is
the property of Persistent Systems Ltd. It is intended only for the use of
the individual or entity to which it is addressed. If you are not the
intended recipient, you are not authorized to read, retain, copy, print,
distribute or use this message. If you have received this communication in
error, please notify the sender and delete all copies of this message.
Persistent Systems Ltd. does not accept any liability for virus infected
mails.







DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

Some questions about providing corrections to documentation.

[Dhwani Katagade]

Hi,

_Observations:_

While going through the design documentation I noticed a minor error of 
duplication in points 4 and 5 here 
https://kafka.apache.org/documentation/#design_compactionguarantees. 
When I looked up the files in git, I see the duplication in all of the 
below files in kafka-site repo


https://github.com/apache/kafka-site/blob/asf-site/081/design.html
https://github.com/apache/kafka-site/blob/asf-site/082/design.html
https://github.com/apache/kafka-site/blob/asf-site/090/design.html
https://github.com/apache/kafka-site/blob/asf-site/0100/design.html
https://github.com/apache/kafka-site/blob/asf-site/0101/design.html

And also in the following branches under kafka repo
https://github.com/apache/kafka/blob/0.9.0/docs/design.html
https://github.com/apache/kafka/blob/0.10.0/docs/design.html
https://github.com/apache/kafka/blob/0.10.1/docs/design.html

But the same is corrected under trunk here 
https://github.com/apache/kafka/blob/trunk/docs/design.html


_Questions:_

1. If I have to provide a patch/PR to cleanup this documentation error,
   should I provide the fix in files corresponding to all the versions
   under kafka-site?
2. When the next release happens, as I understand, a new directory will
   be added under kafka-site:asf-site. Since this would come from
   kafka:trunk it will have the correction. Is my understanding correct?
3. As I understand, older branches under kafka repo are release
   branches, and hence we should not make any new changes under the
   docs directory on these branches. Is my understanding correct?
4. As I understand, this fix does not require a JIRA issue to be
   logged. Is my understanding correct?

Thanks in advance for the clarifications.

-dhwani


DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.