[GitHub] twill issue #16: [TWILL-199] Return the offset to read next message in Kafka...

[chtyim]

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/16
  
LGTM. Please squash the commits to one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] twill issue #16: [TWILL-199] Return the offset to read next message in Kafka...

[maochf]

Github user maochf commented on the issue:

https://github.com/apache/twill/pull/16
  
Thank you! Just squashed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15814415#comment-15814415
 ] 

ASF GitHub Bot commented on TWILL-199:
--

Github user maochf commented on the issue:

https://github.com/apache/twill/pull/16
  
Thank you! Just squashed.


> Get next offset and handle offset error in KafkaConsumer.MessageCallback
> 
>
> Key: TWILL-199
> URL: https://issues.apache.org/jira/browse/TWILL-199
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Chengfeng Mao
>
> The method {{void onReceived(Iterator messages)}} in 
> {{KafkaConsumer.MessageCallback}} can be more flexible with the change to 
> {{Long onReceived(Iterator messages)}} so that it can provide 
> additional functionalities:
> 1. To return the next offset to be fetched
> 2. To handle offset non-existence or offset mismatch error and take action on 
> the error
> This method will return null for backward compatibility when it doesn't need 
> to provide the next offset.
> In concrete implementation,  a class of a new interface 
> {{KafkaOffsetProvider}} can be added as a member in 
> {{KafkaConsumer.MessageCallback}} to perform the offset error handling and 
> provide the next offset. Besides, {{KafkaOffsetProvider}} also has methods to 
> provide the following functionalities:
> 1. To fetch earliest/latest offset in Kafka
> 2. To find the offset of a message with timestamp equal to the given 
> timestamp in Kafka
> For backward compatibility, if {{KafkaOffsetProvider}} instance is not 
> provided, its default value will be null and none of its methods will be 
> called.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15814407#comment-15814407
 ] 

ASF GitHub Bot commented on TWILL-199:
--

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/16
  
LGTM. Please squash the commits to one.


> Get next offset and handle offset error in KafkaConsumer.MessageCallback
> 
>
> Key: TWILL-199
> URL: https://issues.apache.org/jira/browse/TWILL-199
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Chengfeng Mao
>
> The method {{void onReceived(Iterator messages)}} in 
> {{KafkaConsumer.MessageCallback}} can be more flexible with the change to 
> {{Long onReceived(Iterator messages)}} so that it can provide 
> additional functionalities:
> 1. To return the next offset to be fetched
> 2. To handle offset non-existence or offset mismatch error and take action on 
> the error
> This method will return null for backward compatibility when it doesn't need 
> to provide the next offset.
> In concrete implementation,  a class of a new interface 
> {{KafkaOffsetProvider}} can be added as a member in 
> {{KafkaConsumer.MessageCallback}} to perform the offset error handling and 
> provide the next offset. Besides, {{KafkaOffsetProvider}} also has methods to 
> provide the following functionalities:
> 1. To fetch earliest/latest offset in Kafka
> 2. To find the offset of a message with timestamp equal to the given 
> timestamp in Kafka
> For backward compatibility, if {{KafkaOffsetProvider}} instance is not 
> provided, its default value will be null and none of its methods will be 
> called.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[Terence Yim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim updated TWILL-199:
--
Assignee: Chengfeng Mao

> Get next offset and handle offset error in KafkaConsumer.MessageCallback
> 
>
> Key: TWILL-199
> URL: https://issues.apache.org/jira/browse/TWILL-199
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.10.0
>
>
> The method {{void onReceived(Iterator messages)}} in 
> {{KafkaConsumer.MessageCallback}} can be more flexible with the change to 
> {{Long onReceived(Iterator messages)}} so that it can provide 
> additional functionalities:
> 1. To return the next offset to be fetched
> 2. To handle offset non-existence or offset mismatch error and take action on 
> the error
> This method will return null for backward compatibility when it doesn't need 
> to provide the next offset.
> In concrete implementation,  a class of a new interface 
> {{KafkaOffsetProvider}} can be added as a member in 
> {{KafkaConsumer.MessageCallback}} to perform the offset error handling and 
> provide the next offset. Besides, {{KafkaOffsetProvider}} also has methods to 
> provide the following functionalities:
> 1. To fetch earliest/latest offset in Kafka
> 2. To find the offset of a message with timestamp equal to the given 
> timestamp in Kafka
> For backward compatibility, if {{KafkaOffsetProvider}} instance is not 
> provided, its default value will be null and none of its methods will be 
> called.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[Terence Yim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim updated TWILL-199:
--
Fix Version/s: 0.10.0

> Get next offset and handle offset error in KafkaConsumer.MessageCallback
> 
>
> Key: TWILL-199
> URL: https://issues.apache.org/jira/browse/TWILL-199
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.10.0
>
>
> The method {{void onReceived(Iterator messages)}} in 
> {{KafkaConsumer.MessageCallback}} can be more flexible with the change to 
> {{Long onReceived(Iterator messages)}} so that it can provide 
> additional functionalities:
> 1. To return the next offset to be fetched
> 2. To handle offset non-existence or offset mismatch error and take action on 
> the error
> This method will return null for backward compatibility when it doesn't need 
> to provide the next offset.
> In concrete implementation,  a class of a new interface 
> {{KafkaOffsetProvider}} can be added as a member in 
> {{KafkaConsumer.MessageCallback}} to perform the offset error handling and 
> provide the next offset. Besides, {{KafkaOffsetProvider}} also has methods to 
> provide the following functionalities:
> 1. To fetch earliest/latest offset in Kafka
> 2. To find the offset of a message with timestamp equal to the given 
> timestamp in Kafka
> For backward compatibility, if {{KafkaOffsetProvider}} instance is not 
> provided, its default value will be null and none of its methods will be 
> called.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[Terence Yim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim updated TWILL-199:
--
Component/s: core

> Get next offset and handle offset error in KafkaConsumer.MessageCallback
> 
>
> Key: TWILL-199
> URL: https://issues.apache.org/jira/browse/TWILL-199
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.10.0
>
>
> The method {{void onReceived(Iterator messages)}} in 
> {{KafkaConsumer.MessageCallback}} can be more flexible with the change to 
> {{Long onReceived(Iterator messages)}} so that it can provide 
> additional functionalities:
> 1. To return the next offset to be fetched
> 2. To handle offset non-existence or offset mismatch error and take action on 
> the error
> This method will return null for backward compatibility when it doesn't need 
> to provide the next offset.
> In concrete implementation,  a class of a new interface 
> {{KafkaOffsetProvider}} can be added as a member in 
> {{KafkaConsumer.MessageCallback}} to perform the offset error handling and 
> provide the next offset. Besides, {{KafkaOffsetProvider}} also has methods to 
> provide the following functionalities:
> 1. To fetch earliest/latest offset in Kafka
> 2. To find the offset of a message with timestamp equal to the given 
> timestamp in Kafka
> For backward compatibility, if {{KafkaOffsetProvider}} instance is not 
> provided, its default value will be null and none of its methods will be 
> called.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] twill pull request #16: [TWILL-199] Return the offset to read next message i...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/16


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15815946#comment-15815946
 ] 

ASF GitHub Bot commented on TWILL-199:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/16


> Get next offset and handle offset error in KafkaConsumer.MessageCallback
> 
>
> Key: TWILL-199
> URL: https://issues.apache.org/jira/browse/TWILL-199
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.10.0
>
>
> The method {{void onReceived(Iterator messages)}} in 
> {{KafkaConsumer.MessageCallback}} can be more flexible with the change to 
> {{Long onReceived(Iterator messages)}} so that it can provide 
> additional functionalities:
> 1. To return the next offset to be fetched
> 2. To handle offset non-existence or offset mismatch error and take action on 
> the error
> This method will return null for backward compatibility when it doesn't need 
> to provide the next offset.
> In concrete implementation,  a class of a new interface 
> {{KafkaOffsetProvider}} can be added as a member in 
> {{KafkaConsumer.MessageCallback}} to perform the offset error handling and 
> provide the next offset. Besides, {{KafkaOffsetProvider}} also has methods to 
> provide the following functionalities:
> 1. To fetch earliest/latest offset in Kafka
> 2. To find the offset of a message with timestamp equal to the given 
> timestamp in Kafka
> For backward compatibility, if {{KafkaOffsetProvider}} instance is not 
> provided, its default value will be null and none of its methods will be 
> called.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[Terence Yim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim resolved TWILL-199.
---
Resolution: Fixed

> Get next offset and handle offset error in KafkaConsumer.MessageCallback
> 
>
> Key: TWILL-199
> URL: https://issues.apache.org/jira/browse/TWILL-199
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.10.0
>
>
> The method {{void onReceived(Iterator messages)}} in 
> {{KafkaConsumer.MessageCallback}} can be more flexible with the change to 
> {{Long onReceived(Iterator messages)}} so that it can provide 
> additional functionalities:
> 1. To return the next offset to be fetched
> 2. To handle offset non-existence or offset mismatch error and take action on 
> the error
> This method will return null for backward compatibility when it doesn't need 
> to provide the next offset.
> In concrete implementation,  a class of a new interface 
> {{KafkaOffsetProvider}} can be added as a member in 
> {{KafkaConsumer.MessageCallback}} to perform the offset error handling and 
> provide the next offset. Besides, {{KafkaOffsetProvider}} also has methods to 
> provide the following functionalities:
> 1. To fetch earliest/latest offset in Kafka
> 2. To find the offset of a message with timestamp equal to the given 
> timestamp in Kafka
> For backward compatibility, if {{KafkaOffsetProvider}} instance is not 
> provided, its default value will be null and none of its methods will be 
> called.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: CPU count in Resource Spec

[Terence Yim]

Hi,

Do you know what resource schedule that YARN is running with?

Terence

> On Jan 6, 2017, at 10:32 AM, Yuliya Feldman  wrote:
> 
> Yes, I did
> 
> See attached
> 
> On Fri, Jan 6, 2017 at 10:15 AM, Terence Yim  > wrote:
> Hi,
> 
> We've never observe this before. The AM container itself is always running
> with 1 vcore. Have you look at the YARN container info in the RM UI that is
> having that particular TwillRunnable running inside?
> 
> Terence
> 
> On Fri, Jan 6, 2017 at 9:54 AM, Yuliya Feldman  > wrote:
> 
> > yes
> >
> > On Fri, Jan 6, 2017 at 9:40 AM, Terence Yim  > > wrote:
> >
> > > Where do you verify the CPU cores? Was it from the YARN resource manager
> > > UI?
> > >
> > > Terence
> > >
> > > Sent from my iPhone
> > >
> > > > On Jan 6, 2017, at 9:01 AM, Yuliya Feldman  > > > > wrote:
> > > >
> > > > Sorry for so many questions - unfortunately not much information I can
> > > fish
> > > > in the wild :).
> > > >
> > > > I noticed that CPU count set on Application Specification is not
> > getting
> > > > reflected while starting container - it is always 1 CPU.
> > > >
> > > > Did anybody experience this? It really feels like it is lost somewhere
> > in
> > > > between Twill and YARN. I am using YARN 2.7.2
> > > >
> > > > Thanks in advance
> > >
> >
> 

Re: CPU count in Resource Spec

[Yuliya Feldman]

Capacity scheduler

On Tue, Jan 10, 2017 at 11:00 PM, Terence Yim  wrote:

> Hi,
>
> Do you know what resource schedule that YARN is running with?
>
> Terence
>
> > On Jan 6, 2017, at 10:32 AM, Yuliya Feldman  wrote:
> >
> > Yes, I did
> >
> > See attached
> >
> > On Fri, Jan 6, 2017 at 10:15 AM, Terence Yim  cht...@gmail.com>> wrote:
> > Hi,
> >
> > We've never observe this before. The AM container itself is always
> running
> > with 1 vcore. Have you look at the YARN container info in the RM UI that
> is
> > having that particular TwillRunnable running inside?
> >
> > Terence
> >
> > On Fri, Jan 6, 2017 at 9:54 AM, Yuliya Feldman  > wrote:
> >
> > > yes
> > >
> > > On Fri, Jan 6, 2017 at 9:40 AM, Terence Yim  cht...@gmail.com>> wrote:
> > >
> > > > Where do you verify the CPU cores? Was it from the YARN resource
> manager
> > > > UI?
> > > >
> > > > Terence
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On Jan 6, 2017, at 9:01 AM, Yuliya Feldman  > wrote:
> > > > >
> > > > > Sorry for so many questions - unfortunately not much information I
> can
> > > > fish
> > > > > in the wild :).
> > > > >
> > > > > I noticed that CPU count set on Application Specification is not
> > > getting
> > > > > reflected while starting container - it is always 1 CPU.
> > > > >
> > > > > Did anybody experience this? It really feels like it is lost
> somewhere
> > > in
> > > > > between Twill and YARN. I am using YARN 2.7.2
> > > > >
> > > > > Thanks in advance
> > > >
> > >
> >
>
>

Re: CPU count in Resource Spec

[Terence Yim]

Hi,

I don’t see any attachment in your previous email. Would you mind attaching it 
again?

Terence
> On Jan 10, 2017, at 11:02 PM, Yuliya Feldman  wrote:
> 
> Capacity scheduler
> 
> On Tue, Jan 10, 2017 at 11:00 PM, Terence Yim  wrote:
> 
>> Hi,
>> 
>> Do you know what resource schedule that YARN is running with?
>> 
>> Terence
>> 
>>> On Jan 6, 2017, at 10:32 AM, Yuliya Feldman  wrote:
>>> 
>>> Yes, I did
>>> 
>>> See attached
>>> 
>>> On Fri, Jan 6, 2017 at 10:15 AM, Terence Yim > cht...@gmail.com>> wrote:
>>> Hi,
>>> 
>>> We've never observe this before. The AM container itself is always
>> running
>>> with 1 vcore. Have you look at the YARN container info in the RM UI that
>> is
>>> having that particular TwillRunnable running inside?
>>> 
>>> Terence
>>> 
>>> On Fri, Jan 6, 2017 at 9:54 AM, Yuliya Feldman > > wrote:
>>> 
 yes
 
 On Fri, Jan 6, 2017 at 9:40 AM, Terence Yim > cht...@gmail.com>> wrote:
 
> Where do you verify the CPU cores? Was it from the YARN resource
>> manager
> UI?
> 
> Terence
> 
> Sent from my iPhone
> 
>> On Jan 6, 2017, at 9:01 AM, Yuliya Feldman > > wrote:
>> 
>> Sorry for so many questions - unfortunately not much information I
>> can
> fish
>> in the wild :).
>> 
>> I noticed that CPU count set on Application Specification is not
 getting
>> reflected while starting container - it is always 1 CPU.
>> 
>> Did anybody experience this? It really feels like it is lost
>> somewhere
 in
>> between Twill and YARN. I am using YARN 2.7.2
>> 
>> Thanks in advance
> 
 
>>> 
>> 
>> 

Re: CPU count in Resource Spec

[Yuliya Feldman]

sure

attached

Hopefully attachments are allowed

On Tue, Jan 10, 2017 at 11:07 PM, Terence Yim  wrote:

> Hi,
>
> I don’t see any attachment in your previous email. Would you mind
> attaching it again?
>
> Terence
> > On Jan 10, 2017, at 11:02 PM, Yuliya Feldman  wrote:
> >
> > Capacity scheduler
> >
> > On Tue, Jan 10, 2017 at 11:00 PM, Terence Yim  wrote:
> >
> >> Hi,
> >>
> >> Do you know what resource schedule that YARN is running with?
> >>
> >> Terence
> >>
> >>> On Jan 6, 2017, at 10:32 AM, Yuliya Feldman  wrote:
> >>>
> >>> Yes, I did
> >>>
> >>> See attached
> >>>
> >>> On Fri, Jan 6, 2017 at 10:15 AM, Terence Yim   >> cht...@gmail.com>> wrote:
> >>> Hi,
> >>>
> >>> We've never observe this before. The AM container itself is always
> >> running
> >>> with 1 vcore. Have you look at the YARN container info in the RM UI
> that
> >> is
> >>> having that particular TwillRunnable running inside?
> >>>
> >>> Terence
> >>>
> >>> On Fri, Jan 6, 2017 at 9:54 AM, Yuliya Feldman  >> > wrote:
> >>>
>  yes
> 
>  On Fri, Jan 6, 2017 at 9:40 AM, Terence Yim   >> cht...@gmail.com>> wrote:
> 
> > Where do you verify the CPU cores? Was it from the YARN resource
> >> manager
> > UI?
> >
> > Terence
> >
> > Sent from my iPhone
> >
> >> On Jan 6, 2017, at 9:01 AM, Yuliya Feldman  >> > wrote:
> >>
> >> Sorry for so many questions - unfortunately not much information I
> >> can
> > fish
> >> in the wild :).
> >>
> >> I noticed that CPU count set on Application Specification is not
>  getting
> >> reflected while starting container - it is always 1 CPU.
> >>
> >> Did anybody experience this? It really feels like it is lost
> >> somewhere
>  in
> >> between Twill and YARN. I am using YARN 2.7.2
> >>
> >> Thanks in advance
> >
> 
> >>>
> >>
> >>
>
>

Re: CPU count in Resource Spec

[Terence Yim]

Probably the apache mailing list does not allow attachment. Would you mind 
creating a JIRA an have it attached to the ticket?

https://issues.apache.org/jira/browse/TWILL 


Terence

> On Jan 10, 2017, at 11:09 PM, Yuliya Feldman  wrote:
> 
> sure
> 
> attached
> 
> Hopefully attachments are allowed
> 
> On Tue, Jan 10, 2017 at 11:07 PM, Terence Yim  > wrote:
> Hi,
> 
> I don’t see any attachment in your previous email. Would you mind attaching 
> it again?
> 
> Terence
> > On Jan 10, 2017, at 11:02 PM, Yuliya Feldman  > > wrote:
> >
> > Capacity scheduler
> >
> > On Tue, Jan 10, 2017 at 11:00 PM, Terence Yim  > > wrote:
> >
> >> Hi,
> >>
> >> Do you know what resource schedule that YARN is running with?
> >>
> >> Terence
> >>
> >>> On Jan 6, 2017, at 10:32 AM, Yuliya Feldman  >>> > wrote:
> >>>
> >>> Yes, I did
> >>>
> >>> See attached
> >>>
> >>> On Fri, Jan 6, 2017 at 10:15 AM, Terence Yim  >>>   >> cht...@gmail.com >> wrote:
> >>> Hi,
> >>>
> >>> We've never observe this before. The AM container itself is always
> >> running
> >>> with 1 vcore. Have you look at the YARN container info in the RM UI that
> >> is
> >>> having that particular TwillRunnable running inside?
> >>>
> >>> Terence
> >>>
> >>> On Fri, Jan 6, 2017 at 9:54 AM, Yuliya Feldman  >>> 
> >> >> wrote:
> >>>
>  yes
> 
>  On Fri, Jan 6, 2017 at 9:40 AM, Terence Yim     >> cht...@gmail.com >> wrote:
> 
> > Where do you verify the CPU cores? Was it from the YARN resource
> >> manager
> > UI?
> >
> > Terence
> >
> > Sent from my iPhone
> >
> >> On Jan 6, 2017, at 9:01 AM, Yuliya Feldman  >> 
> >> >> wrote:
> >>
> >> Sorry for so many questions - unfortunately not much information I
> >> can
> > fish
> >> in the wild :).
> >>
> >> I noticed that CPU count set on Application Specification is not
>  getting
> >> reflected while starting container - it is always 1 CPU.
> >>
> >> Did anybody experience this? It really feels like it is lost
> >> somewhere
>  in
> >> between Twill and YARN. I am using YARN 2.7.2
> >>
> >> Thanks in advance
> >
> 
> >>>
> >>
> >>
> 
> 

Re: CPU count in Resource Spec

[Yuliya Feldman]

sure

will do

Thanks

On Tue, Jan 10, 2017 at 11:11 PM, Terence Yim  wrote:

> Probably the apache mailing list does not allow attachment. Would you mind
> creating a JIRA an have it attached to the ticket?
>
> https://issues.apache.org/jira/browse/TWILL  jira/browse/TWILL>
>
> Terence
>
> > On Jan 10, 2017, at 11:09 PM, Yuliya Feldman  wrote:
> >
> > sure
> >
> > attached
> >
> > Hopefully attachments are allowed
> >
> > On Tue, Jan 10, 2017 at 11:07 PM, Terence Yim  cht...@gmail.com>> wrote:
> > Hi,
> >
> > I don’t see any attachment in your previous email. Would you mind
> attaching it again?
> >
> > Terence
> > > On Jan 10, 2017, at 11:02 PM, Yuliya Feldman  > wrote:
> > >
> > > Capacity scheduler
> > >
> > > On Tue, Jan 10, 2017 at 11:00 PM, Terence Yim  > wrote:
> > >
> > >> Hi,
> > >>
> > >> Do you know what resource schedule that YARN is running with?
> > >>
> > >> Terence
> > >>
> > >>> On Jan 6, 2017, at 10:32 AM, Yuliya Feldman  > wrote:
> > >>>
> > >>> Yes, I did
> > >>>
> > >>> See attached
> > >>>
> > >>> On Fri, Jan 6, 2017 at 10:15 AM, Terence Yim    > >> cht...@gmail.com >> wrote:
> > >>> Hi,
> > >>>
> > >>> We've never observe this before. The AM container itself is always
> > >> running
> > >>> with 1 vcore. Have you look at the YARN container info in the RM UI
> that
> > >> is
> > >>> having that particular TwillRunnable running inside?
> > >>>
> > >>> Terence
> > >>>
> > >>> On Fri, Jan 6, 2017 at 9:54 AM, Yuliya Feldman  
> > >> >> wrote:
> > >>>
> >  yes
> > 
> >  On Fri, Jan 6, 2017 at 9:40 AM, Terence Yim    > >> cht...@gmail.com >> wrote:
> > 
> > > Where do you verify the CPU cores? Was it from the YARN resource
> > >> manager
> > > UI?
> > >
> > > Terence
> > >
> > > Sent from my iPhone
> > >
> > >> On Jan 6, 2017, at 9:01 AM, Yuliya Feldman  
> > >> >> wrote:
> > >>
> > >> Sorry for so many questions - unfortunately not much information I
> > >> can
> > > fish
> > >> in the wild :).
> > >>
> > >> I noticed that CPU count set on Application Specification is not
> >  getting
> > >> reflected while starting container - it is always 1 CPU.
> > >>
> > >> Did anybody experience this? It really feels like it is lost
> > >> somewhere
> >  in
> > >> between Twill and YARN. I am using YARN 2.7.2
> > >>
> > >> Thanks in advance
> > >
> > 
> > >>>
> > >>
> > >>
> >
> >
>
>

[jira] [Created] (TWILL-203) irrespective of number of CPUs specified in App Config it is always 1

[Yuliya Feldman (JIRA)]

Yuliya Feldman created TWILL-203:


 Summary: irrespective of number of CPUs specified in App Config it 
is always 1
 Key: TWILL-203
 URL: https://issues.apache.org/jira/browse/TWILL-203
 Project: Apache Twill
  Issue Type: Bug
  Components: yarn
Affects Versions: 0.8.0
Reporter: Yuliya Feldman
 Attachments: Screen Shot 2017-01-06 at 10.17.24 AM.png, rm.log, 
twillclient.log

When trying to deploy Bundled Jar app and specifying number of CPUs > 1 it 
still defaults to 1 when application is starting.

Version of YARN is: 2.7.2, Version of Twill is 0.8. Capacity scheduler.

Looks like (from the logs) all the info is passed through Twill correctly and 
gets "lost" while getting to RM

Please see attached logs form Twill Client, RM and Screenshot



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: CPU count in Resource Spec

[Yuliya Feldman]

Created: https://issues.apache.org/jira/browse/TWILL-203

Thanks

On Tue, Jan 10, 2017 at 11:15 PM, Yuliya Feldman  wrote:

> sure
>
> will do
>
> Thanks
>
> On Tue, Jan 10, 2017 at 11:11 PM, Terence Yim  wrote:
>
>> Probably the apache mailing list does not allow attachment. Would you
>> mind creating a JIRA an have it attached to the ticket?
>>
>> https://issues.apache.org/jira/browse/TWILL <
>> https://issues.apache.org/jira/browse/TWILL>
>>
>> Terence
>>
>> > On Jan 10, 2017, at 11:09 PM, Yuliya Feldman  wrote:
>> >
>> > sure
>> >
>> > attached
>> >
>> > Hopefully attachments are allowed
>> >
>> > On Tue, Jan 10, 2017 at 11:07 PM, Terence Yim > > wrote:
>> > Hi,
>> >
>> > I don’t see any attachment in your previous email. Would you mind
>> attaching it again?
>> >
>> > Terence
>> > > On Jan 10, 2017, at 11:02 PM, Yuliya Feldman > > wrote:
>> > >
>> > > Capacity scheduler
>> > >
>> > > On Tue, Jan 10, 2017 at 11:00 PM, Terence Yim > > wrote:
>> > >
>> > >> Hi,
>> > >>
>> > >> Do you know what resource schedule that YARN is running with?
>> > >>
>> > >> Terence
>> > >>
>> > >>> On Jan 6, 2017, at 10:32 AM, Yuliya Feldman > > wrote:
>> > >>>
>> > >>> Yes, I did
>> > >>>
>> > >>> See attached
>> > >>>
>> > >>> On Fri, Jan 6, 2017 at 10:15 AM, Terence Yim >  > > >> cht...@gmail.com >> wrote:
>> > >>> Hi,
>> > >>>
>> > >>> We've never observe this before. The AM container itself is always
>> > >> running
>> > >>> with 1 vcore. Have you look at the YARN container info in the RM UI
>> that
>> > >> is
>> > >>> having that particular TwillRunnable running inside?
>> > >>>
>> > >>> Terence
>> > >>>
>> > >>> On Fri, Jan 6, 2017 at 9:54 AM, Yuliya Feldman > 
>> > >> >> wrote:
>> > >>>
>> >  yes
>> > 
>> >  On Fri, Jan 6, 2017 at 9:40 AM, Terence Yim >  > > >> cht...@gmail.com >> wrote:
>> > 
>> > > Where do you verify the CPU cores? Was it from the YARN resource
>> > >> manager
>> > > UI?
>> > >
>> > > Terence
>> > >
>> > > Sent from my iPhone
>> > >
>> > >> On Jan 6, 2017, at 9:01 AM, Yuliya Feldman > 
>> > >> >> wrote:
>> > >>
>> > >> Sorry for so many questions - unfortunately not much information
>> I
>> > >> can
>> > > fish
>> > >> in the wild :).
>> > >>
>> > >> I noticed that CPU count set on Application Specification is not
>> >  getting
>> > >> reflected while starting container - it is always 1 CPU.
>> > >>
>> > >> Did anybody experience this? It really feels like it is lost
>> > >> somewhere
>> >  in
>> > >> between Twill and YARN. I am using YARN 2.7.2
>> > >>
>> > >> Thanks in advance
>> > >
>> > 
>> > >>>
>> > >>
>> > >>
>> >
>> >
>>
>>
>