[jira] [Comment Edited] (BEAM-1986) Job ALREADY_EXISTS in post commit

2017-05-02 Thread Mark Liu (JIRA)

[ 
https://issues.apache.org/jira/browse/BEAM-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15993906#comment-15993906
 ] 

Mark Liu edited comment on BEAM-1986 at 5/2/17 10:28 PM:
-

What is "name of the main application" exactly mean? In my opinion, for testing 
it probably can be like "TestClassName-TestFunctionName". For general pipeline, 
it can be "Class/ModuleName".

I found 
[inspect|https://docs.python.org/2.7/library/inspect.html#inspect.getframeinfo] 
may be useful, which can get caller stacktrace. However, someone mentioned 
[here|http://stackoverflow.com/a/2654130] that it's not recommended for 
production code since it depends on CPython environment. 


was (Author: markflyhigh):
What is "name of the main application" exactly mean? Ideally for testing, it 
probably can be like "TestClassName-TestFunctionName". For general pipeline, it 
can be "Class/ModuleName".

I found 
[inspect|https://docs.python.org/2.7/library/inspect.html#inspect.getframeinfo] 
may be useful, which can get caller stacktrace. However, someone mentioned 
[here|http://stackoverflow.com/a/2654130] that it's not recommended for 
production code since it depends on CPython environment. 

> Job ALREADY_EXISTS in post commit
> -
>
> Key: BEAM-1986
> URL: https://issues.apache.org/jira/browse/BEAM-1986
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Mark Liu
>Priority: Minor
>
> I noticed a job failed with ALREDY_EXISTS error, a sign of same {{job_name}} 
> auto generated twice. Could we add a 1 second delay to prevent things like 
> this?
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1877/consoleFull
> cc: [~pabloem] Another perspective, would it make sense to add a small random 
> component (e.g. 1-2 digits) to job name to reduce this issue? Or perhaps 
> include ms resolution. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Comment Edited] (BEAM-1986) Job ALREADY_EXISTS in post commit

2017-05-02 Thread Mark Liu (JIRA)

[ 
https://issues.apache.org/jira/browse/BEAM-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15993850#comment-15993850
 ] 

Mark Liu edited comment on BEAM-1986 at 5/2/17 9:52 PM:


After investigation, I have some ideas to solve it and want to discuss it here:

1. Append some random number in default job_name which is pretty straight 
forward. 

2. I want to improve the descriptive of job_name, which can solve this problem 
and potentially benefit for other test runner in the further. Currently, the 
default pattern is {code}"beamapp-${USER_NAME}-${DATETIME}"{code} which is hard 
to tell which test is running. In Java SDK, default job_name format is more 
helpful: {code}"ApplicationName-UserName-Date-RandomInteger"{code} 
(https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java#L261).
 However, Java provides ApplicationNameOptions.class to set this value and is 
used by constructing default job_name, but Python doesn't provide the similar 
options. 

I prefer the second solution but it involves with pipeline options design. Just 
want to rise ideas here and hear more thoughts. [~altay] [~pabloem]


was (Author: markflyhigh):
After investigation, I have some ideas to solve it and want to discuss it here:

1. Append some random number in default job_name which is pretty straight 
forward. 

2. I want to improve the descriptive of job_name, which can solve this problem 
and potentially benefit for other test runner in the further. Currently, the 
default pattern is {code}"beamapp-${USER_NAME}-${DATETIME}"{code} which is hard 
to tell which test is running. In Java SDK, default job_name format is more 
helpful: {code}"ApplicationName-UserName-Date-RandomInteger"{code} 
(https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java#L261).
 However, Java provides ApplicationNameOptions.class to set this value and is 
used by constructing default job_name, but Python doesn't provide the similar 
options. 

I prefer the second solution but it involves with pipeline options design, so I 
want to rise ideas here and here more thoughts. [~altay] [~pabloem]

> Job ALREADY_EXISTS in post commit
> -
>
> Key: BEAM-1986
> URL: https://issues.apache.org/jira/browse/BEAM-1986
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Mark Liu
>Priority: Minor
>
> I noticed a job failed with ALREDY_EXISTS error, a sign of same {{job_name}} 
> auto generated twice. Could we add a 1 second delay to prevent things like 
> this?
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1877/consoleFull
> cc: [~pabloem] Another perspective, would it make sense to add a small random 
> component (e.g. 1-2 digits) to job name to reduce this issue? Or perhaps 
> include ms resolution. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Comment Edited] (BEAM-1986) Job ALREADY_EXISTS in post commit

2017-05-02 Thread Mark Liu (JIRA)

[ 
https://issues.apache.org/jira/browse/BEAM-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15993850#comment-15993850
 ] 

Mark Liu edited comment on BEAM-1986 at 5/2/17 9:51 PM:


After investigation, I have some ideas to solve it and want to discuss it here:

1. Append some random number in default job_name which is pretty straight 
forward. 

2. I want to improve the descriptive of job_name, which can solve this problem 
and potentially benefit for other test runner in the further. Currently, the 
default pattern is {code}"beamapp-${USER_NAME}-${DATETIME}"{code} which is hard 
to tell which test is running. In Java SDK, default job_name format is more 
helpful: {code}"ApplicationName-UserName-Date-RandomInteger"{code} 
(https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java#L261).
 However, Java provides ApplicationNameOptions.class to set this value and is 
used by constructing default job_name, but Python doesn't provide the similar 
options. 

I prefer the second solution but it involves with pipeline options design, so I 
want to rise ideas here and here more thoughts. [~altay] [~pabloem]


was (Author: markflyhigh):
After investigation, I have some ideas to solve it and want to discuss it here:

1. Append some random number in default job_name which is pretty straight 
forward. 

2. I want to improve the descriptive of job_name, which can solve this problem 
and potentially benefit for other test runner in the further. Currently, the 
default pattern is "beamapp-${USER_NAME}-${DATETIME}" which is hard to tell 
which test is running. In Java SDK, default job_name format is more helpful: 
"ApplicationName-UserName-Date-RandomInteger" 
(https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java#L261).
 However, Java provides ApplicationNameOptions.class to set this value and is 
used by constructing default job_name, but Python doesn't provide the similar 
options. 

I prefer the second solution but it involves with pipeline options design, so I 
want to rise ideas here and here more thoughts. [~altay] [~pabloem]

> Job ALREADY_EXISTS in post commit
> -
>
> Key: BEAM-1986
> URL: https://issues.apache.org/jira/browse/BEAM-1986
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Mark Liu
>Priority: Minor
>
> I noticed a job failed with ALREDY_EXISTS error, a sign of same {{job_name}} 
> auto generated twice. Could we add a 1 second delay to prevent things like 
> this?
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1877/consoleFull
> cc: [~pabloem] Another perspective, would it make sense to add a small random 
> component (e.g. 1-2 digits) to job name to reduce this issue? Or perhaps 
> include ms resolution. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)