Re: Pipelines, iterating maps and more headaches

[Jesse Glick]

On Wed, Apr 27, 2016 at 6:00 PM, Norbert Lange  wrote:
> node {
> 
> for (it2 in mapToList(depmap)) {
> name = it2[0]
> revision = it2[1]
> }
> }
> @NonCPS
> def mapToList(depmap) {
> def dlist = []
> for (entry in depmap) {
> dlist.add([entry.key, entry.value])
> }
> dlist
> }

This is the proper workaround until I can provide a serialization-safe
version of the JRE’s `Map` iterator.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CANfRfr0tav3X8BH5pxi7nDLucvxobP-mN0Wbi3tYqkbjhV3SPQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Pipelines, iterating maps and more headaches

[Jesse Glick]

On Thu, Apr 28, 2016 at 3:48 AM, Norbert Lange  wrote:
> the replay feature isnt mentioned in any of the documents from
> https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin, you should
> really mention it there to get people using it

Pipeline documentation in general is in a state of limbo at the moment.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CANfRfr319G2zLiycrVukfYVjuAf82YRC8iN7dY_UprnxTh7Tqg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Pipelines, iterating maps and more headaches

[Norbert Lange]

Nope, I initially used groovysh for rapid testing, until I found out good 
Groovy in Groovy is not necessarily good Groovy in Jenkins. Then I created 
a job with a Pipeline Script an ran that violently.
AFAIK, the replay feature isnt mentioned in any of the documents 
from https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin, you 
should really mention it there to get people using it (I doubt many people 
will dig through your blog if they want to start with pipelines)


Am Donnerstag, 28. April 2016 00:37:11 UTC+2 schrieb Patrick Wolf:
>
> Did make use of the "Replay" feature at all to troubleshoot your script, 
> Norbert? Just curious.
>
> https://jenkins.io/blog/2016/04/14/replay-with-pipeline/
>
>
>
> On Wed, Apr 27, 2016 at 3:34 PM, Brian Ray  > wrote:
>
>> Nice hackaround. I will try that in the uncooperative parts of my script.
>>
>> Ah yes, *def x* vs *x*. On the face of it the two declarations should be 
>> identical--roughly typing *x* as an *Object*. But there are different 
>> scoping implications 
>> 
>>  
>> for each.
>>
>>
>> On Wednesday, April 27, 2016 at 3:00:43 PM UTC-7, Norbert Lange wrote:
>>>
>>> Hi Brian,
>>>
>>> every single method in the "final foo" - including the String 
>>> Constructor requires approval. I was hoping for a proper subset that would 
>>> work within the sandbox.
>>> What I ended with (several dozen "Builds" later ) is using a helper 
>>> function squeezing the map into a list, seems the most sane aproach so far 
>>> =(
>>>
>>> Btw, whats the difference of using "def x" vs "x"?
>>>
>>> node {
>>> 
>>> for (it2 in mapToList(depmap)) {
>>> name = it2[0]
>>> revision = it2[1]
>>> }
>>> }
>>> @NonCPS
>>> def mapToList(depmap) {
>>> def dlist = []
>>> for (entry in depmap) {
>>> dlist.add([entry.key, entry.value])
>>> }
>>> dlist
>>> }
>>>
>>>
>>>
>>>
>>> Am Mittwoch, 27. April 2016 18:26:51 UTC+2 schrieb Brian Ray:

 FWIW I recently replaced several C-style loops with *for ( x in y )* 
 for iterating over both lists and maps in CPS code and for the most part 
 conversion went fine. There were a couple of CPS sections where I could 
 not 
 use that construct and had to fall back on the C-loops and further do a 
 torturous cast to avoid a serialization error getting keys and values from 
 the map. I want to say *Set>* caused the exception 
 because *AbstractMap.SimpleEntry* and *.SimpleImmutableEntry* are 
 serializable, while *Set *is not, per the JDK.

 for ( int i = 0; i < myMap.size(); i++ ) {
   
   // hacktacular String() cloning to avoid NotSerializableException; 
 also 
   // hacktacular Map > Set > Array morph to enable C-style looping
   final foo = new String( myMap.entrySet().toArray()[i].value )

   // do stuff with foo...

 }


 Nonetheless as mentioned in another part of the script I had no problem 
 using the shorter alternative, nor accessing keys and values using 
 *myMap.key* and *myMap.value*. Not sure what the difference is with my 
 more stubborn loop.

 On Wednesday, April 27, 2016 at 8:21:45 AM UTC-7, Norbert Lange wrote:
>
>
>
> Am Mittwoch, 27. April 2016 16:41:40 UTC+2 schrieb Jesse Glick:
>>
>> On Tuesday, April 26, 2016 at 7:18:55 PM UTC-4, Norbert Lange wrote:
>>>
>>> There seem to be some arcane rules on how to iterate over some 
>>> builtin Groovy/Java Types within a sandbox. I haven`t found a way 
>>> that 
>>> works without manually allowing the function.
>>
>>
>> Which methods did you need to approve? We can easily add them to the 
>> default whitelist in the Script Security plugin. But anyway
>>
>> The map`s each (at least)
>  
>
>>
>>> 2) Serialization issues for iterators. 
>>>
>>
>> `for (x : list) {…}` works as of `workflow-cps` 2.x. Other iterators 
>> do not yet work (outside a `@NonCPS` method). Probably fixing them is 
>> not 
>> hard, just have not gotten to it yet
>>
>
> Yes, these issues I can very likely work around. For someone who is 
> new to Groovy and Jenkins sandboxing, a list of preferred methods would 
> be 
> very welcome (the examples from the workflow libs are rather simple). 
> There 
> are atleast 3 different ways to iterate over containers, and several 
> variations of those for maps.
>  
>
>>  
>>
>>>
>>> take the createDList 
>>> which seems to execute the code differently (throws errrors, need to 
>>> define a explicit variable)
>>>
>>
>> Not sure what this is about. If you find something you think should 
>> work which does not work in a minimal reproducible script, please file a 
>> bug report for it.
>>
> Where? Is that a feat

Re: Pipelines, iterating maps and more headaches

[Craig Rodrigues]

Norm,

Take a look at this my posting from a few months ago:

https://groups.google.com/d/msg/jenkinsci-users/P7VMQQuMdsY/bHfBDSn9GgAJ

That link has pointers to the Groovy Script class and Groovy shell.
I think understanding those two classes will help improve your understanding
of Pipeline scripts, especially with respect to global variables.
I am new to Groovy, and it took me a long time to understand
what is going on with Pipeline scripts, before I started digging into the
code to figure things out.

--
Craig

On Tue, Apr 26, 2016 at 4:18 PM, Norbert Lange  wrote:

>
> 4) Are variables from Closures global??? Is this a Groovy thing or a bug???
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CAG%3DrPVeYYBSX9Fo_mypnXcpzpcvJ0PzveVghBL%3DQLyuG_4C%2B%2Bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Pipelines, iterating maps and more headaches

[Brian Ray]

Unless you're talking about your method declaration *def mapToList()*. In 
that case, the def is just window dressing AFAIK. The return type is the 
same with or without *def*: ie, *Object*.

On Wednesday, April 27, 2016 at 3:34:24 PM UTC-7, Brian Ray wrote:
>
> Nice hackaround. I will try that in the uncooperative parts of my script.
>
> Ah yes, *def x* vs *x*. On the face of it the two declarations should be 
> identical--roughly typing *x* as an *Object*. But there are different 
> scoping implications 
> 
>  
> for each.
>
> On Wednesday, April 27, 2016 at 3:00:43 PM UTC-7, Norbert Lange wrote:
>>
>> Hi Brian,
>>
>> every single method in the "final foo" - including the String 
>> Constructor requires approval. I was hoping for a proper subset that would 
>> work within the sandbox.
>> What I ended with (several dozen "Builds" later ) is using a helper 
>> function squeezing the map into a list, seems the most sane aproach so far 
>> =(
>>
>> Btw, whats the difference of using "def x" vs "x"?
>>
>> node {
>> 
>> for (it2 in mapToList(depmap)) {
>> name = it2[0]
>> revision = it2[1]
>> }
>> }
>> @NonCPS
>> def mapToList(depmap) {
>> def dlist = []
>> for (entry in depmap) {
>> dlist.add([entry.key, entry.value])
>> }
>> dlist
>> }
>>
>>
>>
>>
>> Am Mittwoch, 27. April 2016 18:26:51 UTC+2 schrieb Brian Ray:
>>>
>>> FWIW I recently replaced several C-style loops with *for ( x in y )* 
>>> for iterating over both lists and maps in CPS code and for the most part 
>>> conversion went fine. There were a couple of CPS sections where I could not 
>>> use that construct and had to fall back on the C-loops and further do a 
>>> torturous cast to avoid a serialization error getting keys and values from 
>>> the map. I want to say *Set>* caused the exception 
>>> because *AbstractMap.SimpleEntry* and *.SimpleImmutableEntry* are 
>>> serializable, while *Set *is not, per the JDK.
>>>
>>> for ( int i = 0; i < myMap.size(); i++ ) {
>>>   
>>>   // hacktacular String() cloning to avoid NotSerializableException; 
>>> also 
>>>   // hacktacular Map > Set > Array morph to enable C-style looping
>>>   final foo = new String( myMap.entrySet().toArray()[i].value )
>>>
>>>   // do stuff with foo...
>>>
>>> }
>>>
>>>
>>> Nonetheless as mentioned in another part of the script I had no problem 
>>> using the shorter alternative, nor accessing keys and values using 
>>> *myMap.key* and *myMap.value*. Not sure what the difference is with my 
>>> more stubborn loop.
>>>
>>> On Wednesday, April 27, 2016 at 8:21:45 AM UTC-7, Norbert Lange wrote:



 Am Mittwoch, 27. April 2016 16:41:40 UTC+2 schrieb Jesse Glick:
>
> On Tuesday, April 26, 2016 at 7:18:55 PM UTC-4, Norbert Lange wrote:
>>
>> There seem to be some arcane rules on how to iterate over some 
>> builtin Groovy/Java Types within a sandbox. I haven`t found a way 
>> that 
>> works without manually allowing the function.
>
>
> Which methods did you need to approve? We can easily add them to the 
> default whitelist in the Script Security plugin. But anyway
>
> The map`s each (at least)
  

>
>> 2) Serialization issues for iterators. 
>>
>
> `for (x : list) {…}` works as of `workflow-cps` 2.x. Other iterators 
> do not yet work (outside a `@NonCPS` method). Probably fixing them is not 
> hard, just have not gotten to it yet
>

 Yes, these issues I can very likely work around. For someone who is new 
 to Groovy and Jenkins sandboxing, a list of preferred methods would be 
 very 
 welcome (the examples from the workflow libs are rather simple). There are 
 atleast 3 different ways to iterate over containers, and several 
 variations 
 of those for maps.
  

>  
>
>>
>> take the createDList 
>> which seems to execute the code differently (throws errrors, need to 
>> define a explicit variable)
>>
>
> Not sure what this is about. If you find something you think should 
> work which does not work in a minimal reproducible script, please file a 
> bug report for it.
>
 Where? Is that a feature-not-bug of Groovy, an issue in Jenkins or some 
 Plugin? I was hoping for some feedback as I am not proficient in either of 
 those to pinpoint issues.
 The code above should be able to explain the issue, the exact same 
 method body in the node scope works fine, the call will result in some 
 message about "it not defined". Similary there seems some issues with name 
 clashes (if variables in functions are named like those in the node 
 scope), 
 but it mightve been some flukes during trial-and-error
  

>  
>
>>
>> Are variables from Closures global?
>>
>
> Local `def` var

Re: Pipelines, iterating maps and more headaches

[Patrick Wolf]

Did make use of the "Replay" feature at all to troubleshoot your script,
Norbert? Just curious.

https://jenkins.io/blog/2016/04/14/replay-with-pipeline/



On Wed, Apr 27, 2016 at 3:34 PM, Brian Ray  wrote:

> Nice hackaround. I will try that in the uncooperative parts of my script.
>
> Ah yes, *def x* vs *x*. On the face of it the two declarations should be
> identical--roughly typing *x* as an *Object*. But there are different
> scoping implications
> 
> for each.
>
>
> On Wednesday, April 27, 2016 at 3:00:43 PM UTC-7, Norbert Lange wrote:
>>
>> Hi Brian,
>>
>> every single method in the "final foo" - including the String
>> Constructor requires approval. I was hoping for a proper subset that would
>> work within the sandbox.
>> What I ended with (several dozen "Builds" later ) is using a helper
>> function squeezing the map into a list, seems the most sane aproach so far
>> =(
>>
>> Btw, whats the difference of using "def x" vs "x"?
>>
>> node {
>> 
>> for (it2 in mapToList(depmap)) {
>> name = it2[0]
>> revision = it2[1]
>> }
>> }
>> @NonCPS
>> def mapToList(depmap) {
>> def dlist = []
>> for (entry in depmap) {
>> dlist.add([entry.key, entry.value])
>> }
>> dlist
>> }
>>
>>
>>
>>
>> Am Mittwoch, 27. April 2016 18:26:51 UTC+2 schrieb Brian Ray:
>>>
>>> FWIW I recently replaced several C-style loops with *for ( x in y )*
>>> for iterating over both lists and maps in CPS code and for the most part
>>> conversion went fine. There were a couple of CPS sections where I could not
>>> use that construct and had to fall back on the C-loops and further do a
>>> torturous cast to avoid a serialization error getting keys and values from
>>> the map. I want to say *Set>* caused the exception
>>> because *AbstractMap.SimpleEntry* and *.SimpleImmutableEntry* are
>>> serializable, while *Set *is not, per the JDK.
>>>
>>> for ( int i = 0; i < myMap.size(); i++ ) {
>>>
>>>   // hacktacular String() cloning to avoid NotSerializableException;
>>> also
>>>   // hacktacular Map > Set > Array morph to enable C-style looping
>>>   final foo = new String( myMap.entrySet().toArray()[i].value )
>>>
>>>   // do stuff with foo...
>>>
>>> }
>>>
>>>
>>> Nonetheless as mentioned in another part of the script I had no problem
>>> using the shorter alternative, nor accessing keys and values using
>>> *myMap.key* and *myMap.value*. Not sure what the difference is with my
>>> more stubborn loop.
>>>
>>> On Wednesday, April 27, 2016 at 8:21:45 AM UTC-7, Norbert Lange wrote:



 Am Mittwoch, 27. April 2016 16:41:40 UTC+2 schrieb Jesse Glick:
>
> On Tuesday, April 26, 2016 at 7:18:55 PM UTC-4, Norbert Lange wrote:
>>
>> There seem to be some arcane rules on how to iterate over some
>> builtin Groovy/Java Types within a sandbox. I haven`t found a way
>> that
>> works without manually allowing the function.
>
>
> Which methods did you need to approve? We can easily add them to the
> default whitelist in the Script Security plugin. But anyway
>
> The map`s each (at least)


>
>> 2) Serialization issues for iterators.
>>
>
> `for (x : list) {…}` works as of `workflow-cps` 2.x. Other iterators
> do not yet work (outside a `@NonCPS` method). Probably fixing them is not
> hard, just have not gotten to it yet
>

 Yes, these issues I can very likely work around. For someone who is new
 to Groovy and Jenkins sandboxing, a list of preferred methods would be very
 welcome (the examples from the workflow libs are rather simple). There are
 atleast 3 different ways to iterate over containers, and several variations
 of those for maps.


>
>
>>
>> take the createDList
>> which seems to execute the code differently (throws errrors, need to
>> define a explicit variable)
>>
>
> Not sure what this is about. If you find something you think should
> work which does not work in a minimal reproducible script, please file a
> bug report for it.
>
 Where? Is that a feature-not-bug of Groovy, an issue in Jenkins or some
 Plugin? I was hoping for some feedback as I am not proficient in either of
 those to pinpoint issues.
 The code above should be able to explain the issue, the exact same
 method body in the node scope works fine, the call will result in some
 message about "it not defined". Similary there seems some issues with name
 clashes (if variables in functions are named like those in the node scope),
 but it mightve been some flukes during trial-and-error


>
>
>>
>> Are variables from Closures global?
>>
>
> Local `def` variables in a closure? Not sure what you are referring to
> here.
>
 See last point, and the code were I can access

Re: Pipelines, iterating maps and more headaches

[Brian Ray]

Nice hackaround. I will try that in the uncooperative parts of my script.

Ah yes, *def x* vs *x*. On the face of it the two declarations should be 
identical--roughly typing *x* as an *Object*. But there are different 
scoping implications 

 
for each.

On Wednesday, April 27, 2016 at 3:00:43 PM UTC-7, Norbert Lange wrote:
>
> Hi Brian,
>
> every single method in the "final foo" - including the String 
> Constructor requires approval. I was hoping for a proper subset that would 
> work within the sandbox.
> What I ended with (several dozen "Builds" later ) is using a helper 
> function squeezing the map into a list, seems the most sane aproach so far 
> =(
>
> Btw, whats the difference of using "def x" vs "x"?
>
> node {
> 
> for (it2 in mapToList(depmap)) {
> name = it2[0]
> revision = it2[1]
> }
> }
> @NonCPS
> def mapToList(depmap) {
> def dlist = []
> for (entry in depmap) {
> dlist.add([entry.key, entry.value])
> }
> dlist
> }
>
>
>
>
> Am Mittwoch, 27. April 2016 18:26:51 UTC+2 schrieb Brian Ray:
>>
>> FWIW I recently replaced several C-style loops with *for ( x in y )* for 
>> iterating over both lists and maps in CPS code and for the most part 
>> conversion went fine. There were a couple of CPS sections where I could not 
>> use that construct and had to fall back on the C-loops and further do a 
>> torturous cast to avoid a serialization error getting keys and values from 
>> the map. I want to say *Set>* caused the exception 
>> because *AbstractMap.SimpleEntry* and *.SimpleImmutableEntry* are 
>> serializable, while *Set *is not, per the JDK.
>>
>> for ( int i = 0; i < myMap.size(); i++ ) {
>>   
>>   // hacktacular String() cloning to avoid NotSerializableException; 
>> also 
>>   // hacktacular Map > Set > Array morph to enable C-style looping
>>   final foo = new String( myMap.entrySet().toArray()[i].value )
>>
>>   // do stuff with foo...
>>
>> }
>>
>>
>> Nonetheless as mentioned in another part of the script I had no problem 
>> using the shorter alternative, nor accessing keys and values using 
>> *myMap.key* and *myMap.value*. Not sure what the difference is with my 
>> more stubborn loop.
>>
>> On Wednesday, April 27, 2016 at 8:21:45 AM UTC-7, Norbert Lange wrote:
>>>
>>>
>>>
>>> Am Mittwoch, 27. April 2016 16:41:40 UTC+2 schrieb Jesse Glick:

 On Tuesday, April 26, 2016 at 7:18:55 PM UTC-4, Norbert Lange wrote:
>
> There seem to be some arcane rules on how to iterate over some 
> builtin Groovy/Java Types within a sandbox. I haven`t found a way that 
> works without manually allowing the function.


 Which methods did you need to approve? We can easily add them to the 
 default whitelist in the Script Security plugin. But anyway

 The map`s each (at least)
>>>  
>>>

> 2) Serialization issues for iterators. 
>

 `for (x : list) {…}` works as of `workflow-cps` 2.x. Other iterators do 
 not yet work (outside a `@NonCPS` method). Probably fixing them is not 
 hard, just have not gotten to it yet

>>>
>>> Yes, these issues I can very likely work around. For someone who is new 
>>> to Groovy and Jenkins sandboxing, a list of preferred methods would be very 
>>> welcome (the examples from the workflow libs are rather simple). There are 
>>> atleast 3 different ways to iterate over containers, and several variations 
>>> of those for maps.
>>>  
>>>
  

>
> take the createDList 
> which seems to execute the code differently (throws errrors, need to 
> define a explicit variable)
>

 Not sure what this is about. If you find something you think should 
 work which does not work in a minimal reproducible script, please file a 
 bug report for it.

>>> Where? Is that a feature-not-bug of Groovy, an issue in Jenkins or some 
>>> Plugin? I was hoping for some feedback as I am not proficient in either of 
>>> those to pinpoint issues.
>>> The code above should be able to explain the issue, the exact same 
>>> method body in the node scope works fine, the call will result in some 
>>> message about "it not defined". Similary there seems some issues with name 
>>> clashes (if variables in functions are named like those in the node scope), 
>>> but it mightve been some flukes during trial-and-error
>>>  
>>>
  

>
> Are variables from Closures global?
>

 Local `def` variables in a closure? Not sure what you are referring to 
 here.

>>> See last point, and the code were I can access the it variable after the 
>>> closure were its used (Noted with "// Weird !" )
>>>

 The main problem you are presumably hitting is the well-known 
 JENKINS-26481 . We 
 are working on a fix, but in the meantime, do not use any method 

Re: Pipelines, iterating maps and more headaches

[Norbert Lange]

Hi Brian,

every single method in the "final foo" - including the String 
Constructor requires approval. I was hoping for a proper subset that would 
work within the sandbox.
What I ended with (several dozen "Builds" later ) is using a helper 
function squeezing the map into a list, seems the most sane aproach so far 
=(

Btw, whats the difference of using "def x" vs "x"?

node {

for (it2 in mapToList(depmap)) {
name = it2[0]
revision = it2[1]
}
}
@NonCPS
def mapToList(depmap) {
def dlist = []
for (entry in depmap) {
dlist.add([entry.key, entry.value])
}
dlist
}




Am Mittwoch, 27. April 2016 18:26:51 UTC+2 schrieb Brian Ray:
>
> FWIW I recently replaced several C-style loops with *for ( x in y )* for 
> iterating over both lists and maps in CPS code and for the most part 
> conversion went fine. There were a couple of CPS sections where I could not 
> use that construct and had to fall back on the C-loops and further do a 
> torturous cast to avoid a serialization error getting keys and values from 
> the map. I want to say *Set>* caused the exception because 
> *AbstractMap.SimpleEntry* and *.SimpleImmutableEntry* are serializable, 
> while *Set *is not, per the JDK.
>
> for ( int i = 0; i < myMap.size(); i++ ) {
>   
>   // hacktacular String() cloning to avoid NotSerializableException; also 
>   // hacktacular Map > Set > Array morph to enable C-style looping
>   final foo = new String( myMap.entrySet().toArray()[i].value )
>
>   // do stuff with foo...
>
> }
>
>
> Nonetheless as mentioned in another part of the script I had no problem 
> using the shorter alternative, nor accessing keys and values using 
> *myMap.key* and *myMap.value*. Not sure what the difference is with my 
> more stubborn loop.
>
> On Wednesday, April 27, 2016 at 8:21:45 AM UTC-7, Norbert Lange wrote:
>>
>>
>>
>> Am Mittwoch, 27. April 2016 16:41:40 UTC+2 schrieb Jesse Glick:
>>>
>>> On Tuesday, April 26, 2016 at 7:18:55 PM UTC-4, Norbert Lange wrote:

 There seem to be some arcane rules on how to iterate over some 
 builtin Groovy/Java Types within a sandbox. I haven`t found a way that 
 works without manually allowing the function.
>>>
>>>
>>> Which methods did you need to approve? We can easily add them to the 
>>> default whitelist in the Script Security plugin. But anyway
>>>
>>> The map`s each (at least)
>>  
>>
>>>
 2) Serialization issues for iterators. 

>>>
>>> `for (x : list) {…}` works as of `workflow-cps` 2.x. Other iterators do 
>>> not yet work (outside a `@NonCPS` method). Probably fixing them is not 
>>> hard, just have not gotten to it yet
>>>
>>
>> Yes, these issues I can very likely work around. For someone who is new 
>> to Groovy and Jenkins sandboxing, a list of preferred methods would be very 
>> welcome (the examples from the workflow libs are rather simple). There are 
>> atleast 3 different ways to iterate over containers, and several variations 
>> of those for maps.
>>  
>>
>>>  
>>>

 take the createDList 
 which seems to execute the code differently (throws errrors, need to 
 define a explicit variable)

>>>
>>> Not sure what this is about. If you find something you think should work 
>>> which does not work in a minimal reproducible script, please file a bug 
>>> report for it.
>>>
>> Where? Is that a feature-not-bug of Groovy, an issue in Jenkins or some 
>> Plugin? I was hoping for some feedback as I am not proficient in either of 
>> those to pinpoint issues.
>> The code above should be able to explain the issue, the exact same method 
>> body in the node scope works fine, the call will result in some message 
>> about "it not defined". Similary there seems some issues with name clashes 
>> (if variables in functions are named like those in the node scope), but it 
>> mightve been some flukes during trial-and-error
>>  
>>
>>>  
>>>

 Are variables from Closures global?

>>>
>>> Local `def` variables in a closure? Not sure what you are referring to 
>>> here.
>>>
>> See last point, and the code were I can access the it variable after the 
>> closure were its used (Noted with "// Weird !" )
>>
>>>
>>> The main problem you are presumably hitting is the well-known 
>>> JENKINS-26481 . We 
>>> are working on a fix, but in the meantime, do not use any method built into 
>>> Groovy which takes a closure argument, such as `list.each {x -> …}` or 
>>> `someText.eachLine {line -> …}`. Rather use a Java-style loop. (To be on 
>>> the safe side, also avoid iterators, meaning use a C- or JDK 1.4-style loop 
>>> with an index.)
>>>
>> Could you please post me the preferable code for iterating a map in this 
>> way? (Not sure I fully understand the bug )
>>
>>
>>> Incidentally `it` does not currently work in closures as noted in 
>>> JENKINS-33468 
>>> 

Re: Pipelines, iterating maps and more headaches

[Brian Ray]

Side note: The map keys and values are simply *String*s.

On Wednesday, April 27, 2016 at 9:26:51 AM UTC-7, Brian Ray wrote:
>
> FWIW I recently replaced several C-style loops with *for ( x in y )* for 
> iterating over both lists and maps in CPS code and for the most part 
> conversion went fine. There were a couple of CPS sections where I could not 
> use that construct and had to fall back on the C-loops and further do a 
> torturous cast to avoid a serialization error getting keys and values from 
> the map. I want to say *Set>* caused the exception because 
> *AbstractMap.SimpleEntry* and *.SimpleImmutableEntry* are serializable, 
> while *Set *is not, per the JDK.
>
> for ( int i = 0; i < myMap.size(); i++ ) {
>   
>   // hacktacular String() cloning to avoid NotSerializableException; also 
>   // hacktacular Map > Set > Array morph to enable C-style looping
>   final foo = new String( myMap.entrySet().toArray()[i].value )
>
>   // do stuff with foo...
>
> }
>
>
> Nonetheless as mentioned in another part of the script I had no problem 
> using the shorter alternative, nor accessing keys and values using 
> *myMap.key* and *myMap.value*. Not sure what the difference is with my 
> more stubborn loop.
>
> On Wednesday, April 27, 2016 at 8:21:45 AM UTC-7, Norbert Lange wrote:
>>
>>
>>
>> Am Mittwoch, 27. April 2016 16:41:40 UTC+2 schrieb Jesse Glick:
>>>
>>> On Tuesday, April 26, 2016 at 7:18:55 PM UTC-4, Norbert Lange wrote:

 There seem to be some arcane rules on how to iterate over some 
 builtin Groovy/Java Types within a sandbox. I haven`t found a way that 
 works without manually allowing the function.
>>>
>>>
>>> Which methods did you need to approve? We can easily add them to the 
>>> default whitelist in the Script Security plugin. But anyway
>>>
>>> The map`s each (at least)
>>  
>>
>>>
 2) Serialization issues for iterators. 

>>>
>>> `for (x : list) {…}` works as of `workflow-cps` 2.x. Other iterators do 
>>> not yet work (outside a `@NonCPS` method). Probably fixing them is not 
>>> hard, just have not gotten to it yet
>>>
>>
>> Yes, these issues I can very likely work around. For someone who is new 
>> to Groovy and Jenkins sandboxing, a list of preferred methods would be very 
>> welcome (the examples from the workflow libs are rather simple). There are 
>> atleast 3 different ways to iterate over containers, and several variations 
>> of those for maps.
>>  
>>
>>>  
>>>

 take the createDList 
 which seems to execute the code differently (throws errrors, need to 
 define a explicit variable)

>>>
>>> Not sure what this is about. If you find something you think should work 
>>> which does not work in a minimal reproducible script, please file a bug 
>>> report for it.
>>>
>> Where? Is that a feature-not-bug of Groovy, an issue in Jenkins or some 
>> Plugin? I was hoping for some feedback as I am not proficient in either of 
>> those to pinpoint issues.
>> The code above should be able to explain the issue, the exact same method 
>> body in the node scope works fine, the call will result in some message 
>> about "it not defined". Similary there seems some issues with name clashes 
>> (if variables in functions are named like those in the node scope), but it 
>> mightve been some flukes during trial-and-error
>>  
>>
>>>  
>>>

 Are variables from Closures global?

>>>
>>> Local `def` variables in a closure? Not sure what you are referring to 
>>> here.
>>>
>> See last point, and the code were I can access the it variable after the 
>> closure were its used (Noted with "// Weird !" )
>>
>>>
>>> The main problem you are presumably hitting is the well-known 
>>> JENKINS-26481 . We 
>>> are working on a fix, but in the meantime, do not use any method built into 
>>> Groovy which takes a closure argument, such as `list.each {x -> …}` or 
>>> `someText.eachLine {line -> …}`. Rather use a Java-style loop. (To be on 
>>> the safe side, also avoid iterators, meaning use a C- or JDK 1.4-style loop 
>>> with an index.)
>>>
>> Could you please post me the preferable code for iterating a map in this 
>> way? (Not sure I fully understand the bug )
>>
>>
>>> Incidentally `it` does not currently work in closures as noted in 
>>> JENKINS-33468 
>>> ;
>>>  
>>> use an explicit parameter name instead.
>>>
>> That seems to be one of the issues I am fighting with, and might be that 
>> the supposed name-clashes came from unfocused variations of the code. 
>> Strangely it does seem to somewhat work in the node body?
>>  
>> Kind Regards, 
>> Norbert 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr..

Re: Pipelines, iterating maps and more headaches

[Brian Ray]

FWIW I recently replaced several C-style loops with *for ( x in y )* for 
iterating over both lists and maps in CPS code and for the most part 
conversion went fine. There were a couple of CPS sections where I could not 
use that construct and had to fall back on the C-loops and further do a 
torturous cast to avoid a serialization error getting keys and values from 
the map. I want to say *Set>* caused the exception because 
*AbstractMap.SimpleEntry* and *.SimpleImmutableEntry* are serializable, 
while *Set *is not, per the JDK.

for ( int i = 0; i < myMap.size(); i++ ) {
  
  // hacktacular String() cloning to avoid NotSerializableException; also 
  // hacktacular Map > Set > Array morph to enable C-style looping
  final foo = new String( myMap.entrySet().toArray()[i].value )

  // do stuff with foo...

}


Nonetheless as mentioned in another part of the script I had no problem 
using the shorter alternative, nor accessing keys and values using 
*myMap.key* and *myMap.value*. Not sure what the difference is with my more 
stubborn loop.

On Wednesday, April 27, 2016 at 8:21:45 AM UTC-7, Norbert Lange wrote:
>
>
>
> Am Mittwoch, 27. April 2016 16:41:40 UTC+2 schrieb Jesse Glick:
>>
>> On Tuesday, April 26, 2016 at 7:18:55 PM UTC-4, Norbert Lange wrote:
>>>
>>> There seem to be some arcane rules on how to iterate over some 
>>> builtin Groovy/Java Types within a sandbox. I haven`t found a way that 
>>> works without manually allowing the function.
>>
>>
>> Which methods did you need to approve? We can easily add them to the 
>> default whitelist in the Script Security plugin. But anyway
>>
>> The map`s each (at least)
>  
>
>>
>>> 2) Serialization issues for iterators. 
>>>
>>
>> `for (x : list) {…}` works as of `workflow-cps` 2.x. Other iterators do 
>> not yet work (outside a `@NonCPS` method). Probably fixing them is not 
>> hard, just have not gotten to it yet
>>
>
> Yes, these issues I can very likely work around. For someone who is new to 
> Groovy and Jenkins sandboxing, a list of preferred methods would be very 
> welcome (the examples from the workflow libs are rather simple). There are 
> atleast 3 different ways to iterate over containers, and several variations 
> of those for maps.
>  
>
>>  
>>
>>>
>>> take the createDList 
>>> which seems to execute the code differently (throws errrors, need to 
>>> define a explicit variable)
>>>
>>
>> Not sure what this is about. If you find something you think should work 
>> which does not work in a minimal reproducible script, please file a bug 
>> report for it.
>>
> Where? Is that a feature-not-bug of Groovy, an issue in Jenkins or some 
> Plugin? I was hoping for some feedback as I am not proficient in either of 
> those to pinpoint issues.
> The code above should be able to explain the issue, the exact same method 
> body in the node scope works fine, the call will result in some message 
> about "it not defined". Similary there seems some issues with name clashes 
> (if variables in functions are named like those in the node scope), but it 
> mightve been some flukes during trial-and-error
>  
>
>>  
>>
>>>
>>> Are variables from Closures global?
>>>
>>
>> Local `def` variables in a closure? Not sure what you are referring to 
>> here.
>>
> See last point, and the code were I can access the it variable after the 
> closure were its used (Noted with "// Weird !" )
>
>>
>> The main problem you are presumably hitting is the well-known 
>> JENKINS-26481 . We 
>> are working on a fix, but in the meantime, do not use any method built into 
>> Groovy which takes a closure argument, such as `list.each {x -> …}` or 
>> `someText.eachLine {line -> …}`. Rather use a Java-style loop. (To be on 
>> the safe side, also avoid iterators, meaning use a C- or JDK 1.4-style loop 
>> with an index.)
>>
> Could you please post me the preferable code for iterating a map in this 
> way? (Not sure I fully understand the bug )
>
>
>> Incidentally `it` does not currently work in closures as noted in 
>> JENKINS-33468 
>> ;
>>  
>> use an explicit parameter name instead.
>>
> That seems to be one of the issues I am fighting with, and might be that 
> the supposed name-clashes came from unfocused variations of the code. 
> Strangely it does seem to somewhat work in the node body?
>  
> Kind Regards, 
> Norbert 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/53ef9445-93a4-42aa-885a-e5a5238758ce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Pipelines, iterating maps and more headaches

[Norbert Lange]

Am Mittwoch, 27. April 2016 16:41:40 UTC+2 schrieb Jesse Glick:
>
> On Tuesday, April 26, 2016 at 7:18:55 PM UTC-4, Norbert Lange wrote:
>>
>> There seem to be some arcane rules on how to iterate over some 
>> builtin Groovy/Java Types within a sandbox. I haven`t found a way that 
>> works without manually allowing the function.
>
>
> Which methods did you need to approve? We can easily add them to the 
> default whitelist in the Script Security plugin. But anyway
>
> The map`s each (at least)
 

>
>> 2) Serialization issues for iterators. 
>>
>
> `for (x : list) {…}` works as of `workflow-cps` 2.x. Other iterators do 
> not yet work (outside a `@NonCPS` method). Probably fixing them is not 
> hard, just have not gotten to it yet
>

Yes, these issues I can very likely work around. For someone who is new to 
Groovy and Jenkins sandboxing, a list of preferred methods would be very 
welcome (the examples from the workflow libs are rather simple). There are 
atleast 3 different ways to iterate over containers, and several variations 
of those for maps.
 

>  
>
>>
>> take the createDList 
>> which seems to execute the code differently (throws errrors, need to 
>> define a explicit variable)
>>
>
> Not sure what this is about. If you find something you think should work 
> which does not work in a minimal reproducible script, please file a bug 
> report for it.
>
Where? Is that a feature-not-bug of Groovy, an issue in Jenkins or some 
Plugin? I was hoping for some feedback as I am not proficient in either of 
those to pinpoint issues.
The code above should be able to explain the issue, the exact same method 
body in the node scope works fine, the call will result in some message 
about "it not defined". Similary there seems some issues with name clashes 
(if variables in functions are named like those in the node scope), but it 
mightve been some flukes during trial-and-error
 

>  
>
>>
>> Are variables from Closures global?
>>
>
> Local `def` variables in a closure? Not sure what you are referring to 
> here.
>
See last point, and the code were I can access the it variable after the 
closure were its used (Noted with "// Weird !" )

>
> The main problem you are presumably hitting is the well-known 
> JENKINS-26481 . We 
> are working on a fix, but in the meantime, do not use any method built into 
> Groovy which takes a closure argument, such as `list.each {x -> …}` or 
> `someText.eachLine {line -> …}`. Rather use a Java-style loop. (To be on 
> the safe side, also avoid iterators, meaning use a C- or JDK 1.4-style loop 
> with an index.)
>
Could you please post me the preferable code for iterating a map in this 
way? (Not sure I fully understand the bug )


> Incidentally `it` does not currently work in closures as noted in 
> JENKINS-33468 
> ;
>  
> use an explicit parameter name instead.
>
That seems to be one of the issues I am fighting with, and might be that 
the supposed name-clashes came from unfocused variations of the code. 
Strangely it does seem to somewhat work in the node body?
 
Kind Regards, 
Norbert 

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/c8a00b42-49e3-4152-a624-ff8bb0b0518a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Pipelines, iterating maps and more headaches

[Jesse Glick]

On Tuesday, April 26, 2016 at 7:18:55 PM UTC-4, Norbert Lange wrote:
>
> There seem to be some arcane rules on how to iterate over some 
> builtin Groovy/Java Types within a sandbox. I haven`t found a way that 
> works without manually allowing the function.


Which methods did you need to approve? We can easily add them to the 
default whitelist in the Script Security plugin. But anyway


> 2) Serialization issues for iterators. 
>

`for (x : list) {…}` works as of `workflow-cps` 2.x. Other iterators do not 
yet work (outside a `@NonCPS` method). Probably fixing them is not hard, 
just have not gotten to it yet.
 

>
> take the createDList 
> which seems to execute the code differently (throws errrors, need to 
> define a explicit variable)
>

Not sure what this is about. If you find something you think should work 
which does not work in a minimal reproducible script, please file a bug 
report for it.
 

>
> Are variables from Closures global?
>

Local `def` variables in a closure? Not sure what you are referring to here.

The main problem you are presumably hitting is the well-known JENKINS-26481 
. We are working on a 
fix, but in the meantime, do not use any method built into Groovy which 
takes a closure argument, such as `list.each {x -> …}` or 
`someText.eachLine {line -> …}`. Rather use a Java-style loop. (To be on 
the safe side, also avoid iterators, meaning use a C- or JDK 1.4-style loop 
with an index.)

Incidentally `it` does not currently work in closures as noted in 
JENKINS-33468 ; use an 
explicit parameter name instead.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/02dfacb5-935b-433e-aa75-e58cff069163%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Pipelines, iterating maps and more headaches

[Norbert Lange]

Hello,

I trying to get comfortable with Pipelines, so far its a rather
unpleasant experience since I cant even get a simple script going.

1) There seem to be some arcane rules on how to iterate over some
builtin Groovy/Java Types within a sandbox. I haven`t found a way that
works without manually allowing the function.

2) Serialization issues for iterators.

3) Language rules seem inconsistent, for example take the createDList
which seems to execute the code differently (throws errrors, need to
define a explicit variable)

4) Are variables from Closures global??? Is this a Groovy thing or a bug???

There seem to be scarce information on how to write useful simple yet
non-trivial pipeline scripts, even the most basic builtin types can`t
be used freely.
I dont know how to use maps, or if there is a prefered way (which
doesnt needs approval). Layering out code into seperate functions to
not step into serialization issues often results in the very same
lines not working, it even appears there are clashes with variable
names across all scopes?. I hope you got some feedback or pointers on
the issues above.

Kind Regards,
Norbert


Example Pipeline Code follows.
This should fetch some additional modules (the url-mapping shouldnt be
coded in the script and come from a configuration file later), a file
.builddepencies should define name and potentially revision for a
project, should be able to override these (thus a map seems
fitting).All repositories should be checked out from the pipeline
script, remaining build can (and will) be run from shell scripts.

---
// Gonna be a separate, global function when its grown up
def repomap = [
'myscripts': 'ssh://git@localhost:10022/scripts.git',
'myprojekt': 'ssh://git@localhost:10022/myprojekt.git'
]

node {
// Mark the code checkout 'stage'
stage 'Checkout'

// Get some code from a GitHub repository
//git branch: 'master', url: repomap['myprojekt']
sh('echo "fake checkout: ' +  repomap['myprojekt'] + '"')
sh('echo "myscripts=somebranch" > .builddepencies')

// Get name of needed modules
def depmap = [:]
readFile('.builddepencies').eachLine {
def m = (it =~ /([^=]*)=?(.*)/)
depmap[m[0][1].trim()]=m[0][2].trim()
m = null
}

// Just a check so it works (after script approval )
depmap.each {
println 'map: ' + it.key + '=' + it.value
}
// Weird !
println 'it still defined: ' + it.key
// Weirder, no name-scopes in functions?, why is this not working ??
createDList (depmap)

depmap.each {
name = it.key
revision = it.value
it = null // Ok, I could find my way around serialization issues..
dir (name) {
//git url: repomap[name], changelog: false, poll: false
sh('echo "fake checkout: ' +  repomap[name] + '"')
}
}
println 'end'
}

@NonCPS
def createDList(depmap) {
// Need to change the iterator name, then this works?
depmap.each {
println 'map: ' + it.key + '=' + it.value
}
}

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CADYdroP99RG%2BcShBvVTbsCrrSs1oFDxRo%3DiYsRqTKcevhciBRw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.