Hi Michael,
I had work pipeline job. But after add more environments (envList) job
broke with log:
[Pipeline] End of Pipeline
an exception which occurred:
in field delegate
in field closures
in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@1845f26
Caused: java.io.NotSerializableException: java.util.LinkedHashMap$Entry
at
org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
at
org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
at
org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
at
org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
at
org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at java.util.LinkedHashMap.internalWriteEntries(LinkedHashMap.java:333)
at java.util.HashMap.writeObject(HashMap.java:1362)
at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
I use NonCPS function sortSuites() and it always return list. I don't
have more ideas how to fix. Can you help?
Thank you!
My test pipeline:
#!groovy
def envList = []
envList.add('NODE=behat3 ORO_PHP=7.1 ORO_APP=application/commerce-crm-ee
ORO_TEST_SUITE=behat ORO_DB=pgsql BEHAT_MODE=standalone')
envList.add('NODE=docker-doc ORO_TEST_SUITE=documentation
ORO_APP=documentation/crm ORO_DOC=OroCRM_Documentation')
envList.add('NODE=docker-doc ORO_TEST_SUITE=documentation
ORO_APP=documentation/commerce ORO_DOC=OroCommerce_Documentation')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_APP=application/crm-enterprise ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=crm-enterprise_1.12 ORO_SE=ES ES_VER=2.4.3
UPGRADE=true TEST_RUNNER_OPTIONS=--group=search,dist')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_APP=application/crm-enterprise ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=crm-enterprise_2.0.0 ORO_SE=ES ES_VER=2.4.3')
envList.add('NODE=func_2cpu ORO_PHP=7.0
ORO_APP=application/crm-enterprise ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=crm-enterprise_1.12 UPGRADE=true')
envList.add('NODE=func ORO_PHP=7.1
ORO_APP=application/commerce-enterprise ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=commerce_1.3.0 ORO_SE=ES ES_VER=2.3.5')
envList.add('NODE=func ORO_PHP=7.0
ORO_APP=application/commerce-enterprise ORO_TEST_SUITE=functional
ORO_DB=MYSQL ORO_INSTALLED=commerce_1.0.0')
envList.add('NODE=func ORO_PHP=7.1
ORO_APP=application/commerce ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=commerce_1.0.0')
envList.add('NODE=func ORO_PHP=7.1
ORO_APP=application/commerce-crm ORO_TEST_SUITE=functional
ORO_DB=PG ORO_ENABLE_PRICE_SHARDING=true')
envList.add('NODE=func ORO_PHP=7.1
ORO_APP=application/commerce-crm-ee ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=commerce_1.3.0')
envList.add('NODE=func ORO_PHP=7.1
ORO_APP=application/commerce-crm-ee ORO_TEST_SUITE=functional
ORO_DB=MYSQL ORO_INSTALLED=crm-enterprise_1.12 UPGRADE=true')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_APP=application/platform ORO_TEST_SUITE=functional
ORO_DB=PG')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_APP=application/crm ORO_TEST_SUITE=functional
ORO_DB=MYSQL')
envList.add('NODE=func_2cpu ORO_PHP=7.1 ORO_TEST_SUITE=behat_wiring')
envList.add('NODE=func ORO_PHP=7.1 ORO_TEST_SUITE=javascript JS=YES CS=YES')
envList.add('NODE=func_2cpu ORO_PHP=7.0
ORO_APP=application/commerce-crm-ee ORO_TEST_SUITE=unit')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_APP=application/commerce-crm-ee ORO_TEST_SUITE=unit')
def behatList = []
def enviroments = [:]
for (int i = 0; i < envList.size() ; i++) {
int index=i, e = i+1
enviroments["TestEnv_${e}"] = {
stage ("TestEnv_${e}"){
withEnv(["NETWORK=${index}"] + ["TESTENV=TestEnv_${e}"] +
envList[index].tokenize()) {
if (env.ORO_TEST_SUITE == 'behat') {
node() {
ws("${HOME}/workspace/dev_${EXECUTOR_NUMBER}") {
sh """
echo "Install"
"""
behatStr = sh (
script: 'echo AutoSuiteSet_1 AutoSuiteSet_2
AutoSuiteSet_3',
returnStdout: true
).trim()
}
}
behatList=behatStr.tokenize()
echo "SUITES=${behatList}"
behatList=sortSuites(behatList)
echo "SUITES SORTED=${behatList}"
echo "behatList TYPE=${behatList.getClass()}"
def enviroments_b = [:]
for (int j = 0; j < behatList.size() ; j++) {
int index_b=j
enviroments_b["TestEnv_${e}
behat=${behatList[index_b]}"] = {
withEnv(["BEHAT_SUIT=${behatList[index_b]}"]) {
node() {
ws("${HOME}/workspace/dev_${EXECUTOR_NUMBER}") {
sh """
echo "Run BEHAT_SUIT=$BEHAT_SUIT"
"""
} //workspace
} //node
} //withEnv
} //enviroments_b
} //for
try {
parallel enviroments_b
} catch (error) {
throw (error)
}
} else if (env.ORO_TEST_SUITE == 'functional' ||
env.ORO_TEST_SUITE == 'unit' || env.ORO_TEST_SUITE == 'behat_wiring' ||
env.ORO_TEST_SUITE == 'javascript'){
node() {
ws("${HOME}/workspace/dev_${EXECUTOR_NUMBER}") {
sh """
echo "Run $ORO_TEST_SUITE"
"""
}
}
} else {
node() {
ws("${HOME}/workspace/dev_${EXECUTOR_NUMBER}") {
sh """
echo "Run $ORO_TEST_SUITE"
"""
}
}
}
}
}
}
}
try {
parallel enviroments
} catch (error) {
throw (error)
}
@NonCPS
def sortSuites(suites){
def suitesStat=[AutoSuiteSet_1:435, AutoSuiteSet_2:323,
AutoSuiteSet_3:363]
def suites_time = suitesStat
// def suites_time = [:]
def suitesMap = [:]
if (suites_time.size()>0){
def timeLimit = suites_time.values().max() //find max value
for(s in suites){
x = suites_time.find{ artifact -> artifact.key == s}
if(x){
suitesMap.put(x.key, x.value)
}else{
suitesMap.put(s, timeLimit)
}
}
}else{
def suitesList = suites.collect { t -> [t] }
return suitesList
}
tasks = divideList(suitesMap)
tasks = sortListInMap(tasks)
tasksList = tasks.collect { t -> t.keySet()}
return tasksList
}
@NonCPS
def divideList(inputMap){
// Sort an incoming array in descending order
inputMap=sortMap(inputMap)
maxValue = inputMap.values().max()
// Resulting set of maps
def outputSet = []
// Variable for storing an individual map
def outputMap = [:]
for (inputElement in inputMap) {
// We add to the temporary map and consider the resulting sum of
the maps items
possibleMap = outputMap + [inputElement]
possibleMapSum = possibleMap.values().sum()
// If the amount in the temporary map is less than maxValue, go
to the next iteration, we will try to supplement the map with one more
element
if (possibleMapSum < maxValue) {
outputMap = possibleMap
// If the amount in the temporary map is maxValue, add a
temporary map to the result set and zero the temporary map
}else if(possibleMapSum == maxValue) {
outputSet.add(possibleMap)
outputMap = [:]
// If the amount in the temporary map exceeded maxValue, add
the previous version of the temporary map to the result set (without the
current element), and create a new temporary map with the current element
}else if(possibleMapSum > maxValue) {
outputSet.add(outputMap)
outputMap = [:]
outputMap.put(inputElement.key, inputElement.value)
}
}
if (outputMap) {
outputSet.add(outputMap)
}
return outputSet
}
@NonCPS
def sortMap(map) {
map.sort{a, b -> b.value <=> a.value }
}
@NonCPS
def sortListInMap(map) {
map.sort { a, b -> b.values().sum() <=> a.values().sum() }
}
06.08.2017 17:47, Michael Pailloncy пишет:
It seems like there are some variables used out of their scope and
some misused Groovy methods.
Here is a working version, IIUC your pipeline :-)
#!groovy
def behatList =['AutoSuiteSet_0', 'AutoSuiteSet_1', 'AutoSuiteSet_2',
'AutoSuiteSet_3', 'AutoSuiteSet_4', 'AutoSuiteSet_5']
def suitesStat=[AutoSuiteSet_0:0, AutoSuiteSet_1:1, AutoSuiteSet_2:2,
AutoSuiteSet_3:3, AutoSuiteSet_4:4, AutoSuiteSet_5:5]
stage("test") {
node('master') {
behatList2=sortSuites(behatList, suitesStat)
echo "SUITES2=${behatList2}"
}
}
@NonCPS
def sortSuites(suites, suites_time){
def timeLimit = suites_time.values().max()
def suitesMap= [:]
for(s in suites){
def x = suites_time.find{ artifact -> artifact.key == s}
if(x){
suitesMap.put(x.key, x.value)
}else{
suitesMap.put(s, timeLimit)
}
}
def tasks = [suitesMap]
timeLimit = suitesMap.values().max()
while(canSplit(tasks, timeLimit)) {
tasks = tasks.collect { t ->
if(checkLimit(t, timeLimit)){
t = splitTo2(t)
}
t
}.flatten()
}
tasks.sort { a, b -> b.values().sum() <=> a.values().sum() }
// tasks = tasks.collect { t -> t.keySet()} // not working, multiple
elements are collected here
tasks = tasks.collectMany { t -> t.keySet()}
return tasks
}
@NonCPS
def checkLimit(t, timeLimit) {
if(t.values().sum()>timeLimit && t.size()>1){
return true
}else{
return false
}
}
@NonCPS
def canSplit(tasks, timeLimit) {
for(t in tasks) {
if(checkLimit(t, timeLimit)){
return true
}
}
return false
}
@NonCPS
def splitTo2(int_map) {
A=[:]
B=[:]
// int_map.sort{it.value} not working
// for(n in int_map.sort{it.value}) {
for(n in int_map) {
if (A.size() < B.size()) {
A.put(n.key, n.value)
}else{
B.put(n.key, n.value)
}
}
return [A, B]
}
However, what do you want to achieve exactly ? One of the main
advantage of Pipeline is its resumability after restart. But if you
use too many @NonCPS, you avoid this feature and this could lead to
unexpected behaviour sometimes.
Hopefully it helps :-)
2017-08-04 19:17 GMT+02:00 Slava Dubrovskiy <dub...@gmail.com
<mailto:dub...@gmail.com>>:
Hi.
I use a special algorithm to pre-sort the steps for parallel start.
Here is my test pipeline:
|
#!groovy
defbehatList
=['AutoSuiteSet_0','AutoSuiteSet_1','AutoSuiteSet_2','AutoSuiteSet_3','AutoSuiteSet_4','AutoSuiteSet_5']
defsuitesStat=[AutoSuiteSet_0:0,AutoSuiteSet_1:1,AutoSuiteSet_2:2,AutoSuiteSet_3:3,AutoSuiteSet_4:4,AutoSuiteSet_5:5]
stage("test"){
node('master'){
behatList2=sortSuites(behatList,suitesStat)
echo "SUITES2=${behatList2}"
}
}
@NonCPS
defsortSuites(suites,suites_time){
timeLimit =suites_time.values().max()
defsuitesMap=[:]
for(s insuites){
x=suites_time.find{artifact ->artifact.key ==s}
if(x){
suitesMap.put(x.key,x.value)
}else{
suitesMap.put(s,timeLimit)
}
}
tasks =[suitesMap]
timeLimit =suitesMap.values().max()
while(canSplit()){
tasks =tasks.collect {t ->
if(checkLimit(t)){
t =splitTo2(t)
}
t
}.flatten()
}
tasks.sort {a,b ->b.values().sum()<=>a.values().sum()}
tasks =tasks.collect {t ->t.keySet()}
returntasks
}
@NonCPS
defcheckLimit(t){
if(t.values().sum()>timeLimit &&t.size()>1){
returntrue
}else{
returnfalse
}
}
@NonCPS
defcanSplit(){
for(t intasks){
if(checkLimit(t)){
returntrue
}
}
returnfalse
}
@NonCPS
defsplitTo2(int_map){
A=[:]
B=[:]
for(n inint_map.sort{it.value}){
if(A.size()<B.size()){
A.put(n.key,n.value)
}else{
B.put(n.key,n.value)
}
}
return[A,B]
}
|
If I run it, I get the error:
|
an exception which occurred:
infield delegate
infield closures
inobjectorg.jenkinsci.plugins.workflow.cps.CpsThreadGroup@7fd2cde1
Caused:java.io.NotSerializableException:java.util.LinkedHashMap$Entry
at
org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
at
org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
at
org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
at
org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOu
...
|
Please, help me what is wrong?
All methods under @NonCPS directive.
--
WBR,
Slava.
--
You received this message because you are subscribed to the Google
Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to jenkinsci-users+unsubscr...@googlegroups.com
<mailto:jenkinsci-users+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/e7a79c56-512f-4706-ab65-9a347966abb3%40googlegroups.com
<https://groups.google.com/d/msgid/jenkinsci-users/e7a79c56-512f-4706-ab65-9a347966abb3%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google
Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to jenkinsci-users+unsubscr...@googlegroups.com
<mailto:jenkinsci-users+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/CAPO77c0_uu7GwQGL3GNGoyh-D878mrbiMj-1fUZefgN0bRPmsA%40mail.gmail.com
<https://groups.google.com/d/msgid/jenkinsci-users/CAPO77c0_uu7GwQGL3GNGoyh-D878mrbiMj-1fUZefgN0bRPmsA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
WBD,
Viacheslav Dubrovskyi
--
You received this message because you are subscribed to the Google Groups "Jenkins
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/2493fd53-6fd8-e528-462f-663457813e98%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
