See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Direct/190/display/redirect?page=changes>
Changes: [yifanzou] [BEAM-4963] update org.apache.httpcomponents [yifanzou] [BEAM-4906] upgrade org.apache.derby [apilloud] [BEAM-4761] Run Nexmark in thread pool [aaltay] Automate "Run validation tests" step (#6081) ------------------------------------------ [...truncated 673.03 KB...] 0000 24.6 4065.0 92000 0001 20.4 4895.2 92000 0002 14.1 7077.6 351 0003 *** not run *** 0004 *** not run *** 0005 *** not run *** 0006 *** not run *** 0007 *** not run *** 0008 *** not run *** 0009 *** not run *** 0010 *** not run *** 0011 *** not run *** 0012 *** not run *** ========================================================================================== 2018-08-04T00:14:36.864Z Generating 100000 events in batch mode Aug 04, 2018 12:14:36 AM org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel INFO: SQL: SELECT `P`.`name`, `P`.`city`, `P`.`state`, `A`.`id` FROM `beam`.`Auction` AS `A` INNER JOIN `beam`.`Person` AS `P` ON `A`.`seller` = `P`.`id` WHERE `A`.`category` = 10 AND (`P`.`state` = 'OR' OR `P`.`state` = 'ID' OR `P`.`state` = 'CA') Aug 04, 2018 12:14:36 AM org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel INFO: SQLPlan> LogicalProject(name=[$11], city=[$14], state=[$15], id=[$0]) LogicalFilter(condition=[AND(=($8, 10), OR(=($15, 'OR'), =($15, 'ID'), =($15, 'CA')))]) LogicalJoin(condition=[=($7, $10)], joinType=[inner]) BeamIOSourceRel(table=[[beam, Auction]]) BeamIOSourceRel(table=[[beam, Person]]) Aug 04, 2018 12:14:37 AM org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel INFO: BEAMPlan> BeamCalcRel(expr#0..17=[{inputs}], name=[$t11], city=[$t14], state=[$t15], id=[$t0]) BeamJoinRel(condition=[=($7, $10)], joinType=[inner]) BeamCalcRel(expr#0..9=[{inputs}], expr#10=[10], expr#11=[=($t8, $t10)], proj#0..9=[{exprs}], $condition=[$t11]) BeamIOSourceRel(table=[[beam, Auction]]) BeamCalcRel(expr#0..7=[{inputs}], expr#8=['OR'], expr#9=[=($t5, $t8)], expr#10=['ID'], expr#11=[=($t5, $t10)], expr#12=['CA'], expr#13=[=($t5, $t12)], expr#14=[OR($t9, $t11, $t13)], proj#0..7=[{exprs}], $condition=[$t14]) BeamIOSourceRel(table=[[beam, Person]]) 2018-08-04T00:14:51.883Z Waiting for main pipeline to 'finish' 2018-08-04T00:14:51.883Z DONE SqlQuery3 2018-08-04T00:14:51.884Z Running query:4; exportSummaryToBigQuery:true; numEvents:10000 2018-08-04T00:14:51.884Z skipping since configuration is not implemented 2018-08-04T00:14:51.884Z Running query:5; exportSummaryToBigQuery:true ========================================================================================== Run started 2018-08-04T00:13:31.860Z and ran for PT80.024S Default configuration: {"debug":true,"query":0,"sourceType":"DIRECT","sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1} Configurations: Conf Description 0000 query:0; exportSummaryToBigQuery:true 0001 query:1; exportSummaryToBigQuery:true 0002 query:2; exportSummaryToBigQuery:true 0003 query:3; exportSummaryToBigQuery:true 0004 query:4; exportSummaryToBigQuery:true; numEvents:10000 0005 query:5; exportSummaryToBigQuery:true 0006 query:6; exportSummaryToBigQuery:true; numEvents:10000 0007 query:7; exportSummaryToBigQuery:true 0008 query:8; exportSummaryToBigQuery:true 0009 query:9; exportSummaryToBigQuery:true; numEvents:10000 0010 query:10; exportSummaryToBigQuery:true 0011 query:11; exportSummaryToBigQuery:true 0012 query:12; exportSummaryToBigQuery:true Performance: Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline) 0000 24.6 4065.0 92000 0001 20.4 4895.2 92000 0002 14.1 7077.6 351 0003 14.5 6917.1 580 0004 *** not run *** 0005 *** not run *** 0006 *** not run *** 0007 *** not run *** 0008 *** not run *** 0009 *** not run *** 0010 *** not run *** 0011 *** not run *** 0012 *** not run *** ========================================================================================== ========================================================================================== Run started 2018-08-04T00:13:31.860Z and ran for PT80.025S Default configuration: {"debug":true,"query":0,"sourceType":"DIRECT","sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1} Configurations: Conf Description 0000 query:0; exportSummaryToBigQuery:true 0001 query:1; exportSummaryToBigQuery:true 0002 query:2; exportSummaryToBigQuery:true 0003 query:3; exportSummaryToBigQuery:true 0004 query:4; exportSummaryToBigQuery:true; numEvents:10000 0005 query:5; exportSummaryToBigQuery:true 0006 query:6; exportSummaryToBigQuery:true; numEvents:10000 0007 query:7; exportSummaryToBigQuery:true 0008 query:8; exportSummaryToBigQuery:true 0009 query:9; exportSummaryToBigQuery:true; numEvents:10000 0010 query:10; exportSummaryToBigQuery:true 0011 query:11; exportSummaryToBigQuery:true 0012 query:12; exportSummaryToBigQuery:true Performance: Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline) 0000 24.6 4065.0 92000 Exception in thread "main" java.lang.NullPointerException 0001 20.4 4895.2 92000 0002 14.1 7077.6 351 0003 14.5 6917.1 580 at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:150) 0004 *** not run *** at org.apache.beam.sdk.nexmark.Main.main(Main.java:475) 0005 *** not run *** 0006 *** not run *** 0007 *** not run *** 0008 *** not run *** 0009 *** not run *** 0010 *** not run *** 0011 *** not run *** 0012 *** not run *** ========================================================================================== 2018-08-04T00:14:51.888Z Generating 100000 events in batch mode Aug 04, 2018 12:14:51 AM org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel INFO: SQL: SELECT `AuctionBids`.`auction`, `AuctionBids`.`num` FROM (SELECT `B1`.`auction`, COUNT(*) AS `num`, HOP_START(`B1`.`dateTime`, INTERVAL '5' SECOND, INTERVAL '10' SECOND) AS `starttime` FROM `beam`.`Bid` AS `B1` GROUP BY `B1`.`auction`, HOP(`B1`.`dateTime`, INTERVAL '5' SECOND, INTERVAL '10' SECOND)) AS `AuctionBids` INNER JOIN (SELECT MAX(`CountBids`.`num`) AS `maxnum`, `CountBids`.`starttime` FROM (SELECT COUNT(*) AS `num`, HOP_START(`B2`.`dateTime`, INTERVAL '5' SECOND, INTERVAL '10' SECOND) AS `starttime` FROM `beam`.`Bid` AS `B2` GROUP BY `B2`.`auction`, HOP(`B2`.`dateTime`, INTERVAL '5' SECOND, INTERVAL '10' SECOND)) AS `CountBids` GROUP BY `CountBids`.`starttime`) AS `MaxBids` ON `AuctionBids`.`starttime` = `MaxBids`.`starttime` AND `AuctionBids`.`num` >= `MaxBids`.`maxnum` Aug 04, 2018 12:14:51 AM org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel INFO: SQLPlan> LogicalProject(auction=[$0], num=[$1]) LogicalJoin(condition=[AND(=($2, $4), >=($1, $3))], joinType=[inner]) LogicalProject(auction=[$0], num=[$2], starttime=[$1]) LogicalAggregate(group=[{0, 1}], num=[COUNT()]) LogicalProject(auction=[$0], $f1=[HOP($3, 5000, 10000)]) BeamIOSourceRel(table=[[beam, Bid]]) LogicalProject(maxnum=[$1], starttime=[$0]) LogicalAggregate(group=[{0}], maxnum=[MAX($1)]) LogicalProject(starttime=[$1], num=[$0]) LogicalProject(num=[$2], starttime=[$1]) LogicalAggregate(group=[{0, 1}], num=[COUNT()]) LogicalProject(auction=[$0], $f1=[HOP($3, 5000, 10000)]) BeamIOSourceRel(table=[[beam, Bid]]) Aug 04, 2018 12:14:52 AM org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel INFO: BEAMPlan> BeamCalcRel(expr#0..4=[{inputs}], proj#0..1=[{exprs}]) BeamJoinRel(condition=[AND(=($2, $4), >=($1, $3))], joinType=[inner]) BeamCalcRel(expr#0..2=[{inputs}], auction=[$t0], num=[$t2], starttime=[$t1]) BeamAggregationRel(group=[{0, 1}], num=[COUNT()]) BeamCalcRel(expr#0..4=[{inputs}], expr#5=[5000], expr#6=[10000], expr#7=[HOP($t3, $t5, $t6)], auction=[$t0], $f1=[$t7]) BeamIOSourceRel(table=[[beam, Bid]]) BeamCalcRel(expr#0..1=[{inputs}], maxnum=[$t1], starttime=[$t0]) BeamAggregationRel(group=[{1}], maxnum=[MAX($0)]) BeamCalcRel(expr#0..2=[{inputs}], num=[$t2], starttime=[$t1]) BeamAggregationRel(group=[{0, 1}], num=[COUNT()]) BeamCalcRel(expr#0..4=[{inputs}], expr#5=[5000], expr#6=[10000], expr#7=[HOP($t3, $t5, $t6)], auction=[$t0], $f1=[$t7]) BeamIOSourceRel(table=[[beam, Bid]]) 2018-08-04T00:15:18.822Z Waiting for main pipeline to 'finish' 2018-08-04T00:15:18.822Z DONE SqlQuery5 2018-08-04T00:15:18.823Z Running query:6; exportSummaryToBigQuery:true; numEvents:10000 2018-08-04T00:15:18.823Z skipping since configuration is not implemented 2018-08-04T00:15:18.823Z Running query:7; exportSummaryToBigQuery:true 2018-08-04T00:15:18.827Z Generating 100000 events in batch mode Aug 04, 2018 12:15:18 AM org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel INFO: SQL: SELECT `B`.`auction`, `B`.`price`, `B`.`bidder`, `B`.`dateTime`, `B`.`extra` FROM (SELECT `B`.`auction`, `B`.`price`, `B`.`bidder`, `B`.`dateTime`, `B`.`extra`, TUMBLE_START(`B`.`dateTime`, INTERVAL '10' SECOND) AS `starttime` FROM `beam`.`Bid` AS `B` GROUP BY `B`.`auction`, `B`.`price`, `B`.`bidder`, `B`.`dateTime`, `B`.`extra`, TUMBLE(`B`.`dateTime`, INTERVAL '10' SECOND)) AS `B` INNER JOIN (SELECT MAX(`B1`.`price`) AS `maxprice`, TUMBLE_START(`B1`.`dateTime`, INTERVAL '10' SECOND) AS `starttime` FROM `beam`.`Bid` AS `B1` GROUP BY TUMBLE(`B1`.`dateTime`, INTERVAL '10' SECOND)) AS `B1` ON `B`.`starttime` = `B1`.`starttime` AND `B`.`price` = `B1`.`maxprice` Aug 04, 2018 12:15:18 AM org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel INFO: SQLPlan> LogicalProject(auction=[$0], price=[$1], bidder=[$2], dateTime=[$3], extra=[$4]) LogicalJoin(condition=[AND(=($5, $7), =($1, $6))], joinType=[inner]) LogicalProject(auction=[$0], price=[$1], bidder=[$2], dateTime=[$3], extra=[$4], starttime=[$5]) LogicalAggregate(group=[{0, 1, 2, 3, 4, 5}]) LogicalProject(auction=[$0], price=[$2], bidder=[$1], dateTime=[$3], extra=[$4], $f5=[TUMBLE($3, 10000)]) BeamIOSourceRel(table=[[beam, Bid]]) LogicalProject(maxprice=[$1], starttime=[$0]) LogicalAggregate(group=[{0}], maxprice=[MAX($1)]) LogicalProject($f0=[TUMBLE($3, 10000)], price=[$2]) BeamIOSourceRel(table=[[beam, Bid]]) Aug 04, 2018 12:15:18 AM org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel INFO: BEAMPlan> BeamCalcRel(expr#0..7=[{inputs}], proj#0..4=[{exprs}]) BeamJoinRel(condition=[AND(=($5, $7), =($1, $6))], joinType=[inner]) BeamCalcRel(expr#0..5=[{inputs}], proj#0..5=[{exprs}]) BeamAggregationRel(group=[{0, 1, 2, 3, 4, 5}]) BeamCalcRel(expr#0..4=[{inputs}], expr#5=[10000], expr#6=[TUMBLE($t3, $t5)], auction=[$t0], price=[$t2], bidder=[$t1], dateTime=[$t3], extra=[$t4], $f5=[$t6]) BeamIOSourceRel(table=[[beam, Bid]]) BeamCalcRel(expr#0..1=[{inputs}], maxprice=[$t1], starttime=[$t0]) BeamAggregationRel(group=[{0}], maxprice=[MAX($1)]) BeamCalcRel(expr#0..4=[{inputs}], expr#5=[10000], expr#6=[TUMBLE($t3, $t5)], $f0=[$t6], price=[$t2]) BeamIOSourceRel(table=[[beam, Bid]]) 2018-08-04T00:16:08.047Z Waiting for main pipeline to 'finish' 2018-08-04T00:16:08.047Z DONE SqlQuery7 2018-08-04T00:16:08.048Z Running query:8; exportSummaryToBigQuery:true 2018-08-04T00:16:08.048Z skipping since configuration is not implemented 2018-08-04T00:16:08.048Z Running query:9; exportSummaryToBigQuery:true; numEvents:10000 2018-08-04T00:16:08.048Z skipping since configuration is not implemented 2018-08-04T00:16:08.048Z Running query:10; exportSummaryToBigQuery:true 2018-08-04T00:16:08.048Z skipping since configuration is not implemented 2018-08-04T00:16:08.048Z Running query:11; exportSummaryToBigQuery:true 2018-08-04T00:16:08.049Z skipping since configuration is not implemented 2018-08-04T00:16:08.049Z Running query:12; exportSummaryToBigQuery:true 2018-08-04T00:16:08.049Z skipping since configuration is not implemented > Task :beam-sdks-java-nexmark:run FAILED :beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 7,5,main]) completed. Took 2 mins 37.573 secs. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':beam-sdks-java-nexmark:run'. > Process 'command '/usr/local/asfpackages/java/jdk1.8.0_152/bin/java'' > finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. See https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 2m 51s 62 actionable tasks: 2 executed, 60 up-to-date Publishing build scan... https://gradle.com/s/toi7t5qbijzxg Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure