[jira] [Work logged] (BEAM-5210) VARBINARY is not supported in BeamSqlPrimitive

2018-08-26 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-5210?focusedWorklogId=138178&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-138178
 ]

ASF GitHub Bot logged work on BEAM-5210:


Author: ASF GitHub Bot
Created on: 26/Aug/18 06:59
Start Date: 26/Aug/18 06:59
Worklog Time Spent: 10m 
  Work Description: XuMingmin commented on issue #6270: [BEAM-5210] Support 
VARBINARY in BeamSqlPrimitive
URL: https://github.com/apache/beam/pull/6270#issuecomment-416018485
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 138178)
Time Spent: 2h 40m  (was: 2.5h)

> VARBINARY is not supported in BeamSqlPrimitive
> --
>
> Key: BEAM-5210
> URL: https://issues.apache.org/jira/browse/BEAM-5210
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> See error as below
> {code}
> Caused by: java.lang.UnsupportedOperationException: Unsupported Beam SQL type 
> in expression: VARBINARY
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive.accept(BeamSqlPrimitive.java:158)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive.(BeamSqlPrimitive.java:44)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive.of(BeamSqlPrimitive.java:56)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlInputRefExpression.evaluate(BeamSqlInputRefExpression.java:42)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlExpressionEnvironments$ListEnvironment.getLocalRef(BeamSqlExpressionEnvironments.java:92)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlLocalRefExpression.evaluate(BeamSqlLocalRefExpression.java:44)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.lambda$execute$0(BeamSqlFnExecutor.java:557)
>   at 
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
>   at 
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
>   at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
>   at 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
>   at 
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
>   at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
>   at 
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.execute(BeamSqlFnExecutor.java:558)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamCalcRel$CalcFn.processElement(BeamCalcRel.java:116)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Java_Cron #273

2018-08-26 Thread Apache Jenkins Server 
See 


--
[...truncated 15.86 MB...]
INFO: 2018-08-26T12:18:18.680Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Aug 26, 2018 12:18:28 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:18:18.991Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Aug 26, 2018 12:18:28 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:18:19.032Z: Unzipping flatten s13 for input 
s12.org.apache.beam.sdk.values.PCollection.:384#639855871860d444
Aug 26, 2018 12:18:28 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:18:19.076Z: Fusing unzipped copy of 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void 
key/AddKeys/Map, through flatten 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections,
 into producer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum
Aug 26, 2018 12:18:28 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:18:19.121Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
Aug 26, 2018 12:18:28 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:18:19.164Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key
Aug 26, 2018 12:18:28 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:18:19.210Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Aug 26, 2018 12:18:28 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:18:19.254Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Aug 26, 2018 12:18:28 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:18:19.301Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Aug 26, 2018 12:18:28 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:18:19.346Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Aug 26, 2018 12:18:28 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:18:19.391Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map
Aug 26, 2018 12:18:28 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:18:19.429Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles
Aug 26, 2018 12:18:28 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:18:19.475Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Aug 26, 2018 12:18:28 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:18:19.519Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Ass

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1353

2018-08-26 Thread Apache Jenkins Server 
See 


--
[...truncated 21.20 MB...]
Aug 26, 2018 12:43:22 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:43:20.762Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Extract
Aug 26, 2018 12:43:22 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:43:20.808Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForKey) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForKeys/Read
Aug 26, 2018 12:43:22 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:43:20.845Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Extract
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues
Aug 26, 2018 12:43:22 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:43:20.886Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/WithKeys/AddKeys/Map into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/ParDo(CollectWindows)
Aug 26, 2018 12:43:22 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:43:20.932Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Read
Aug 26, 2018 12:43:22 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:43:20.977Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Write mutations to Spanner into 
SpannerIO.Write/Write mutations to Cloud Spanner/Batch mutations together
Aug 26, 2018 12:43:22 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:43:21.022Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey+SpannerIO.Write/Write
 mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Partial
Aug 26, 2018 12:43:22 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:43:21.061Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey+SpannerIO.Write/Write
 mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Partial
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 0/Sample.Any/Combine.globally(SampleAny)/WithKeys/AddKeys/Map
Aug 26, 2018 12:43:22 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:43:21.101Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Group by partition/Reify into SpannerIO.Write/Write 
mutations to Cloud Spanner/Partition input
Aug 26, 2018 12:43:22 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:43:21.147Z: Fusing consumer SpannerIO.Write/To mutation 
group into ParDo(GenerateMutations)
Aug 26, 2018 12:43:22 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:43:21.186Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Group by partition/GroupByWindow into 
SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Read
Aug 26, 2018 12:43:22 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T12:43:21.229Z: Fusing

[jira] [Work logged] (BEAM-3310) Push metrics to a backend in an runner agnostic way

2018-08-26 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-3310?focusedWorklogId=138207&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-138207
 ]

ASF GitHub Bot logged work on BEAM-3310:


Author: ASF GitHub Bot
Created on: 26/Aug/18 15:30
Start Date: 26/Aug/18 15:30
Worklog Time Spent: 10m 
  Work Description: JozoVilcek commented on issue #4548: [BEAM-3310] 
Metrics pusher
URL: https://github.com/apache/beam/pull/4548#issuecomment-416046976
 
 
   Beam metrics are available through Flink UI and also being pushed to 
`MetricReporter`. But Flink and Beam world of metrics are sort of disconnected 
for the user. In Beam, metric is defined by `(metricType: [Counter, 
Distribution, Gauge], namespace: String, name: String)` maybe also `stepName: 
String` but that one is kind of internal to beam metrics system.
   
   In flink `MetricReporter` world, user gets notification about added metrics 
he can react to. They have form of `(metricType: Metric, metricName: String, 
group: MetricGroup)` . This form us much richer and contains some extra 
dimensions to beam model within `MetricGroup`. One have to be very much aware 
to internals, to be able to work with this.
   
   Metric pusher (and whole runner agnostic metrics idea) is currently the only 
easy way how to conveniently retrieve aggregated metrics from Beam definition 
domain. But, this feature is not always available, not in jobs launched in 
detached mode. Therefore user have now only the option of implementing custom 
Flink MetricReporter and extract metrics from there (requires a lot of internal 
insight).
   
   The question is, how can this be improved and have a convenience way of 
reporting beam metrics also in detached jobs. I hope I do make sense and am not 
confusing things too much  


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 138207)
Time Spent: 17h  (was: 16h 50m)

> Push metrics to a backend in an runner agnostic way
> ---
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-extensions-metrics, sdk-java-core
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 17h
>  Remaining Estimate: 0h
>
> The idea is to avoid relying on the runners to provide access to the metrics 
> (either at the end of the pipeline or while it runs) because they don't have 
> all the same capabilities towards metrics (e.g. spark runner configures sinks 
>  like csv, graphite or in memory sinks using the spark engine conf). The 
> target is to push the metrics in the common runner code so that no matter the 
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML: 
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5180) Broken FileResultCoder via parseSchema change

2018-08-26 Thread Jozef Vilcek (JIRA) 


[ 
https://issues.apache.org/jira/browse/BEAM-5180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592931#comment-16592931
 ] 

Jozef Vilcek commented on BEAM-5180:


of put a strong requirement that every path declares a `scheme`. That way a 
default_scheme could be dropped and never guessed
[https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java#L443]

windows paths will needs to be `file:///c:/something/in/here`

But my sight is limited so this might not feasible for some reasons I do not 
foresee. Just an idea

> Broken FileResultCoder via parseSchema change
> -
>
> Key: BEAM-5180
> URL: https://issues.apache.org/jira/browse/BEAM-5180
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.6.0
>Reporter: Jozef Vilcek
>Assignee: Ankur Goenka
>Priority: Blocker
> Fix For: 2.7.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Recently this commit
> [https://github.com/apache/beam/commit/3fff58c21f94415f3397e185377e36d3df662384]
> introduced more strict schema parsing which is breaking the contract between 
> _FileResultCoder_ and _FileSystems.matchNewResource()_.
> Coder takes _ResourceId_ and serialize it via `_toString_` methods and then 
> relies on filesystem being able to parse it back again. Having strict 
> _scheme://_ breaks this at least for Hadoop filesystem which use _URI_ for 
> _ResourceId_ and produce _toString()_ in form of `_hdfs:/some/path_`
> I guess the _ResourceIdCoder_ is suffering the same problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PreCommit_Java_Cron #274

2018-08-26 Thread Apache Jenkins Server 
See

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1354

2018-08-26 Thread Apache Jenkins Server 
See 


--
[...truncated 19.45 MB...]
Aug 26, 2018 6:42:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T18:42:20.487Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Batch mutations together into SpannerIO.Write/Write 
mutations to Cloud Spanner/Group by partition/GroupByWindow
Aug 26, 2018 6:42:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T18:42:20.532Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as 
view/ParMultiDo(ToIsmRecordForMapLike) into SpannerIO.Write/Write mutations to 
Cloud Spanner/Keys sample as 
view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Aug 26, 2018 6:42:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T18:42:20.578Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)
Aug 26, 2018 6:42:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T18:42:20.622Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/Combine.GroupedValues/Extract
Aug 26, 2018 6:42:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T18:42:20.667Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Reify
Aug 26, 2018 6:42:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T18:42:20.711Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Extract into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/Combine.GroupedValues
Aug 26, 2018 6:42:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T18:42:20.756Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey/Reify into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/Combine.GroupedValues/Partial
Aug 26, 2018 6:42:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T18:42:20.796Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey+SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Partial into 
SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys
Aug 26, 2018 6:42:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T18:42:20.837Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify
Aug 26, 2018 6:42:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T18:42:20.872Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/ParMultiDo(ToIsmRecordForMapLike)
Aug 26, 2018 6:42:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T18:42:20.908Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/View.AsList/ParDo(ToIsmRecordForGlobalWindow) into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap
Aug 26, 2018 6:42:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T18:42:20.950Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/ParMultiDo(ToIsmRecordForMapLike)
Aug 26, 2018 6:42:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-26T18:42:20.995Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/T

[jira] [Work logged] (BEAM-3310) Push metrics to a backend in an runner agnostic way

2018-08-26 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-3310?focusedWorklogId=138261&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-138261
 ]

ASF GitHub Bot logged work on BEAM-3310:


Author: ASF GitHub Bot
Created on: 26/Aug/18 22:14
Start Date: 26/Aug/18 22:14
Worklog Time Spent: 10m 
  Work Description: zorro786 commented on issue #4548: [BEAM-3310] Metrics 
pusher
URL: https://github.com/apache/beam/pull/4548#issuecomment-416076601
 
 
   @JozoVilcek I don't see Beam (2.4.0) metrics from Flink (1.4.0) UI. Where do 
you see them?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 138261)
Time Spent: 17h 10m  (was: 17h)

> Push metrics to a backend in an runner agnostic way
> ---
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-extensions-metrics, sdk-java-core
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 17h 10m
>  Remaining Estimate: 0h
>
> The idea is to avoid relying on the runners to provide access to the metrics 
> (either at the end of the pipeline or while it runs) because they don't have 
> all the same capabilities towards metrics (e.g. spark runner configures sinks 
>  like csv, graphite or in memory sinks using the spark engine conf). The 
> target is to push the metrics in the common runner code so that no matter the 
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML: 
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Java_Cron #275

2018-08-26 Thread Apache Jenkins Server 
See 


--
[...truncated 16.31 MB...]
INFO: 2018-08-27T00:18:01.441Z: Expanding GroupByKey operations into 
optimizable parts.
Aug 27, 2018 12:18:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T00:18:01.488Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Aug 27, 2018 12:18:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T00:18:01.810Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Aug 27, 2018 12:18:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T00:18:01.862Z: Unzipping flatten s13 for input 
s12.org.apache.beam.sdk.values.PCollection.:384#639855871860d444
Aug 27, 2018 12:18:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T00:18:01.907Z: Fusing unzipped copy of 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void 
key/AddKeys/Map, through flatten 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections,
 into producer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum
Aug 27, 2018 12:18:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T00:18:01.946Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
Aug 27, 2018 12:18:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T00:18:01.993Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key
Aug 27, 2018 12:18:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T00:18:02.039Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Aug 27, 2018 12:18:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T00:18:02.085Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Aug 27, 2018 12:18:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T00:18:02.126Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Aug 27, 2018 12:18:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T00:18:02.175Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Aug 27, 2018 12:18:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T00:18:02.211Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map
Aug 27, 2018 12:18:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T00:18:02.260Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles
Aug 27, 2018 12:18:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T00:18:02.300Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Aug 27, 2018 12:18:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T00:18:02.335Z: Fusing consumer 
Write

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1355

2018-08-26 Thread Apache Jenkins Server 
See

[jira] [Work logged] (BEAM-5210) VARBINARY is not supported in BeamSqlPrimitive

2018-08-26 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-5210?focusedWorklogId=138276&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-138276
 ]

ASF GitHub Bot logged work on BEAM-5210:


Author: ASF GitHub Bot
Created on: 27/Aug/18 03:07
Start Date: 27/Aug/18 03:07
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6270: [BEAM-5210] Support 
VARBINARY in BeamSqlPrimitive
URL: https://github.com/apache/beam/pull/6270#issuecomment-416101800
 
 
   Thanks for this fix!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 138276)
Time Spent: 2h 50m  (was: 2h 40m)

> VARBINARY is not supported in BeamSqlPrimitive
> --
>
> Key: BEAM-5210
> URL: https://issues.apache.org/jira/browse/BEAM-5210
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> See error as below
> {code}
> Caused by: java.lang.UnsupportedOperationException: Unsupported Beam SQL type 
> in expression: VARBINARY
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive.accept(BeamSqlPrimitive.java:158)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive.(BeamSqlPrimitive.java:44)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive.of(BeamSqlPrimitive.java:56)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlInputRefExpression.evaluate(BeamSqlInputRefExpression.java:42)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlExpressionEnvironments$ListEnvironment.getLocalRef(BeamSqlExpressionEnvironments.java:92)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlLocalRefExpression.evaluate(BeamSqlLocalRefExpression.java:44)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.lambda$execute$0(BeamSqlFnExecutor.java:557)
>   at 
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
>   at 
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
>   at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
>   at 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
>   at 
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
>   at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
>   at 
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.execute(BeamSqlFnExecutor.java:558)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamCalcRel$CalcFn.processElement(BeamCalcRel.java:116)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5220) Clean TODOs and unused classes.

2018-08-26 Thread Vaclav Plajt (JIRA) 
Vaclav Plajt created BEAM-5220:
--

 Summary: Clean TODOs and unused classes.
 Key: BEAM-5220
 URL: https://issues.apache.org/jira/browse/BEAM-5220
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-euphoria
Reporter: Vaclav Plajt
Assignee: Vaclav Plajt






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3310) Push metrics to a backend in an runner agnostic way

2018-08-26 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-3310?focusedWorklogId=138297&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-138297
 ]

ASF GitHub Bot logged work on BEAM-3310:


Author: ASF GitHub Bot
Created on: 27/Aug/18 06:20
Start Date: 27/Aug/18 06:20
Worklog Time Spent: 10m 
  Work Description: JozoVilcek commented on issue #4548: [BEAM-3310] 
Metrics pusher
URL: https://github.com/apache/beam/pull/4548#issuecomment-416126131
 
 
   @zorro786 I do see them (same beam, same flink versions) under JobOverview 
-> Task Metrics -> then select task and choose metrics from there. Combo has 
prefix search so it does not really work (appears to be fixed in 1.5.0) ... 
metrics are prefixed with task index and operator / step names ... so my metric 
names are on the far right side (need scroll) ... but I see them there fine.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 138297)
Time Spent: 17h 20m  (was: 17h 10m)

> Push metrics to a backend in an runner agnostic way
> ---
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-extensions-metrics, sdk-java-core
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 17h 20m
>  Remaining Estimate: 0h
>
> The idea is to avoid relying on the runners to provide access to the metrics 
> (either at the end of the pipeline or while it runs) because they don't have 
> all the same capabilities towards metrics (e.g. spark runner configures sinks 
>  like csv, graphite or in memory sinks using the spark engine conf). The 
> target is to push the metrics in the common runner code so that no matter the 
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML: 
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PreCommit_Java_Cron #276

2018-08-26 Thread Apache Jenkins Server 
See

[jira] [Work logged] (BEAM-3310) Push metrics to a backend in an runner agnostic way

2018-08-26 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-3310?focusedWorklogId=138298&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-138298
 ]

ASF GitHub Bot logged work on BEAM-3310:


Author: ASF GitHub Bot
Created on: 27/Aug/18 06:25
Start Date: 27/Aug/18 06:25
Worklog Time Spent: 10m 
  Work Description: zorro786 commented on issue #4548: [BEAM-3310] Metrics 
pusher
URL: https://github.com/apache/beam/pull/4548#issuecomment-416126979
 
 
   @JozoVilcek Those are Flink metrics not Beam. The Beam ones are supposed to 
be under JobOverview -> Accumulators. These are the ones from your user code 
that get accumulated by Flink.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 138298)
Time Spent: 17.5h  (was: 17h 20m)

> Push metrics to a backend in an runner agnostic way
> ---
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-extensions-metrics, sdk-java-core
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 17.5h
>  Remaining Estimate: 0h
>
> The idea is to avoid relying on the runners to provide access to the metrics 
> (either at the end of the pipeline or while it runs) because they don't have 
> all the same capabilities towards metrics (e.g. spark runner configures sinks 
>  like csv, graphite or in memory sinks using the spark engine conf). The 
> target is to push the metrics in the common runner code so that no matter the 
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML: 
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1356

2018-08-26 Thread Apache Jenkins Server 
See 


--
[...truncated 19.96 MB...]
Aug 27, 2018 6:42:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T06:42:07.539Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Batch mutations together into SpannerIO.Write/Write 
mutations to Cloud Spanner/Group by partition/GroupByWindow
Aug 27, 2018 6:42:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T06:42:07.582Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as 
view/ParMultiDo(ToIsmRecordForMapLike) into SpannerIO.Write/Write mutations to 
Cloud Spanner/Keys sample as 
view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Aug 27, 2018 6:42:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T06:42:07.630Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)
Aug 27, 2018 6:42:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T06:42:07.673Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/Combine.GroupedValues/Extract
Aug 27, 2018 6:42:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T06:42:07.719Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Reify
Aug 27, 2018 6:42:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T06:42:07.764Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Extract into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/Combine.GroupedValues
Aug 27, 2018 6:42:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T06:42:07.815Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey/Reify into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/Combine.GroupedValues/Partial
Aug 27, 2018 6:42:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T06:42:07.849Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey+SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Partial into 
SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys
Aug 27, 2018 6:42:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T06:42:07.904Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify
Aug 27, 2018 6:42:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T06:42:07.948Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/ParMultiDo(ToIsmRecordForMapLike)
Aug 27, 2018 6:42:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T06:42:07.992Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/View.AsList/ParDo(ToIsmRecordForGlobalWindow) into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap
Aug 27, 2018 6:42:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T06:42:08.028Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/ParMultiDo(ToIsmRecordForMapLike)
Aug 27, 2018 6:42:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-27T06:42:08.072Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/T

[jira] [Work logged] (BEAM-3310) Push metrics to a backend in an runner agnostic way

2018-08-26 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-3310?focusedWorklogId=138303&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-138303
 ]

ASF GitHub Bot logged work on BEAM-3310:


Author: ASF GitHub Bot
Created on: 27/Aug/18 06:55
Start Date: 27/Aug/18 06:55
Worklog Time Spent: 10m 
  Work Description: JozoVilcek commented on issue #4548: [BEAM-3310] 
Metrics pusher
URL: https://github.com/apache/beam/pull/4548#issuecomment-416132575
 
 
   @zorro786 I definitely lack terminology so I apologise if I confuse apples 
and pears. I thought they are sort of equivalent, in a sense, that what is 
accumulators is also published as metrics. This is what I guessed from 
`FlinkMetricContainer` updates here
   
https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java#L86
   
   It they are not equivalent in values, that whole idea of possibility to use 
flink's `MetricReporter` is wrong.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 138303)
Time Spent: 17h 40m  (was: 17.5h)

> Push metrics to a backend in an runner agnostic way
> ---
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-extensions-metrics, sdk-java-core
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 17h 40m
>  Remaining Estimate: 0h
>
> The idea is to avoid relying on the runners to provide access to the metrics 
> (either at the end of the pipeline or while it runs) because they don't have 
> all the same capabilities towards metrics (e.g. spark runner configures sinks 
>  like csv, graphite or in memory sinks using the spark engine conf). The 
> target is to push the metrics in the common runner code so that no matter the 
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML: 
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)