[jira] [Assigned] (BEAM-1025) User guide - "How to create Beam IO Transforms"

2017-10-06 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-1025: -- Assignee: Reuven Lax (was: Stephen Sisk) > User guide - "How to create Beam IO Transforms" >

[jira] [Assigned] (BEAM-2352) Enable elasticsearch IO ITs

2017-10-06 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-2352: -- Assignee: Reuven Lax (was: Stephen Sisk) > Enable elasticsearch IO ITs >

[jira] [Assigned] (BEAM-1940) Add Capabilities info to "Built-in I/O Transforms" page

2017-10-06 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-1940: -- Assignee: Reuven Lax (was: Stephen Sisk) > Add Capabilities info to "Built-in I/O Transforms"

[jira] [Assigned] (BEAM-1598) Build support for Kubernetes and Beam IO Testing into PKB

2017-10-06 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-1598: -- Assignee: Jason Kuster (was: Stephen Sisk) > Build support for Kubernetes and Beam IO Testing

[jira] [Assigned] (BEAM-2385) IO authoring overview - improve steps for beginners

2017-10-06 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-2385: -- Assignee: Reuven Lax (was: Stephen Sisk) > IO authoring overview - improve steps for beginners

[jira] [Assigned] (BEAM-2404) BigQueryIO reading stalls if no data is returned by query

2017-10-06 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-2404: -- Assignee: Chamikara Jayalath (was: Stephen Sisk) > BigQueryIO reading stalls if no data is

[jira] [Assigned] (BEAM-2403) Provide BigQueryIO with method to indicate Priority (INTERACTIVE/BATCH)

2017-10-06 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-2403: -- Assignee: Chamikara Jayalath (was: Stephen Sisk) > Provide BigQueryIO with method to indicate

[jira] [Assigned] (BEAM-1643) IO ITs: add validation of row content

2017-10-06 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-1643: -- Assignee: Kenneth Knowles (was: Stephen Sisk) > IO ITs: add validation of row content >

[jira] [Assigned] (BEAM-1670) Pipeline I/O docs - add content on contributing I/O transforms

2017-10-06 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-1670: -- Assignee: Reuven Lax (was: Stephen Sisk) > Pipeline I/O docs - add content on contributing I/O

[jira] [Assigned] (BEAM-2141) beam_PerformanceTests_JDBC have not passed in weeks

2017-10-06 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-2141: -- Assignee: Jason Kuster (was: Stephen Sisk) > beam_PerformanceTests_JDBC have not passed in

[jira] [Assigned] (BEAM-2622) I/O Testing docs - Integration Testing section

2017-10-06 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-2622: -- Assignee: Reuven Lax (was: Stephen Sisk) > I/O Testing docs - Integration Testing section >

[jira] [Assigned] (BEAM-2660) Set PubsubIO batch size using builder

2017-10-06 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-2660: -- Assignee: Chamikara Jayalath (was: Stephen Sisk) > Set PubsubIO batch size using builder >

[jira] [Created] (BEAM-2659) JdbcIOIT flaky when run using io-it-suite-local

2017-07-22 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2659: -- Summary: JdbcIOIT flaky when run using io-it-suite-local Key: BEAM-2659 URL: https://issues.apache.org/jira/browse/BEAM-2659 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-2654) io-it-suite profile: support ability to use other runners

2017-07-21 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2654: -- Summary: io-it-suite profile: support ability to use other runners Key: BEAM-2654 URL: https://issues.apache.org/jira/browse/BEAM-2654 Project: Beam Issue Type:

[jira] [Created] (BEAM-2650) migrate beam_it_args -> beam_it_options

2017-07-20 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2650: -- Summary: migrate beam_it_args -> beam_it_options Key: BEAM-2650 URL: https://issues.apache.org/jira/browse/BEAM-2650 Project: Beam Issue Type: Task

[jira] [Created] (BEAM-2635) Add better documentation on testing Python I/O transforms

2017-07-18 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2635: -- Summary: Add better documentation on testing Python I/O transforms Key: BEAM-2635 URL: https://issues.apache.org/jira/browse/BEAM-2635 Project: Beam Issue Type:

[jira] [Created] (BEAM-2634) Add better documentation on testing unbounded I/O scenarios

2017-07-18 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2634: -- Summary: Add better documentation on testing unbounded I/O scenarios Key: BEAM-2634 URL: https://issues.apache.org/jira/browse/BEAM-2634 Project: Beam Issue

[jira] [Commented] (BEAM-1669) Pipeline I/O - add content on how to unit test

2017-07-14 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087688#comment-16087688 ] Stephen Sisk commented on BEAM-1669: Started a PR - https://github.com/apache/beam-site/pull/274 >

[jira] [Assigned] (BEAM-2458) Move HashingFn from test -> main to make it more accessible

2017-07-14 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-2458: -- Assignee: (was: Stephen Sisk) > Move HashingFn from test -> main to make it more accessible

[jira] [Commented] (BEAM-1799) IO ITs: simplify data loading design pattern

2017-07-14 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087685#comment-16087685 ] Stephen Sisk commented on BEAM-1799: I realized [~echauchot] and I resolved this discussion on the PR,

[jira] [Created] (BEAM-2622) I/O Testing docs - Integration Testing section

2017-07-14 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2622: -- Summary: I/O Testing docs - Integration Testing section Key: BEAM-2622 URL: https://issues.apache.org/jira/browse/BEAM-2622 Project: Beam Issue Type: Sub-task

[jira] [Updated] (BEAM-1669) Pipeline I/O - add content on how to unit test

2017-07-14 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk updated BEAM-1669: --- Summary: Pipeline I/O - add content on how to unit test (was: Pipeline I/O - add content on how to

[jira] [Assigned] (BEAM-2532) BigQueryIO source should avoid expensive JSON schema parsing for every record

2017-07-14 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-2532: -- Assignee: Chamikara Jayalath (was: Stephen Sisk) > BigQueryIO source should avoid expensive

[jira] [Commented] (BEAM-2341) Make our Python pom files virtualenv-compatible.

2017-07-05 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075610#comment-16075610 ] Stephen Sisk commented on BEAM-2341: for what it's worth - for the IO ITs, we expect users to be able

[jira] [Created] (BEAM-2554) failIfNoTests: causes issues when trying to run integration tests

2017-07-05 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2554: -- Summary: failIfNoTests: causes issues when trying to run integration tests Key: BEAM-2554 URL: https://issues.apache.org/jira/browse/BEAM-2554 Project: Beam

[jira] [Closed] (BEAM-2533) GCP IO: Use new bigtable client version

2017-06-30 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk closed BEAM-2533. -- Resolution: Fixed > GCP IO: Use new bigtable client version > --- >

[jira] [Updated] (BEAM-2545) bigtable e2e tests failing - UNKNOWN: Stale requests/Error mutating row

2017-06-30 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk updated BEAM-2545: --- Fix Version/s: (was: 2.1.0) > bigtable e2e tests failing - UNKNOWN: Stale requests/Error mutating

[jira] [Closed] (BEAM-2537) Can't run IO ITs - conflicting project vs projectId pipeline options

2017-06-30 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk closed BEAM-2537. -- Resolution: Fixed Fix Version/s: 2.1.0 PR submitted > Can't run IO ITs - conflicting project vs

[jira] [Commented] (BEAM-2545) bigtable e2e tests failing - UNKNOWN: Stale requests/Error mutating row

2017-06-30 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16070838#comment-16070838 ] Stephen Sisk commented on BEAM-2545: Given that the tests have stabilized, we have decided not to bump

[jira] [Updated] (BEAM-2545) bigtable e2e tests failing - UNKNOWN: Stale requests/Error mutating row

2017-06-29 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk updated BEAM-2545: --- Fix Version/s: 2.1.0 > bigtable e2e tests failing - UNKNOWN: Stale requests/Error mutating row >

[jira] [Commented] (BEAM-2545) bigtable e2e tests failing - UNKNOWN: Stale requests/Error mutating row

2017-06-29 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069203#comment-16069203 ] Stephen Sisk commented on BEAM-2545: cc [~sduskis] I realized this could potentially be related to the

[jira] [Created] (BEAM-2545) bigtable e2e tests failing - UNKNOWN: Stale requests/Error mutating row

2017-06-29 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2545: -- Summary: bigtable e2e tests failing - UNKNOWN: Stale requests/Error mutating row Key: BEAM-2545 URL: https://issues.apache.org/jira/browse/BEAM-2545 Project: Beam

[jira] [Updated] (BEAM-2537) Can't run IO ITs - conflicting project vs projectId pipeline options

2017-06-28 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk updated BEAM-2537: --- Description: Today, the GPC IO ITs use a mix of --project and --projectId pipeline options. eg.

[jira] [Assigned] (BEAM-2538) Spanner IO ITs failing

2017-06-28 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-2538: -- Assignee: Mairbek Khadikov (was: Guy Molinari) > Spanner IO ITs failing >

[jira] [Assigned] (BEAM-2538) Spanner IO ITs failing

2017-06-28 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-2538: -- Assignee: Guy Molinari (was: Davor Bonaci) > Spanner IO ITs failing > -- >

[jira] [Created] (BEAM-2538) Spanner IO ITs failing

2017-06-28 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2538: -- Summary: Spanner IO ITs failing Key: BEAM-2538 URL: https://issues.apache.org/jira/browse/BEAM-2538 Project: Beam Issue Type: Bug Components:

[jira] [Created] (BEAM-2537) Can't run IO ITs - conflicting project vs projectId pipeline options

2017-06-28 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2537: -- Summary: Can't run IO ITs - conflicting project vs projectId pipeline options Key: BEAM-2537 URL: https://issues.apache.org/jira/browse/BEAM-2537 Project: Beam

[jira] [Created] (BEAM-2533) GCP IO: Use new bigtable client version

2017-06-28 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2533: -- Summary: GCP IO: Use new bigtable client version Key: BEAM-2533 URL: https://issues.apache.org/jira/browse/BEAM-2533 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-2460) Make python maven variables available elsewhere

2017-06-20 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056682#comment-16056682 ] Stephen Sisk commented on BEAM-2460: You are correct, that does mean that a project that wants to use

[jira] [Commented] (BEAM-2460) Make python maven variables available elsewhere

2017-06-20 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056609#comment-16056609 ] Stephen Sisk commented on BEAM-2460: I wish that would help but it doesn't. The fundamental problem is

[jira] [Commented] (BEAM-2460) Make python maven variables available elsewhere

2017-06-20 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056558#comment-16056558 ] Stephen Sisk commented on BEAM-2460: With a bit more investigation, I've landed on the following

[jira] [Commented] (BEAM-2460) Make python maven variables available elsewhere

2017-06-20 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056401#comment-16056401 ] Stephen Sisk commented on BEAM-2460: I realized that #2 above implies that I'd have to add a variable

[jira] [Commented] (BEAM-2460) Make python maven variables available elsewhere

2017-06-20 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056396#comment-16056396 ] Stephen Sisk commented on BEAM-2460: okay, this is more complex than I had hoped.See

[jira] [Commented] (BEAM-2458) Move HashingFn from test -> main to make it more accessible

2017-06-20 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056361#comment-16056361 ] Stephen Sisk commented on BEAM-2458: [~tgroh] pointed out that as part of this, we'd be making

[jira] [Created] (BEAM-2460) Make python maven variables available elsewhere

2017-06-16 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2460: -- Summary: Make python maven variables available elsewhere Key: BEAM-2460 URL: https://issues.apache.org/jira/browse/BEAM-2460 Project: Beam Issue Type: Task

[jira] [Created] (BEAM-2458) Move HashingFn from test -> main to make it more accessible

2017-06-16 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2458: -- Summary: Move HashingFn from test -> main to make it more accessible Key: BEAM-2458 URL: https://issues.apache.org/jira/browse/BEAM-2458 Project: Beam Issue

[jira] [Updated] (BEAM-2457) Error: "Unable to find registrar for hdfs" - need to prevent/improve error message

2017-06-16 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk updated BEAM-2457: --- Description: I've noticed a number of user reports where jobs are failing with the error message

[jira] [Commented] (BEAM-2404) BigQueryIO reading stalls if no data is returned by query

2017-06-14 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049768#comment-16049768 ] Stephen Sisk commented on BEAM-2404: Hi Andre, I investigated this bug, and I can't seem to repro it.

[jira] [Commented] (BEAM-2439) Datastore writer can fail to progress if Datastore is slow

2017-06-13 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048080#comment-16048080 ] Stephen Sisk commented on BEAM-2439: thanks for letting us know. > Datastore writer can fail to

[jira] [Commented] (BEAM-1323) Add parallelism/splitting in JdbcIO

2017-06-12 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046824#comment-16046824 ] Stephen Sisk commented on BEAM-1323: can you elaborate on what exactly the SplittingFn would do? What

[jira] [Commented] (BEAM-1323) Add parallelism/splitting in JdbcIO

2017-06-12 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046775#comment-16046775 ] Stephen Sisk commented on BEAM-1323: +1 to eugene's comment. I don't believe we can reliably do this in

[jira] [Commented] (BEAM-2404) BigQueryIO reading stalls if no data is returned by query

2017-06-08 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043133#comment-16043133 ] Stephen Sisk commented on BEAM-2404: I wonder if we're not correctly sending the cleanup signal since

[jira] [Closed] (BEAM-2418) Datastore IO does not work out of the box

2017-06-08 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk closed BEAM-2418. -- > Datastore IO does not work out of the box > - > >

[jira] [Updated] (BEAM-1679) create gRPC IO

2017-06-06 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk updated BEAM-1679: --- Component/s: (was: sdk-java-core) sdk-java-extensions > create gRPC IO >

[jira] [Created] (BEAM-2418) Datastore IO does not work out of the box

2017-06-06 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2418: -- Summary: Datastore IO does not work out of the box Key: BEAM-2418 URL: https://issues.apache.org/jira/browse/BEAM-2418 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-2388) Built-In I/Os page - "discoverability"/"supported with adapters" section

2017-05-30 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2388: -- Summary: Built-In I/Os page - "discoverability"/"supported with adapters" section Key: BEAM-2388 URL: https://issues.apache.org/jira/browse/BEAM-2388 Project: Beam

[jira] [Created] (BEAM-2385) IO authoring overview - improve steps for beginners

2017-05-30 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2385: -- Summary: IO authoring overview - improve steps for beginners Key: BEAM-2385 URL: https://issues.apache.org/jira/browse/BEAM-2385 Project: Beam Issue Type:

[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner

2017-05-25 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024990#comment-16024990 ] Stephen Sisk commented on BEAM-1542: thanks! Much appreciated. > Need Source/Sink for Spanner >

[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner

2017-05-24 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023868#comment-16023868 ] Stephen Sisk commented on BEAM-1542: hi! Mind adding this to the built-in IO transforms page? It's a

[jira] [Commented] (BEAM-2352) Enable elasticsearch IO ITs

2017-05-24 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023585#comment-16023585 ] Stephen Sisk commented on BEAM-2352: I was ready to get these turned on, but we noticed an issue with

[jira] [Commented] (BEAM-2141) beam_PerformanceTests_JDBC have not passed in weeks

2017-05-24 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023572#comment-16023572 ] Stephen Sisk commented on BEAM-2141: I've been watching the behavior of kubernetes, and it changes the

[jira] [Commented] (BEAM-2347) Elasticsearch IO - test ES 5?

2017-05-24 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023281#comment-16023281 ] Stephen Sisk commented on BEAM-2347: thanks for clarifying! I converted this to a sub-task of that

[jira] [Updated] (BEAM-2347) Elasticsearch IO - test ES 5?

2017-05-24 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk updated BEAM-2347: --- Issue Type: Sub-task (was: Bug) Parent: BEAM-1637 > Elasticsearch IO - test ES 5? >

[jira] [Updated] (BEAM-2328) Introduce Apache Tika Input component

2017-05-24 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk updated BEAM-2328: --- Component/s: sdk-java-extensions > Introduce Apache Tika Input component >

[jira] [Closed] (BEAM-2342) k8s scripts - must not create publicly visible service by default

2017-05-23 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk closed BEAM-2342. -- Resolution: Fixed Fix Version/s: Not applicable > k8s scripts - must not create publicly visible

[jira] [Commented] (BEAM-2352) Enable elasticsearch IO ITs

2017-05-23 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021791#comment-16021791 ] Stephen Sisk commented on BEAM-2352: cc [~echauchot] > Enable elasticsearch IO ITs >

[jira] [Created] (BEAM-2352) Enable elasticsearch IO ITs

2017-05-23 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2352: -- Summary: Enable elasticsearch IO ITs Key: BEAM-2352 URL: https://issues.apache.org/jira/browse/BEAM-2352 Project: Beam Issue Type: Bug Components:

[jira] [Commented] (BEAM-2141) beam_PerformanceTests_JDBC have not passed in weeks

2017-05-23 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021773#comment-16021773 ] Stephen Sisk commented on BEAM-2141: The first problem I see is that the k8s cluster changed IP

[jira] [Assigned] (BEAM-2141) beam_PerformanceTests_JDBC have not passed in weeks

2017-05-23 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-2141: -- Assignee: Stephen Sisk > beam_PerformanceTests_JDBC have not passed in weeks >

[jira] [Commented] (BEAM-2141) beam_PerformanceTests_JDBC have not passed in weeks

2017-05-23 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021670#comment-16021670 ] Stephen Sisk commented on BEAM-2141: hi! I'm back in town, happy to investigate. Mind if I grab this

[jira] [Created] (BEAM-2347) Elasticsearch IO - test ES 5?

2017-05-23 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2347: -- Summary: Elasticsearch IO - test ES 5? Key: BEAM-2347 URL: https://issues.apache.org/jira/browse/BEAM-2347 Project: Beam Issue Type: Bug Components:

[jira] [Created] (BEAM-2342) k8s scripts - must not create publicly visible service by default

2017-05-22 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2342: -- Summary: k8s scripts - must not create publicly visible service by default Key: BEAM-2342 URL: https://issues.apache.org/jira/browse/BEAM-2342 Project: Beam

[jira] [Created] (BEAM-2082) I/O Authoring overview - emphasize reading the PTransform style guide

2017-04-26 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2082: -- Summary: I/O Authoring overview - emphasize reading the PTransform style guide Key: BEAM-2082 URL: https://issues.apache.org/jira/browse/BEAM-2082 Project: Beam

[jira] [Assigned] (BEAM-2081) I/O Authoring overview - better clarify how to read from files

2017-04-26 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-2081: -- Assignee: (was: Davor Bonaci) > I/O Authoring overview - better clarify how to read from

[jira] [Created] (BEAM-2081) I/O Authoring overview - better clarify how to read from files

2017-04-26 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2081: -- Summary: I/O Authoring overview - better clarify how to read from files Key: BEAM-2081 URL: https://issues.apache.org/jira/browse/BEAM-2081 Project: Beam Issue

[jira] [Commented] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration

2017-04-25 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983967#comment-15983967 ] Stephen Sisk commented on BEAM-2031: [~lcwik] also mentioned that it's definitely possible to

[jira] [Commented] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration

2017-04-25 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983958#comment-15983958 ] Stephen Sisk commented on BEAM-2031: yeah - in that doc, I think that "2. Construct FileSystemConfig

[jira] [Updated] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration

2017-04-25 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk updated BEAM-2031: --- Summary: Hadoop FileSystem needs to receive Hadoop Configuration (was: Hadoop FileSystem needs to

[jira] [Comment Edited] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration through PipelineOptions

2017-04-25 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983814#comment-15983814 ] Stephen Sisk edited comment on BEAM-2031 at 4/25/17 11:27 PM: -- where/how would

[jira] [Commented] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration through PipelineOptions

2017-04-25 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983814#comment-15983814 ] Stephen Sisk commented on BEAM-2031: where/how would we do that? There's no way right now for a user to

[jira] [Commented] (BEAM-2005) Add a Hadoop FileSystem implementation of Beam's FileSystem

2017-04-24 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982174#comment-15982174 ] Stephen Sisk commented on BEAM-2005: also - all the sub-tasks on this are assigned to me since we don't

[jira] [Created] (BEAM-2070) Implement match for HadoopFileSystem

2017-04-24 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2070: -- Summary: Implement match for HadoopFileSystem Key: BEAM-2070 URL: https://issues.apache.org/jira/browse/BEAM-2070 Project: Beam Issue Type: Sub-task

[jira] [Commented] (BEAM-2069) Remove ResourceId.getCurrentDirectory()?

2017-04-24 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982123#comment-15982123 ] Stephen Sisk commented on BEAM-2069: It may be worth considering moving directory over onto the

[jira] [Updated] (BEAM-2069) Remove ResourceId.getCurrentDirectory()?

2017-04-24 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk updated BEAM-2069: --- Summary: Remove ResourceId.getCurrentDirectory()? (was: Remove FileSystem.getCurrentDirectory()?) >

[jira] [Updated] (BEAM-2069) Remove FileSystem.getCurrentDirectory()?

2017-04-24 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk updated BEAM-2069: --- Affects Version/s: First stable release > Remove FileSystem.getCurrentDirectory()? >

[jira] [Commented] (BEAM-2069) Remove FileSystem.getCurrentDirectory()?

2017-04-24 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982105#comment-15982105 ] Stephen Sisk commented on BEAM-2069: marking FSR until we know whether or not we want to do this, since

[jira] [Commented] (BEAM-2005) Add a Hadoop FileSystem implementation of Beam's FileSystem

2017-04-24 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982100#comment-15982100 ] Stephen Sisk commented on BEAM-2005: Also, I filed https://issues.apache.org/jira/browse/BEAM-2069

[jira] [Created] (BEAM-2069) Remove FileSystem.getCurrentDirectory()?

2017-04-24 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2069: -- Summary: Remove FileSystem.getCurrentDirectory()? Key: BEAM-2069 URL: https://issues.apache.org/jira/browse/BEAM-2069 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-2005) Add a Hadoop FileSystem implementation of Beam's FileSystem

2017-04-24 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982030#comment-15982030 ] Stephen Sisk commented on BEAM-2005: note that adding support for other Hadoop FileSystem should be

[jira] [Commented] (BEAM-2005) Add a Hadoop FileSystem implementation of Beam's FileSystem

2017-04-24 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981966#comment-15981966 ] Stephen Sisk commented on BEAM-2005: Since I haven't heard anything and we are trying to finish soon

[jira] [Created] (BEAM-2066) Strip IOChannel validation calls from file based IOs

2017-04-24 Thread Stephen Sisk (JIRA)
Stephen Sisk created BEAM-2066: -- Summary: Strip IOChannel validation calls from file based IOs Key: BEAM-2066 URL: https://issues.apache.org/jira/browse/BEAM-2066 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-1940) Add Capabilities info to "Built-in I/O Transforms" page

2017-04-24 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981508#comment-15981508 ] Stephen Sisk commented on BEAM-1940: I think having a table where people can compare them is important

[jira] [Assigned] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration through PipelineOptions

2017-04-21 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-2031: -- Assignee: (was: Stephen Sisk) > Hadoop FileSystem needs to receive Hadoop Configuration

[jira] [Assigned] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration through PipelineOptions

2017-04-21 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-2031: -- Assignee: Stephen Sisk (was: Davor Bonaci) > Hadoop FileSystem needs to receive Hadoop

[jira] [Commented] (BEAM-2005) Add a Hadoop FileSystem implementation of Beam's FileSystem

2017-04-20 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977537#comment-15977537 ] Stephen Sisk commented on BEAM-2005: There are a couple possible ways we could adapt to this: *

[jira] [Updated] (BEAM-2033) Implement ResourceIds for HadoopFileSystem

2017-04-20 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk updated BEAM-2033: --- Description: There's an empty class for HadoopResourceId already - we will implement it. This blocks

[jira] [Resolved] (BEAM-1027) Hosting data stores to enable IO Transform testing

2017-04-20 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk resolved BEAM-1027. Resolution: Fixed Fix Version/s: First stable release I decided to go ahead and resolve this

[jira] [Assigned] (BEAM-1166) Source.getDefaultOutputCoder() - hard to implement for adapter style sources

2017-04-20 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Sisk reassigned BEAM-1166: -- Assignee: (was: Stephen Sisk) > Source.getDefaultOutputCoder() - hard to implement for

[jira] [Commented] (BEAM-2030) Implement beam FileSystem's copy()

2017-04-20 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977026#comment-15977026 ] Stephen Sisk commented on BEAM-2030: oh! I like that. That makes sense. Thanks luke! > Implement beam

[jira] [Commented] (BEAM-1166) Source.getDefaultOutputCoder() - hard to implement for adapter style sources

2017-04-20 Thread Stephen Sisk (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977023#comment-15977023 ] Stephen Sisk commented on BEAM-1166: I'm adapting this bug a bit since it's become clear that this is

  1   2   >