Re: [PROPOSAL] Prepare Beam 2.8.0 release

2018-10-03 Thread Ted Yu
+1 On Wed, Oct 3, 2018 at 9:52 AM Jean-Baptiste Onofré wrote: > +1 > > but we have to be fast in release process. 2.7.0 took more than 1 month > to be cut ! > > If no blocker, we have to just move forward. > > Regards > JB > > On 03/10/2018 18:25, Ahmet Altay wrote: > > Hi all, > > > > Release c

Re: [apachecon 2018] Universal metrics with apache beam

2018-10-03 Thread Ted Yu
Very interesting talk, Etienne. Looking forward to the audio recording. Cheers

Re: [DISCUSS] [BEAM-4126] Deleting Maven build files (pom.xml) grace period?

2018-06-06 Thread Ted Yu
+1 on this effort Original message From: Chamikara Jayalath Date: 6/6/18 2:09 PM (GMT-08:00) To: dev@beam.apache.org, u...@beam.apache.org Subject: Re: [DISCUSS] [BEAM-4126] Deleting Maven build files (pom.xml) grace period? +1 for the overall effort. As Pablo mentioned, we

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

2018-03-08 Thread Ted Yu
, once more runners catch up with the support: currently we have Dataflow > and Flink. +Chamikara Jayalath is currently > working on implementing it using SDFs in the Python SDK. > > On Thu, Mar 8, 2018 at 4:34 PM Ted Yu wrote: > >> Eugene: >> Very informative talk. >

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

2018-03-08 Thread Ted Yu
Eugene: Very informative talk. I looked at: sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/OffsetRangeTrackerTest.java Is there some example showing how OffsetRangeTracker works with Kafka partition(s) ? Thanks On Thu, Mar 8, 2018 at 3:58 PM, Eugene Kirpichov wrote:

Re: [VOTE] Release 2.3.0, release candidate #3

2018-02-12 Thread Ted Yu
bq. is OK with staging repo extension ? +1 on using the above approach. Cheers

Re: [DISCUSS] State of the project: Culture and governance

2018-02-04 Thread Ted Yu
bq. 1. Propose a pull request to their fix branch. This is my favorite and I've mentioned it. Everything is straightforward and explicit. The above makes sense. When the contributor merges the reviewer's pull request, it signifies their willingness to adopt the suggestion, making the combined pull

Re: Eclipse support

2018-01-17 Thread Ted Yu
Have you tried running 'mvn eclipse:eclipse' and importing from the root of workspace ? On Wed, Jan 17, 2018 at 10:32 AM, Ron Gonzalez wrote: > Hi, > I've been trying this for a couple of days now, but I can't seem to get > a clean Eclipse import. > I refreshed to latest master, got a clean

Re: Strata Conference this March 6-8

2018-01-16 Thread Ted Yu
+1 to BoF On Tue, Jan 16, 2018 at 5:00 PM, Dmitry Demeshchuk wrote: > Probably won't be attending the conference, but totally down for a BoF. > > On Tue, Jan 16, 2018 at 4:58 PM, Holden Karau > wrote: > >> Do interested folks have any timing constraints around a BoF? >> >> On Tue, Jan 16, 2018

Re: [DISCUSS] State of the project

2018-01-15 Thread Ted Yu
is is less of an issue for many other open-source > projects that do not make such a compatibility guarantee, as they are able > to simply remove or fix the API in the next version. > > I think we still need honest review of PRs, with the criteria being > stronger than "t

Re: [DISCUSS] State of the project

2018-01-14 Thread Ted Yu
bq. if a PR is basically right (it does what it should) without breaking the build, then it has to be merged fast +1 on above. This would give contributors positive feedback. On Sun, Jan 14, 2018 at 8:13 AM, Jean-Baptiste Onofré wrote: > Hi Davor, > > Thanks a lot for this e-mail. > > I would l

Re: Dataflow runner examples build fail

2018-01-08 Thread Ted Yu
+1 Original message From: Jean-Baptiste Onofré Date: 1/8/18 1:26 AM (GMT-08:00) To: dev@beam.apache.org Subject: Dataflow runner examples build fail Hi guys, The PRs and nightly builds are failing due to an issue with the dataflow platform: it seems we have a disk quota exc

Re: [VOTE] Choose the "new" Spark runner

2017-11-17 Thread Ted Yu
[ ] Use Spark 1 & Spark 2 Support Branch [X] Use Spark 2 Only Branch On Thu, Nov 16, 2017 at 5:08 AM, Jean-Baptiste Onofré wrote: > Hi guys, > > To illustrate the current discussion about Spark versions support, you can > take a look on: > > -- > Spark 1 & Spark 2 Support Branch > > http

Re: [VOTE] Release 2.2.0, release candidate #3

2017-11-10 Thread Ted Yu
Considering that the holiday is around the corner, it would be nice to release 2.2.0 sooner.  Cheers  Original message From: Chamikara Jayalath Date: 11/10/17 12:22 PM (GMT-08:00) To: dev@beam.apache.org Subject: Re: [VOTE] Release 2.2.0, release candidate #3 We found anothe

Re: [VOTE] Drop Spark 1.x support to focus on Spark 2.x

2017-11-08 Thread Ted Yu
Having both Spark1 and Spark2 modules would benefit wider user base. I would vote for that. Cheers On Wed, Nov 8, 2017 at 12:51 AM, Jean-Baptiste Onofré wrote: > Hi Robert, > > Thanks for your feedback ! > > From an user perspective, with the current state of the PR, the same > pipelines can r

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-07 Thread Ted Yu
>> > Gradle. My point is also to keep Maven artifacts > > > >and > > > >> >>> > > >"releases" as > > > >> >>> > > >> > most > >

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-07 Thread Ted Yu
gt; > > >> > >> > >Hi! > >> >>> > > >> > >> > > > >> >>> > > >> > >> > >Many of these points sound valid, but AFAICT Maven > >> >doesn't > >> >>> > > >really > >> >>> > &g

Re: [Proposal] Apache Beam Swag Store

2017-11-05 Thread Ted Yu
+1 Original message From: Jacob Marble Date: 11/5/17 1:50 PM (GMT-08:00) To: dev@beam.apache.org Subject: Re: [Proposal] Apache Beam Swag Store I think this is a great idea, ready to order mine. :) Jacob On Sat, Oct 28, 2017 at 11:19 AM, Jean-Baptiste Onofré wrote: > It s

Re: [VOTE] Switch to new JIRA workflow for pending proposals

2017-11-01 Thread Ted Yu
+1 Original message From: Henning Rohde Date: 11/1/17 9:29 AM (GMT-08:00) To: dev@beam.apache.org Subject: Re: [VOTE] Switch to new JIRA workflow for pending proposals + 1. Also +1 to Kenn's suggestion of adding a "triage" stage prior to "open" for other bugs and features.

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Ted Yu
I agree with Ben's comment. Recently I have been using gradle in another Apache project and found it interesting. Cheers

Re: Policy for stale PRs

2017-08-18 Thread Ted Yu
comments > for 60 days. The community will close stale PRs. Author is welcome to > reopen the same PR again in the future. The associated JIRAs will be > unassigned from the author but will stay open. > > On Wed, Aug 16, 2017 at 3:25 PM, Ted Yu wrote: > > > bq. IRAs should

Re: Policy for stale PRs

2017-08-16 Thread Ted Yu
or will be > out > >> of > >> > communication. We can push a commit with a "This closes #xyz #abc" > >> message. > >> > Is there another way to do this? > >> > > >> > Ahmet > >> > > >&

Re: Policy for stale PRs

2017-08-16 Thread Ted Yu
On Wed, Aug 16, 2017 at 12:20 AM Jean-Baptiste Onofré > wrote: > > > If we consider the author, it makes sense. > > > > Regards > > JB > > > > On Aug 15, 2017, 01:29, at 01:29, Ted Yu wrote: > > >The proposal makes sense. > > > > >

Re: Policy for stale PRs

2017-08-14 Thread Ted Yu
The proposal makes sense. If the author of PR doesn't respond for 90 days, the PR is likely out of sync with current repo. Cheers On Mon, Aug 14, 2017 at 5:27 PM, Ahmet Altay wrote: > Hi all, > > Do we have an existing policy for handling stale PRs? If not could we come > up with one. We are g

Re: [ANNOUNCEMENT] New PMC members, August 2017 edition!

2017-08-11 Thread Ted Yu
Congratulations to Ahmet and Aviem. On Fri, Aug 11, 2017 at 10:40 AM, Davor Bonaci wrote: > Please join me and the rest of Beam PMC in welcoming the following > committers as our newest PMC members. They have significantly contributed > to the project in different ways, and we look forward to ma

Re: [ANNOUNCEMENT] New committers, August 2017 edition!

2017-08-11 Thread Ted Yu
Congratulations to all. On Fri, Aug 11, 2017 at 10:40 AM, Davor Bonaci wrote: > Please join me and the rest of Beam PMC in welcoming the following > contributors as our newest committers. They have significantly contributed > to the project in different ways, and we look forward to many more >

Re: beam-site issues with Jenkins and MergeBot

2017-08-09 Thread Ted Yu
However, the following is accessible: https://github.com/apache/beam-site.git Last commit was 13 days ago. On Wed, Aug 9, 2017 at 1:12 PM, Ted Yu wrote: > For #1, under https://git-wip-us.apache.org/repos/asf , I don't see > beam-site > > FYI > > On Wed, Aug 9,

Re: beam-site issues with Jenkins and MergeBot

2017-08-09 Thread Ted Yu
For #1, under https://git-wip-us.apache.org/repos/asf , I don't see beam-site FYI On Wed, Aug 9, 2017 at 1:08 PM, Eugene Kirpichov < kirpic...@google.com.invalid> wrote: > Hello, > > I've been trying to merge a PR https://github.com/apache/ > beam-site/pull/278 > and ran into the following issue

Re: is it ok to have a dicussion without subscribe the list

2017-08-09 Thread Ted Yu
Derek: You can periodically visit: http://search-hadoop.com/Beam where it is easy to find the thread(s) you're interested in. The latency of indexing is very low. FYI On Wed, Aug 9, 2017 at 1:28 AM, derek wrote: > On Tue, Aug 8, 2017 at 1:23 PM, Jason Kuster > wrote: > > Hi Derek, > > > > I

Re: Java 8?

2017-07-14 Thread Ted Yu
Recently on Spark and Flink mailing lists there have been discussion on dropping support for Java 7. For master branch, we should consider moving to Java 8 for compilation. On Fri, Jul 14, 2017 at 2:41 PM, Swapnil Bawaskar wrote: > Hi, > > I am trying to write a write an I/O transform for Apac

Re: BEAM-933 - Not reproduceable

2017-07-10 Thread Ted Yu
Looks like (some of) the findbugs warnings were not addressed. e.g. in TopWikipediaSessions.java : if (Math.abs(c.element().hashCode()) <= Integer.MAX_VALUE * samplingThreshold) { See https://stackoverflow.com/questions/23416264/bad-attempt-to-compute-absolute-value for why fin

Re: MergeBot is here!

2017-07-07 Thread Ted Yu
For https://gitbox.apache.org/setup/ , after completing the first two steps, is there any action needed for "MFA Status" box ? Cheers On Fri, Jul 7, 2017 at 1:37 PM, Lukasz Cwik wrote: > for i in range(0, inf): +1 > > Note that the URL for gitbox linking is: > https://gitbox.apache.org/setup/ (

Re: writing to s3 in beam

2017-07-05 Thread Ted Yu
Please take a look at BEAM-2500 (and related JIRAs). Cheers On Wed, Jul 5, 2017 at 8:00 PM, Jyotirmoy Sundi wrote: > Hi Folks, > > I am trying to write to s3 from beam. > > These are configs I am passing > > --hdfsConfiguration='[{"fs.default.name": "s3://xxx-output", > "fs.s3.awsAccessKey

Re: Build Failure in * release-2.0.0

2017-07-05 Thread Ted Yu
bq. Caused by: java.net.SocketException: Too many open files Please adjust ulimit. FYI On Wed, Jul 5, 2017 at 1:33 PM, Jyotirmoy Sundi wrote: > Hi Folks , > > Any idea why the build is failing in release-2.0.0 , i did "mvn clean > package" > > > *Trace* > > [INFO] Running org.apache.beam.sdk.i

Re: Build failed in Jenkins: beam_PerformanceTests_Python #47

2017-07-01 Thread Ted Yu
AddRunnerProfileMvnArgument is called. On Sat, Jul 1, 2017 at 9:19 AM, Ted Yu wrote: > > File "<https://builds.apache.org/job/beam_PerformanceTests_ > > Python/ws/PerfKitBenchmarker/perfkitbenchmarker/beam_benchma > rk_helper.py",> > > line 136, in I

Re: Build failed in Jenkins: beam_PerformanceTests_Python #47

2017-07-01 Thread Ted Yu
> File " Python/ws/PerfKitBenchmarker/perfkitbenchmarker/beam_ benchmark_helper.py",> > line 136, in InitializeBeamRepo > mvn_command) > TypeError: AddRunnerProfileMvnArgument() takes exactly 3 arguments (2 > given) Looks like 3rd parame

Re: Java Cross-JDK Test Available on Jenkins Postcommit!

2017-06-09 Thread Ted Yu
This is nice. Nit: I don't see JDK 1.8 (Oracle) variant. Is this intentional ? Thanks On Fri, Jun 9, 2017 at 3:49 PM, Mark Liu wrote: > Hi all, > > I worked on BEAM-1544 > for > a while, which is to bring Java cross-JDK test to Jenkins, and fin

Re: Precommit Jenkins Linkage Broken

2017-05-30 Thread Ted Yu
INFRA-14247 is currently marked Major. Suggest raising the priority so that it gets more attention. Cheers On Tue, May 30, 2017 at 2:59 PM, Jason Kuster < jasonkus...@google.com.invalid> wrote: > Hey folks, > > Just wanted to mention on the dev list that Jenkins precommit breakage is a > known

Re: How can I disable running Python SDK tests when testing my Java change?

2017-05-19 Thread Ted Yu
think this kind of information would be very useful to speed up new > developers. > To figure this out one has to go through pom files. > > Cheers > > On Fri 19 May 2017 at 18:41, Ted Yu wrote: > > > Should this tip be added to https://beam.apache.org/contribute/testi

Re: How can I disable running Python SDK tests when testing my Java change?

2017-05-19 Thread Ted Yu
Should this tip be added to https://beam.apache.org/contribute/testing/ ? Cheers On Fri, May 19, 2017 at 10:31 AM, Lukasz Cwik wrote: > Developers can use *-pl \!sdks/python* to skip the python module. > > Breaking it up would help developers working outside of Python and would > decrease the p

Re: First stable release completed!

2017-05-17 Thread Ted Yu
This is great news. Congratulations. On Wed, May 17, 2017 at 5:22 AM, James wrote: > Congratulations to all! > On Wed, 17 May 2017 at 8:03 PM Prabeesh K. wrote: > > > awesome. 📢 > > > > On 17 May 2017 at 15:28, Davor Bonaci wrote: > > > > > The first stable release is now complete! > > > > >

Re: [PROPOSAL] design of DSL SQL interface

2017-05-13 Thread Ted Yu
Can you fill out the Transition Plan ? Thanks On Fri, May 12, 2017 at 10:49 PM, Mingmin Xu wrote: > Hi all, > > As you may know, we're working on BeamSQL to execute SQL queries as a Beam > pipeline. This is a valuable feature, not only shipped as a packaged CLI, > but also as part of the SDK to

Re: First stable release: version designation?

2017-05-04 Thread Ted Yu
hanges can occur. > > I'd kindly ask to use email for this one, as usual, to keep a record (in > this specific case). > > On Thu, May 4, 2017 at 3:39 PM, Ted Yu wrote: > > > What's the difference between first and second, third and fourth columns > ? > &g

Re: First stable release: version designation?

2017-05-04 Thread Ted Yu
fUkFig3hG53lOYl73tdk > Wqz5B6eQ40TEgk/edit?usp=sharing> > . > I have already added all the votes that Davor compiled 3 hours ago and the > responses afterwards. > > On Thu, May 4, 2017 at 12:49 PM, Ted Yu wrote: > > > Maybe create a google doc with columns as the camps. >

Re: First stable release: version designation?

2017-05-04 Thread Ted Yu
ny additional opinions? > > > > > > > > > > Thanks! > > > > > > > > > > Davor > > > > > > > > > > On Wed, Mar 8, 2017 at 12:58 PM, Amit Sela > > > wrote: > > > > >

Re: Congratulations Davor!

2017-05-04 Thread Ted Yu
Congratulations, Davor! On Thu, May 4, 2017 at 12:45 AM, Aviem Zur wrote: > Congrats Davor! :) > > On Thu, May 4, 2017 at 10:42 AM Jean-Baptiste Onofré > wrote: > > > Congrats ! Well deserved ;) > > > > Regards > > JB > > > > On 05/04/2017 09:30 AM, Jason Kuster wrote: > > > Hi all, > > > > > >

Re: Beam spark 2.x runner status

2017-05-02 Thread Ted Yu
it Sela wrote: > >> @Ted definitely makes sense. >> @JB I'm merging https://github.com/apache/beam/pull/2354 soon so any >> deprecated Spark API issues should be resolved. >> >> On Wed, Mar 29, 2017 at 2:46 PM Ted Yu wrote: >> >> This is what I did over

Re: git commit message

2017-04-30 Thread Ted Yu
, almost all changes go via PR and most of with a Jira. > > > > So, if maybe you found couple of, again, it's rare and we follow the > > Jira/PR/review/commit path. > > > > Regards > > JB > > > > On 04/30/2017 04:59 PM, Ted Yu wrote: > > >

git commit message

2017-04-30 Thread Ted Yu
Hi, When I went over git commit messages, I found some without either of the following: JIRA number PR number It would be nice for other people to get background / goal / discussion w.r.t. any commit if either (or both) of the above is present. My two cents.

Re: Status of our CI tools

2017-04-28 Thread Ted Yu
+1 On Fri, Apr 28, 2017 at 6:24 PM, Thomas Groh wrote: > +1! This will be really helpful when looking at my PRs; I basically get no > signal from the current state of the github UI, and this will restore that > to giving me very strong positive signal. > > On Fri, Apr 28, 2017 at 6:22 PM, Davor

Re: Community hackathon

2017-04-24 Thread Ted Yu
+1 > On Apr 24, 2017, at 12:51 AM, Jean-Baptiste Onofré wrote: > > That's a wonderful idea ! > > I think the easiest way to organize this event is using the Slack channels to > discuss, help each other, and sync together. > > Regards > JB > >> On 04/24/2017 09:48 AM, Davor Bonaci wrote: >> W

Re: Hanging Jenkins builds.

2017-04-22 Thread Ted Yu
Looks like this might be the cause for the failed build ( https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/9927/console ): /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java:145:

Re: [DISCUSSION] Encouraging more contributions

2017-04-22 Thread Ted Yu
+1 On Sat, Apr 22, 2017 at 7:31 AM, Aviem Zur wrote: > Hi all, > > I wanted to start a discussion about actions we can take to encourage more > contributions to the project. > > A few points I've been thinking of: > > 1. Have people unassign themselves from issues they're not actively working >

Re: Build failed in Jenkins: beam_SeedJob #214

2017-04-18 Thread Ted Yu
ring too but I'm giving the evaluate thing a try[1][2] (once it > starts running -- executors are full currently). > > [1] https://builds.apache.org/view/Beam/job/beam_SeedJob/215/ > [2] https://github.com/apache/beam/pull/2578 > > On Tue, Apr 18, 2017 at 10:12 AM, Ted

Re: Build failed in Jenkins: beam_SeedJob #214

2017-04-18 Thread Ted Yu
: > Not so simple, unfortunately [1]. Ideas welcome ;-) > > Davor > > [1] > https://github.com/jenkinsci/job-dsl-plugin/wiki/Migration# > migrating-to-160 > > On Tue, Apr 18, 2017 at 9:57 AM, Ted Yu wrote: > > > I wonder if we should adopt the suggestion

Re: Build failed in Jenkins: beam_SeedJob #214

2017-04-18 Thread Ted Yu
I wonder if we should adopt the suggestion here (involving evaluate): http://stackoverflow.com/questions/9136328/including-a-groovy-script-in-another-groovy On Tue, Apr 18, 2017 at 9:45 AM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See

Re: Pipeline termination in the unified Beam model

2017-04-18 Thread Ted Yu
Why is the timeout needed for Spark ? Thanks > On Apr 18, 2017, at 3:05 AM, Etienne Chauchot wrote: > > +1 on "runners really terminate in a timely manner to easily programmatically > orchestrate Beam pipelines in a portable way, you do need to know whether > the pipeline will finish without t

Re: Renaming SideOutput

2017-04-11 Thread Ted Yu
+1 > On Apr 11, 2017, at 5:34 PM, Thomas Groh wrote: > > I think that's a good idea. I would call the outputs of a ParDo the "Main > Output" and "Additional Outputs" - it seems like an easy way to make it > clear that there's one output that is always expected, and there may be > more. > > On T

Re: Build failing on the Apex runner

2017-04-09 Thread Ted Yu
I used maven 3.3.9 with Java 8 on: 3.10.0-327.28.2.el7.x86_64 #1 SMP Wed Aug 3 11:11:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux The test passed. FYI On Sat, Apr 8, 2017 at 11:55 PM, Thomas Weise wrote: > I'm not sure what the issue is, but that commit shouldn't affect the test > listed and th

Re: IO ITs: Hosting Docker images

2017-04-08 Thread Ted Yu
+1 > On Apr 7, 2017, at 10:46 PM, Jean-Baptiste Onofré wrote: > > Hi Stephen, > > I think we should go to 1 and 4: > > 1. Try to use existing images providing what we need. If we don't find > existing image, we can always ask and help other community to provide so. > 4. If we don't find a sui

Re: [PROPOSAL]: a new feature branch for SQL DSL

2017-04-05 Thread Ted Yu
1.11 so far, unless impacted by API change. Before it's merged > back to master, will upgrade to the latest version. > > On Wed, Apr 5, 2017 at 2:14 PM, Ted Yu wrote: > > > Working in feature branch is good - you may want to periodically sync up > > with master. > &

Re: [PROPOSAL]: a new feature branch for SQL DSL

2017-04-05 Thread Ted Yu
Working in feature branch is good - you may want to periodically sync up with master. I noticed that you are using 1.11.0 of calcite. 1.12 is out, FYI On Wed, Apr 5, 2017 at 2:05 PM, Mingmin Xu wrote: > Hi all, > > I'm working on https://issues.apache.org/jira/browse/BEAM-301(Add a Beam > SQL D

Re: Travis-CI Build failures

2017-04-05 Thread Ted Yu
Please see BEAM-1809 On Wed, Apr 5, 2017 at 1:01 AM, Amit Sela wrote: > Unfortunately Travis is not stable enough right now. I don't think this > means that there is an issue with your work on the PR, you should notice > Jenkins after PRing to see if all tests pass and the reviewing committer >

Re: [DISCUSSION] Consistent use of loggers

2017-04-03 Thread Ted Yu
+1 > On Apr 3, 2017, at 11:48 AM, Aviem Zur wrote: > > Upon further inspection there seems to be an issue we may have overlooked: > In cluster mode, some of the runners will have dependencies added directly > to the classpath by the cluster, and since SLF4J can only work with one > binding, the

Re: Want to contribute to Beam project

2017-04-01 Thread Ted Yu
Have you looked at https://beam.apache.org/contribute/ ? You can also check out the latest git commits to see the components which receive active attention. Disclaimer: I am new to Beam as well. Cheers On Sat, Apr 1, 2017 at 12:59 PM, tarush grover wrote: > Hi Members, > > Let me introduce my

Re: [PROPOSAL] ORC support

2017-04-01 Thread Ted Yu
+1 > On Apr 1, 2017, at 8:31 AM, Tibor Kiss wrote: > > Hello, > > Recently the Optimized Row Columnar (ORC) file format was spin off from Hive > and became a top level Apache Project: https://orc.apache.org/ > > It is similar to Parquet in a sense that it uses column major format but > ORC has

Re: Beam spark 2.x runner status

2017-03-29 Thread Ted Yu
gt; > > > > for a spark 2 runner based on RDD because the maintenance burden > > > > > would be even worse. We would have to fight not only with the > double > > > > > merge of fixes (in case the profile idea does not work), but also > > with &g

Re: IO IT Patterns: Simplifying data loading

2017-03-23 Thread Ted Yu
Looks like you forgot to include JIRA number: BEAM-1799 Cheers On Thu, Mar 23, 2017 at 4:26 PM, Stephen Sisk wrote: > hi! > > I just opened a jira ticket that I wanted to make sure the mailing list got > a chance to see. > > The problem is that the current design pattern for doing data loading

Re: [DISCUSSION] Consistent use of loggers

2017-03-22 Thread Ted Yu
+1 to what JB said. > On Mar 21, 2017, at 9:46 PM, Aviem Zur wrote: > > +1 to what JB said. > > Will just have to be documented well as if we provide no binding there will > be no logging out of the box unless the user adds a binding. > > On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofré

Re: Beam spark 2.x runner status

2017-03-22 Thread Ted Yu
t concepts such > as using SparkSession in Spark 2 instead of SparkContext/StreamingContext > in Spark 1. > > On Tue, Mar 21, 2017 at 7:25 PM Ted Yu wrote: > > > I have done some work over in HBASE-16179 where compatibility modules are > > created to isolate changes in Spark 2.x API

Re: Beam spark 2.x runner status

2017-03-21 Thread Ted Yu
I have done some work over in HBASE-16179 where compatibility modules are created to isolate changes in Spark 2.x API so that code in hbase-spark module can be reused. FYI

Re: why Source#validate() is not declared to throw any exception

2017-03-21 Thread Ted Yu
20, 2017, 11:21 PM Jean-Baptiste Onofré > wrote: > > > Hi Ted, > > > > validate() is supposed to throw runtime exception (IllegalStateException, > > RuntimeException, ...) to "traverse" the executor. > > > > Regards > > JB > > > >

why Source#validate() is not declared to throw any exception

2017-03-20 Thread Ted Yu
Hi, I was reading HDFSFileSource.java where: @Override public void validate() { ... } catch (IOException | InterruptedException e) { throw new RuntimeException(e); } Why is validate() not declared to throw any exception ? If validation doesn't pass, there is nothing to cle

Re: Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Spark #1282

2017-03-19 Thread Ted Yu
Looks like Jenkins just came back (to life). Hopefully the next precommit passes. On Sun, Mar 19, 2017 at 11:39 AM, Amit Sela wrote: > I think its a Jenkins issue. Jenkins is shutting down. I'll follow and > relate if this keeps happening. > > -- Forwarded message - > From: Apac

Re: [ANNOUNCEMENT] New committers, March 2017 edition!

2017-03-17 Thread Ted Yu
Congratulations! On Fri, Mar 17, 2017 at 2:13 PM, Davor Bonaci wrote: > Please join me and the rest of Beam PMC in welcoming the following > contributors as our newest committers. They have significantly contributed > to the project in different ways, and we look forward to many more > contribut

Re: Apache Beam (virtual) contributor meeting @ Tue Mar 7, 2017

2017-03-16 Thread Ted Yu
For ApacheCon, you can find the schedule here: https://apachebigdata2017.sched.com/ FYI On Thu, Mar 16, 2017 at 1:47 PM, Davor Bonaci wrote: > I'd like to thank everyone for coming -- notes and summary of the > discussion are below. > > If there's any feedback, ideas for improvement, requests

Re: [VOTE] Release 0.6.0, release candidate #2

2017-03-13 Thread Ted Yu
bq. I would prefer that we have a .tar.gz release +1 On Mon, Mar 13, 2017 at 4:21 PM, Ismaël Mejía wrote: > ​+1 (non-binding) > > - verified signatures + checksums > - run mvn clean install -Prelease, all artifacts build and the tests run > smoothly (modulo some local issues I had with the ins

Re: [VOTE] Release 0.6.0, release candidate #2

2017-03-13 Thread Ted Yu
bq. That name makes sense to me +1 Maybe change the subject of this thread and send to dev@ to raise awareness ? On Mon, Mar 13, 2017 at 11:10 AM, Robert Bradshaw < rober...@google.com.invalid> wrote: > On Sat, Mar 11, 2017 at 11:19 PM, Ahmet Altay > wrote: > > > On Sat, Mar 11, 2017 at 11:48

Re: [VOTE] Release 0.6.0, release candidate #2

2017-03-12 Thread Ted Yu
I was able to run "mvn clean install -Prelease" command successfully, too. On Sun, Mar 12, 2017 at 12:02 AM, Ahmet Altay wrote: > Amit, > > I was able to successfully build in a clean environment with the following > commands: > > git checkout tags/v0.6.0-RC2 -b RC2 > mvn clean install -Prelease

Re: [VOTE] Release 0.6.0, release candidate #2

2017-03-11 Thread Ted Yu
+1 Checked signature. Ran test suite which passed. On Fri, Mar 10, 2017 at 9:05 PM, Ahmet Altay wrote: > Hi everyone, > > Please review and vote on the release candidate #2 for the version 0.6.0, > as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide s

Re: [VOTE] Release 0.6.0, release candidate #1

2017-03-09 Thread Ted Yu
bq. ran into a known issue [14] Currently BEAM-1674 is marked blocker. Would it be pushed to next release ? Cheers On Thu, Mar 9, 2017 at 4:07 PM, Ahmet Altay wrote: > Hi everyone, > > Please review and vote on the release candidate #1 for the version 0.6.0, > as follows: > [ ] +1, Approve the

Re: [VOTE] Release 0.6.0, release candidate #1

2017-03-09 Thread Ted Yu
+1 Checked signature Ran test suite - all passed. On Thu, Mar 9, 2017 at 4:07 PM, Ahmet Altay wrote: > Hi everyone, > > Please review and vote on the release candidate #1 for the version 0.6.0, > as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide spe

Re: First stable release: version designation?

2017-03-06 Thread Ted Yu
If we end up with version 2.0, more effort (trying out more use scenarios e.g.) should go into release process to make sure what is released is indeed stable. Normally people would have higher expectation on 2.0 release compared to 1.0 release. On Mon, Mar 6, 2017 at 6:34 PM, Davor Bonaci wrote:

Re: First stable release: version designation?

2017-03-01 Thread Ted Yu
ly also new ones -- this will make it unambiguous whether referring to > pre-Beam or Beam releases. > > I understand the 1.0 motivation -- it's cleaner in isolation -- but I think > it would lead to long-term confusion in the user community. > > On Wed, Mar 1, 2017 at 1:1

Re: First stable release: version designation?

2017-03-01 Thread Ted Yu
+1 to what Jesse and Amit said. On Wed, Mar 1, 2017 at 12:32 PM, Amit Sela wrote: > I think 1.0.0 for a couple of reasons: > > * It makes sense coming after 0.X (+1 Jesse). > * It is the FIRST stable release as a project, regardless of its roots. > * while the SDK is definitely a 2.0.0, Beam is