Hey all, Cool. I'll revert samza-sql commits, and cut a branch, then re-apply SAMZA-482 and SAMZA-484.
Cheers, Chris On Thu, Feb 12, 2015 at 12:06 PM, Milinda Pathirage <mpath...@umail.iu.edu> wrote: > Thanks Julian. > > @Chris, I'll update the patch with Calcite version change. > > On Thu, Feb 12, 2015 at 1:56 PM, Julian Hyde <julianh...@gmail.com> wrote: > > > +1 > > > > I've pushed a snapshot, versioned "1.1-chi-incubating-SNAPSHOT", and > based > > on > > > https://github.com/julianhyde/incubator-calcite/commit/b8a91c31859b6901976de8dcad43afd4877cad36 > > . > > > > See > > > https://repository.apache.org/content/repositories/snapshots/org/apache/calcite/calcite/ > > > > > > > On Feb 12, 2015, at 9:26 AM, Jakob Homan <jgho...@gmail.com> wrote: > > > > > > +1, assuming we're keeping RTC on the branch (, Jakob said > pedantically). > > > > > > On 12 February 2015 at 08:25, Milinda Pathirage <mpath...@umail.iu.edu > > > > wrote: > > >> I am +1 for this approach. > > >> > > >> Milinda > > >> > > >> On Thu, Feb 12, 2015 at 11:22 AM, Chris Riccomini < > > criccom...@apache.org> > > >> wrote: > > >> > > >>> Hey Julian, > > >>> > > >>>> The rule of thumb I’ve learned integrating Calcite into Hive is that > > only > > >>> a branch should depend on snapshot versions of another component > > >>> > > >>> This is very practical. I hadn't considered the Calcite dependency > > issue > > >>> when I pushed for merging into master rather than a branch. > > >>> > > >>> Based on this, it seems like the best thing to do is: > > >>> > > >>> 1) Work off of Calcite SNAPSHOT Maven publication. > > >>> 2) Create a samza-sql branch. > > >>> 3) Migrate all existing samza-sql commits from master into the > > samza-sql > > >>> branch. > > >>> > > >>> Sorry for the churn on this all. I hadn't considered the binary > > dependency > > >>> issue. Is everyone OK with this? > > >>> > > >>> Cheers, > > >>> Chris > > >>> > > >>> On Wed, Feb 11, 2015 at 10:51 AM, Julian Hyde <jul...@hydromatic.net > > > > >>> wrote: > > >>> > > >>>> This seems more like a branch than a classifier. Calcite is > > developing in > > >>>> a branch, and would produce snapshots from that branch. The rule of > > thumb > > >>>> I’ve learned integrating Calcite into Hive is that only a branch > > should > > >>>> depend on snapshot versions of another component. (Hive broke this > > rule > > >>> and > > >>>> got into a sticky situation where their trunk depended on a Calcite > > >>>> snapshot and they couldn’t release because Calcite was stuck in > > >>> pre-release > > >>>> vote limbo.) > > >>>> > > >>>> I’m going to try to produce maven artifacts with {groupId: > > >>>> “org.apache.calcite”, version: "1.1.0-stream-incubating-SNAPSHOT”, > > >>>> artifactId: “calcite-core”, “calcite-avatica”, etc.} and push them > to > > >>>> Apache nexus. I do recommend that you use it on a branch. > > >>>> > > >>>> However, I don’t think there would be a problem getting the > > functionality > > >>>> into calcite’s master branch in time for Calcite’s next release, in > > >>> about 4 > > >>>> weeks. (We commit to releasing approximately monthly.) The > > functionality > > >>>> and APIs would be marked “experimental, subject to change” but at > > least > > >>> you > > >>>> would be able to merge your work to master at that point. Doubtless > > you > > >>>> would need new features & bug fixes in Calcite after that, and so > work > > >>> that > > >>>> depended on that would have to stay in a branch until Calcite was > > >>> released. > > >>>> > > >>>> Julian > > >>>> > > >>>> On Feb 11, 2015, at 10:34 AM, Felix GV > <fville...@linkedin.com.INVALID > > > > > >>>> wrote: > > >>>> > > >>>>> I guess one concern with Calcite SNAPSHOT releases is that they > imply > > >>>> they'll be included in the next release, which may not be the case > if > > >>>> they're still considered very experimental. > > >>>>> > > >>>>> Alternatively, perhaps Julian can create a new version name with a > > >>>> classifier for streaming. > > >>>>> > > >>>>> > > >>>> > > >>> > > > http://maven.apache.org/plugins/maven-deploy-plugin/examples/deploying-with-classifiers.html > > >>>>> > > >>>>> Then he could release SNAPSHOT artifacts for both the main/master > > code > > >>>> base as well as the streaming branch, no matter how experimental it > > is... > > >>>>> > > >>>>> -- > > >>>>> > > >>>>> Felix GV > > >>>>> Data Infrastructure Engineer > > >>>>> Distributed Data Systems > > >>>>> LinkedIn > > >>>>> > > >>>>> f...@linkedin.com > > >>>>> linkedin.com/in/felixgv > > >>>>> > > >>>>> ________________________________________ > > >>>>> From: Chris Riccomini [criccom...@apache.org] > > >>>>> Sent: Wednesday, February 11, 2015 10:23 AM > > >>>>> To: Chris Riccomini > > >>>>> Cc: dev@samza.apache.org > > >>>>> Subject: Re: [DISCUSS] SQL workflow > > >>>>> > > >>>>> Hey Julian, > > >>>>> > > >>>>> It looks like Calcite is at least setup to publish to Maven: > > >>>>> > > >>>>> https://repository.apache.org/#nexus-search;quick~calcite > > >>>>> > > >>>>> Julian, can you publish SNAPSHOT releases with your streaming > changes > > >>> to > > >>>> it? > > >>>>> > > >>>>> Thanks! > > >>>>> Chris > > >>>>> > > >>>>> On Wed, Feb 11, 2015 at 10:08 AM, Chris Riccomini < > > >>> criccom...@apache.org > > >>>>> > > >>>>> wrote: > > >>>>> > > >>>>>> Hey Milinda, > > >>>>>> > > >>>>>> I've just committed SAMZA-484, so you should be able to re-base > and > > >>> get > > >>>>>> all the latest code. > > >>>>>> > > >>>>>>> But we need to add Calcite as a dependency and Calcite streaming > > >>>>>> support is only in Julian's branch > > >>>>>> > > >>>>>> We'll have to have Julian publish a release to Apache. This can > be a > > >>>>>> SNAPSHOT to Apache's SNAPSHOT repo, or a real Calcite release. We > > >>> can't > > >>>>>> check in binaries to the repo (licensing, release complexity, and > > just > > >>>>>> generally a bad idea). We'll have to coordinate with him. > > >>>>>> > > >>>>>> Cheers, > > >>>>>> Chris > > >>>>>> > > >>>>>> On Wed, Feb 11, 2015 at 9:43 AM, Milinda Pathirage < > > >>>> mpath...@umail.iu.edu> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> No, SAMZA-483 doesn't depend on SAMZA-484. But we need to add > > Calcite > > >>>> as a > > >>>>>>> dependency and Calcite streaming support is only in Julian's > > branch ( > > >>>>>>> https://github.com/julianhyde/incubator-calcite/tree/chi). What > > can > > >>>> we do > > >>>>>>> about that? > > >>>>>>> > > >>>>>>> Milinda > > >>>>>>> > > >>>>>>> On Wed, Feb 11, 2015 at 12:30 PM, Chris Riccomini < > > >>>> criccom...@apache.org> > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>>> Hey Milinda, > > >>>>>>>> > > >>>>>>>> That'd be great. Do you have any dependency on SAMZA-484? I > > haven't > > >>>> had > > >>>>>>> a > > >>>>>>>> chance to commit that one yet. > > >>>>>>>> > > >>>>>>>> Cheers, > > >>>>>>>> Chris > > >>>>>>>> > > >>>>>>>> On Wed, Feb 11, 2015 at 9:23 AM, Milinda Pathirage < > > >>>>>>> mpath...@umail.iu.edu> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> /need/want > > >>>>>>>>> > > >>>>>>>>> Milinda > > >>>>>>>>> > > >>>>>>>>> On Wed, Feb 11, 2015 at 12:22 PM, Milinda Pathirage < > > >>>>>>>> mpath...@umail.iu.edu > > >>>>>>>>>> > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> Hi Chris, > > >>>>>>>>>> > > >>>>>>>>>> Do you need me to create the SAMZA-483 patch against latest > > master > > >>>>>>> with > > >>>>>>>>>> SAMZA-482 patch? I think that will make it easier to review > the > > >>>>>>> patch? > > >>>>>>>>>> > > >>>>>>>>>> Thanks > > >>>>>>>>>> Milinda > > >>>>>>>>>> > > >>>>>>>>>> On Mon, Feb 9, 2015 at 11:39 AM, Chris Riccomini < > > >>>>>>>> criccom...@apache.org> > > >>>>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> Cool. Looks like we've got consensus. I'll move forward with > > RTC > > >>> on > > >>>>>>>> some > > >>>>>>>>>>> of > > >>>>>>>>>>> the early SAMZA-390 tickets. (SAMZA-482, SAMZA-483, > SAMZA-484) > > >>>>>>>>>>> > > >>>>>>>>>>> On Mon, Feb 9, 2015 at 8:35 AM, Yan Fang < > yanfang...@gmail.com > > > > > >>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>>> +1 on this. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Fang, Yan > > >>>>>>>>>>>> yanfang...@gmail.com > > >>>>>>>>>>>> +1 (206) 849-4108 > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Fri, Feb 6, 2015 at 4:38 PM, Chris Riccomini < > > >>>>>>>>> criccom...@apache.org> > > >>>>>>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>>> Hey all, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Are we +1 on this? I think Jakob was the only one who was > > >>>>>>> curious > > >>>>>>>>>>> about > > >>>>>>>>>>>> it. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Cheers, > > >>>>>>>>>>>>> Chris > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On Thu, Feb 5, 2015 at 1:22 PM, Yi Pan < > nickpa...@gmail.com> > > >>>>>>>> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> Hi, Jakob, > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Eh? Not sure what this means... > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> I mean SAMZA-484 depends on SAMZA-482, and neither are > > >>>>>>>>> committed. > > >>>>>>>>>>> So > > >>>>>>>>>>>>>> Navina > > >>>>>>>>>>>>>>> is having to post Yi's patch, as well as her own, on the > > >>>>>>> JIRA. > > >>>>>>>>> It > > >>>>>>>>>>>> makes > > >>>>>>>>>>>>>> it > > >>>>>>>>>>>>>>> really hard to do code reviews because you can't tell > > >>>>>>> whether > > >>>>>>>> Yi > > >>>>>>>>>>> made > > >>>>>>>>>>>>> the > > >>>>>>>>>>>>>>> changes or Navina did. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Just to add to the point. It is also difficult to always > see > > >>>>>>> a > > >>>>>>>>> long > > >>>>>>>>>>>> list > > >>>>>>>>>>>>> of > > >>>>>>>>>>>>>> changed files if the RB request is always based on the > > >>>>>>> master. > > >>>>>>>> It > > >>>>>>>>> is > > >>>>>>>>>>>>>> possible to have RB request based on another RB request (I > > >>>>>>> have > > >>>>>>>>>>> tried > > >>>>>>>>>>>> it > > >>>>>>>>>>>>>> before). But what happens if the base RB request is > > >>>>>>>>>>>> cancelled/discarded? > > >>>>>>>>>>>>> RB > > >>>>>>>>>>>>>> is not designed to track the revision changes in a > > dependency > > >>>>>>>>> chain. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Cheers, > > >>>>>>>>>>>>>>> Chris > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On Thu, Feb 5, 2015 at 12:16 PM, Jakob Homan < > > >>>>>>>> jgho...@gmail.com > > >>>>>>>>>> > > >>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> I want to avoid branches, > > >>>>>>>>>>>>>>>> Just curious, any reason for this? > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> and I also want to avoid revision control over JIRA > > >>>>>>>>>>>>>>>> Eh? Not sure what this means... > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>>>>> jg > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> On 4 February 2015 at 17:11, Chris Riccomini < > > >>>>>>>>>>>> criccom...@apache.org> > > >>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>> Hey all, > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> @Jakob, yeah I was thinking we'll follow our normal > > >>>>>>> flow. > > >>>>>>>>>>> RTC. I > > >>>>>>>>>>>>> just > > >>>>>>>>>>>>>>>>> wanted to set expectation that the code committed > > >>>>>>> might be > > >>>>>>>>>>> not up > > >>>>>>>>>>>>> to > > >>>>>>>>>>>>>>> our > > >>>>>>>>>>>>>>>>> normal quality initially (missing docs, no tests, etc). > > >>>>>>>>> Until > > >>>>>>>>>>> the > > >>>>>>>>>>>>>>> quality > > >>>>>>>>>>>>>>>>> is raised, we should think of this module as > > >>>>>>> experimental. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> @Milinda, awesome! Thanks. :) > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Cheers, > > >>>>>>>>>>>>>>>>> Chris > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> On Wed, Feb 4, 2015 at 11:57 AM, Milinda Pathirage < > > >>>>>>>>>>>>>>>> mpath...@umail.iu.edu> > > >>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Hi Chris, > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Hope we no longer need the SQL API. I'll create a RB > > >>>>>>> for > > >>>>>>>>>>> Calcite > > >>>>>>>>>>>>>>>>>> integration. > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Thanks > > >>>>>>>>>>>>>>>>>> Milinda > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> On Wed, Feb 4, 2015 at 1:31 PM, Chris Riccomini < > > >>>>>>>>>>>>>>> criccom...@apache.org> > > >>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> I think so. There was some RB downtime, but it just > > >>>>>>> got > > >>>>>>>>>>> fixed. > > >>>>>>>>>>>>> Yi, > > >>>>>>>>>>>>>>>>>> Navina, > > >>>>>>>>>>>>>>>>>>> Milinda, can you make sure your JIRAs have up to > > >>>>>>> date > > >>>>>>>>> RBs? > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> On Wed, Feb 4, 2015 at 10:24 AM, sriram < > > >>>>>>>>>>> sriram....@gmail.com > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Can we have updated RBs for all the three sub > > >>>>>>> tasks > > >>>>>>>>>>> before > > >>>>>>>>>>>> we > > >>>>>>>>>>>>>>>> commit? > > >>>>>>>>>>>>>>>>>>> This > > >>>>>>>>>>>>>>>>>>>> would help us to review even after we commit. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> On Wed, Feb 4, 2015 at 10:15 AM, Chris Riccomini < > > >>>>>>>>>>>>>>>>>> criccom...@apache.org> > > >>>>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Hey all, > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Yi, Navina, and Milinda have been working on > > >>>>>>>>> SAMZA-390 > > >>>>>>>>>>>>>>> sub-tickets > > >>>>>>>>>>>>>>>>>>>> related > > >>>>>>>>>>>>>>>>>>>>> to SQL operators. We're getting to the point > > >>>>>>> where > > >>>>>>>>> the > > >>>>>>>>>>>>> amount > > >>>>>>>>>>>>>> of > > >>>>>>>>>>>>>>>> work > > >>>>>>>>>>>>>>>>>>>>> floating around is quite large, and some tickets > > >>>>>>>>> build > > >>>>>>>>>>> off > > >>>>>>>>>>>>> of > > >>>>>>>>>>>>>>>> others. > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> I'm proposing that we commit this work into a > > >>>>>>>>> samza-sql > > >>>>>>>>>>>>>>> submodule > > >>>>>>>>>>>>>>>> on > > >>>>>>>>>>>>>>>>>>>>> master, and treat this module as experimental. I > > >>>>>>>> want > > >>>>>>>>>>> to > > >>>>>>>>>>>>> avoid > > >>>>>>>>>>>>>>>>>>> branches, > > >>>>>>>>>>>>>>>>>>>>> and I also want to avoid revision control over > > >>>>>>>> JIRA. > > >>>>>>>>>>> This > > >>>>>>>>>>>>>> means > > >>>>>>>>>>>>>>>> that > > >>>>>>>>>>>>>>>>>>>> there > > >>>>>>>>>>>>>>>>>>>>> will probably be a fair amount of commits/JIRAs > > >>>>>>> on > > >>>>>>>>> this > > >>>>>>>>>>>>> module > > >>>>>>>>>>>>>>> as > > >>>>>>>>>>>>>>>> we > > >>>>>>>>>>>>>>>>>>>>> iterate, but I think that's OK. > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Does this sound good to everyone? > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Cheers, > > >>>>>>>>>>>>>>>>>>>>> Chris > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>>>>> Milinda Pathirage > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> PhD Student | Research Assistant > > >>>>>>>>>>>>>>>>>> School of Informatics and Computing | Data to Insight > > >>>>>>>>> Center > > >>>>>>>>>>>>>>>>>> Indiana University > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> twitter: milindalakmal > > >>>>>>>>>>>>>>>>>> skype: milinda.pathirage > > >>>>>>>>>>>>>>>>>> blog: http://milinda.pathirage.org > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> -- > > >>>>>>>>>> Milinda Pathirage > > >>>>>>>>>> > > >>>>>>>>>> PhD Student | Research Assistant > > >>>>>>>>>> School of Informatics and Computing | Data to Insight Center > > >>>>>>>>>> Indiana University > > >>>>>>>>>> > > >>>>>>>>>> twitter: milindalakmal > > >>>>>>>>>> skype: milinda.pathirage > > >>>>>>>>>> blog: http://milinda.pathirage.org > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> -- > > >>>>>>>>> Milinda Pathirage > > >>>>>>>>> > > >>>>>>>>> PhD Student | Research Assistant > > >>>>>>>>> School of Informatics and Computing | Data to Insight Center > > >>>>>>>>> Indiana University > > >>>>>>>>> > > >>>>>>>>> twitter: milindalakmal > > >>>>>>>>> skype: milinda.pathirage > > >>>>>>>>> blog: http://milinda.pathirage.org > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> -- > > >>>>>>> Milinda Pathirage > > >>>>>>> > > >>>>>>> PhD Student | Research Assistant > > >>>>>>> School of Informatics and Computing | Data to Insight Center > > >>>>>>> Indiana University > > >>>>>>> > > >>>>>>> twitter: milindalakmal > > >>>>>>> skype: milinda.pathirage > > >>>>>>> blog: http://milinda.pathirage.org > > >>>>>>> > > >>>>>> > > >>>>>> > > >>>> > > >>>> > > >>> > > >> > > >> > > >> > > >> -- > > >> Milinda Pathirage > > >> > > >> PhD Student | Research Assistant > > >> School of Informatics and Computing | Data to Insight Center > > >> Indiana University > > >> > > >> twitter: milindalakmal > > >> skype: milinda.pathirage > > >> blog: http://milinda.pathirage.org > > > > > > > -- > Milinda Pathirage > > PhD Student | Research Assistant > School of Informatics and Computing | Data to Insight Center > Indiana University > > twitter: milindalakmal > skype: milinda.pathirage > blog: http://milinda.pathirage.org >
