Re: About planner questions!

2016-06-15 Thread Milinda Pathirage
t to know what is the difference between HepPlanner and > VolcanoPlanner? For streamSQL ,which one is better? Samza uses which model? > > Thank you very much! > > -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Computing | Data to Insight Center I

Re: How to partition a topic into multiple and how to create multiple Samza Containers

2016-03-19 Thread Milinda Pathirage
t; systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory > systems.kafka.samza.msg.serde=string > systems.kafka.consumer.zookeeper.connect=localhost:2181/ > systems.kafka.producer.bootstrap.servers=localhost:9092 > > # Job Coordinator > job.coordinator.system=kafka > job.coor

Re: Understand Samza default metrics

2016-02-24 Thread Milinda Pathirage
: > I have a document with some of the metrics. I had gathered these around > last summer, so they may be out-of-date. I have attached the document to > this email. Hope it can help. > > > > > > > > On Feb 24, 2016, at 7:10 AM, Milinda Pathirage > wrote: >

Re: Understand Samza default metrics

2016-02-24 Thread Milinda Pathirage
y MetricsSnapshotReporterFactory and JmxReporterFactory? > > > > I'm interested in seeing the basic metrics of the my samza job (e.g. > > messages_processed_per_sec). But it's hard to ping point to the specific > > metric that shows me that. > > > > Thanks,

Re: [DISCUSS] Moving to github/pull-request for code review and check-in

2016-02-18 Thread Milinda Pathirage
ld be much lighter if we take the pull request only approach, as > >>> Kafka already converted to: > >>> - for the developers, the only thing needed is to open a pull request. > >>> - for committers, review and apply patch is from the same PR and merge > >>

Re: Review Request 37506: WIP: SAMZA-552 Operator API change: New Builder API

2015-09-03 Thread Milinda Pathirage
ly, visit: https://reviews.apache.org/r/37506/#review96136 ------- On Aug. 16, 2015, 3:57 p.m., Milinda Pathirage wrote: > > --- > This is an automatically generated e-mail.

Re: Review Request 37506: WIP: SAMZA-552 Operator API change: New Builder API

2015-09-03 Thread Milinda Pathirage
re-writing planner I wrote previously. I read the Drill code and find out a better way to write the planner. I think I can get rid of need for explicit stream insert operator. But AFAIK, we will still need a operator to modify tables. - Milinda

Re: Review Request 37506: WIP: SAMZA-552 Operator API change: New Builder API

2015-08-16 Thread Milinda Pathirage
:test passed Thanks, Milinda Pathirage

Re: Review Request 37506: WIP: SAMZA-552 Operator API change: New Builder API

2015-08-16 Thread Milinda Pathirage
/test/java/org/apache/samza/task/sql/StreamSqlTask.java 9124e3c samza-sql-core/src/test/java/org/apache/samza/task/sql/UserCallbacksSqlTask.java 96e96c3 Diff: https://reviews.apache.org/r/37506/diff/ Testing --- ./gradlew :samza-sql-core:test passed Thanks, Milinda Pathirage

Review Request 37506: WIP: SAMZA-552 Operator API change: New Builder API

2015-08-16 Thread Milinda Pathirage
/StreamSqlTask.java 9124e3c7191aac8a738345c02b260baae811c5c5 samza-sql-core/src/test/java/org/apache/samza/task/sql/UserCallbacksSqlTask.java 96e96c363e723c20c30622d1253588f66166b206 Diff: https://reviews.apache.org/r/37506/diff/ Testing --- ./gradlew :samza-sql-core:test passed Thanks, Milinda

Re: How should Samza be run on AWS?

2015-08-05 Thread Milinda Pathirage
Hi all, Looks like github deployment used by my university doesn't allow public access. I moved it to github (https://github.com/milinda/samza-ec2-ansible). Thanks Milinda On Wed, Aug 5, 2015 at 2:03 PM, Milinda Pathirage wrote: > I wrote several Ansible playbooks to deploy YARN

Re: Review Request 34500: SAMZA-552 Operator API change: builder and simplified operator classes

2015-08-05 Thread Milinda Pathirage
> On Aug. 5, 2015, 5:34 p.m., Milinda Pathirage wrote: > > I went through old discussions and also went through Calcite's RelBuilder > > (https://github.com/milinda/incubator-calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java) > > to l

Re: How should Samza be run on AWS?

2015-08-05 Thread Milinda Pathirage
from our main Kafka cluster. The main reason is that we > wanted > >to isolate off the disk and network load of state compactions and > >restores (we don't use compacted topics in our main Kafka cluster, but > >we do use them with Samza, and the extra load on Kafka can be > >substantial). > > > -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Computing | Data to Insight Center Indiana University twitter: milindalakmal skype: milinda.pathirage blog: http://milinda.pathirage.org

Re: Review Request 34500: SAMZA-552 Operator API change: builder and simplified operator classes

2015-08-05 Thread Milinda Pathirage
``GROUP BY``` clauses. * Above assumes that there is no multi-output operators. * Reusable sub-queries are not present in the above example, I'll think about it and introduce a mechanism to re-use sub-queries (Possibly introducing the view concept) Please feel free to comment on this. - M

Re: [VOTE] Apache Samza 0.9.1 RC1

2015-07-01 Thread Milinda Pathirage
; > > > On 6/30/15, 7:13 PM, "Chris Riccomini" wrote: > > > > >+1 > > > > > >Verified MD5, and asc signature. Build locally, and all tests pass. > > > > > >Cheers, > > >Chris > > > > > >On Tue

Re: [VOTE] Apache Samza 0.9.1 RC1

2015-06-30 Thread Milinda Pathirage
MZA%20AND%20fixVersion%20%3D%200.9.1%20AND%20status%20in%20%28Resolved%2C%20Closed%29 > > > > The vote will be open for 72 hours ( end in 12:00pm Wed, 07/01/2015 ). > > Please download the release candidate, check the hashes/signature, build > it > > and test it, and t

Re: Samza and sliding window

2015-06-29 Thread Milinda Pathirage
just have 2 topics - raw and processed and we use default > partitioning scheme. > Can you share any code snippet so I can understand it better? > > - Shekar > -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Computing | Data to Insight Center

Re: Samza and sliding window

2015-06-29 Thread Milinda Pathirage
hekar > > > > On Fri, Jun 26, 2015 at 11:39 AM, Shekar Tippur > wrote: > > > >> Thanks Milinda. > >> Is this feature available on 0.8 version of Samza? > >> > >> - Shekar > >> > >> On Fri, Jun 26, 2015 at 11:24 AM, Milinda Path

Re: Triggering emits for streaming window aggregates

2015-06-26 Thread Milinda Pathirage
e? > > -Yi > > On Fri, Jun 26, 2015 at 10:49 AM, Milinda Pathirage > > wrote: > > > Hi Julian, > > > > Even though this is a general question across all the streaming > aggregates > > which utilize GROUP BY clause and a monotonic timestamp field for

Re: Samza and sliding window

2015-06-26 Thread Milinda Pathirage
name. I want > to > > be > > > able to count hte number of events happening for that particular > > > application in the past 5 minutes (sliding window) and update either > > > another topic or a local cache. > > > > > > Is this possible via 0.9 version of Samza? > > > If not, what is the easiest way to achieve this? > > > > > > - Shekar > > > > > > -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Computing | Data to Insight Center Indiana University twitter: milindalakmal skype: milinda.pathirage blog: http://milinda.pathirage.org

Triggering emits for streaming window aggregates

2015-06-26 Thread Milinda Pathirage
metadata to LogicalAggregate so that we can get monotonic fields in the GROUP By clause Thanks in advance Milinda -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Computing | Data to Insight Center Indiana University twitter: milindalakmal skype: milinda.pathirage

Re: monitoring best practices

2015-06-25 Thread Milinda Pathirage
than going via Kafka in method #2), but I'd > rather not re-implement the batching and other logic in the existing > SnapshotMetricsReporter. > > Thanks. > -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Computing | Data to Insight Center

Re: Hopping and tumbling windows in streaming SQL

2015-06-25 Thread Milinda Pathirage
unction that returns the lower bound of the > current sub-total; END similarly. > > Note that the "retain" argument does not need to be a whole multiple > of the "emit" argument. This was a major limitation in the previous > proposal. > > There are some straightf

Re: Installing Samza w/o internet connection

2015-06-25 Thread Milinda Pathirage
le.api.plugins:gradle-nexus-plugin:0.7.1 > > Isn't there a simpler way to prepare samza? > Maybe a bunch of relevant jars? > > Thanks in advance, > Amos. > -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Computing | Data to Insight C

Re: Deploying samza job to a remote yarn resource manager

2015-06-22 Thread Milinda Pathirage
ble. What should be the value of this > >variable if my yarn-site.xml is at > >$HOME/hadoop-2.6.0/etc/hadoop/yarn-site.xml? > > > >Thanks, > > > >George > > -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Computing | Data to Insight Center Indiana University twitter: milindalakmal skype: milinda.pathirage blog: http://milinda.pathirage.org

Re: Measuring Samza Job Throughput

2015-06-17 Thread Milinda Pathirage
/samza-core/src/main/scala/org/apache/samza/container/SamzaContainerMetrics.scala > ). > > But this metric is more like a "QPS" type of metric . > > Thanks, > -Tao > > On Tue, Jun 16, 2015 at 9:11 PM, Milinda Pathirage > wrote: > > > Hi Devs, > > &

Measuring Samza Job Throughput

2015-06-16 Thread Milinda Pathirage
g job throughput. Thanks Milinda -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Computing | Data to Insight Center Indiana University twitter: milindalakmal skype: milinda.pathirage blog: http://milinda.pathirage.org

Re: [2/2] samza git commit: Yi's TopologyBuilder RB 34500

2015-06-02 Thread Milinda Pathirage
al mistake. I have reverted the check-in. I am still > >working on that. Thanks! > > > >-Yi > > > >On Mon, Jun 1, 2015 at 9:34 PM, Milinda Pathirage > >wrote: > > > >> Hi Navina, > >> > >> Did we decided to push this patch to s

Re: [2/2] samza git commit: Yi's TopologyBuilder RB 34500

2015-06-01 Thread Milinda Pathirage
Hi Navina, Did we decided to push this patch to samza-sql branch. I thought Yi is still working on this. Some Git conflict related texts are still there in this commit. +<<< HEAD + * The callback object +=== + * The callback function +>>> SAMZA-552: use OperatorCallback to allow i

Re: Review Request 33280: [SAMZA-561] Basic streaming SQL query planning support

2015-05-29 Thread Milinda Pathirage
estion: do we have a way to specify the primary keys/index keys used > > in the stream? > > Milinda Pathirage wrote: > We don't have a way to specify primary key in the current code. But I am > wondering whether primary key is really required during query execu

Re: Review Request 34500: SAMZA-552 Operator API change: builder and simplified operator classes

2015-05-28 Thread Milinda Pathirage
--- On May 20, 2015, 11:13 p.m., Yi Pan (Data Infrastructure) wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/34500/ >

Re: Review Request 34500: SAMZA-552 Operator API change: builder and simplified operator classes

2015-05-28 Thread Milinda Pathirage
two inputs? - Milinda Pathirage On May 20, 2015, 11:13 p.m., Yi Pan (Data Infrastructure) wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://rev

Re: Review Request 33280: [SAMZA-561] Basic streaming SQL query planning support

2015-05-26 Thread Milinda Pathirage
Just a question: is this just a dummy op that never materialized? > > Milinda Pathirage wrote: > Actually, we don't need this. I'll fix the code. Please ignore above comment. StreamScanSpec is needed to handle

Re: Review Request 33280: [SAMZA-561] Basic streaming SQL query planning support

2015-05-26 Thread Milinda Pathirage
gt; > > > Then, in method visit(RelNode node, int ordinal, RelNode parent) > > > > We can always do: > > > > this.opBuilder.setInputs(inputs) // if necessary > > router.addOperator(this.opBuilder.build()); > > Milinda Pathirage w

Re: Review Request 33280: [SAMZA-561] Basic streaming SQL query planning support

2015-05-26 Thread Milinda Pathirage
visit: https://reviews.apache.org/r/33280/#review85153 ----------- On May 23, 2015, 4:27 p.m., Milinda Pathirage wrote: > > --- > This is an automatically generat

Re: Review Request 33280: [SAMZA-561] Basic streaming SQL query planning support

2015-05-26 Thread Milinda Pathirage
chema/AvroSchemaUtils.java, > > line 225 > > <https://reviews.apache.org/r/33280/diff/5/?file=970943#file970943line225> > > > > I think that we should support timestamp and skip date/time now. okay. - Milinda -------

Re: Review Request 33280: [SAMZA-561] Basic streaming SQL query planning support

2015-05-23 Thread Milinda Pathirage
RE-CREATION Diff: https://reviews.apache.org/r/33280/diff/ Testing --- * ./bin/check-all.sh passed. * Integration tests passed including new streaming sql integration test. Thanks, Milinda Pathirage

Re: Review Request 33280: [SAMZA-561] Basic streaming SQL query planning support

2015-05-19 Thread Milinda Pathirage
resources/orders.json PRE-CREATION Diff: https://reviews.apache.org/r/33280/diff/ Testing --- * ./bin/check-all.sh passed. * Integration tests passed including new streaming sql integration test. Thanks, Milinda Pathirage

Re: Review Request 33280: [SAMZA-561] Basic streaming SQL query planning support

2015-05-19 Thread Milinda Pathirage
REATION samza-test/src/main/resources/orders.json PRE-CREATION Diff: https://reviews.apache.org/r/33280/diff/ Testing --- * ./bin/check-all.sh passed. * Integration tests passed including new streaming sql integration test. Thanks, Milinda Pathirage

Re: Review Request 34206: WIP: update operator API to allow callbacks and allow a single API to trigger OperatorRouter execution w/ user callbacks

2015-05-15 Thread Milinda Pathirage
samza-sql branch if others are okay with the patch. - Milinda Pathirage On May 15, 2015, 2:16 a.m., Yi Pan (Data Infrastructure) wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache

Re: Review Request 34206: WIP: update operator API to allow callbacks and allow a single API to trigger OperatorRouter execution w/ user callbacks

2015-05-15 Thread Milinda Pathirage
/Table.java <https://reviews.apache.org/r/34206/#comment134999> How about supporting multi column primary keys? We can add it to the API even if we are not going to support it at the begining. - Milinda Pathirage On May 15, 2015, 2:16 a.m., Yi Pan (Data Infrastructure)

Re: Review Request 33280: [SAMZA-561] Basic streaming SQL query planning support

2015-05-12 Thread Milinda Pathirage
e separate out planning and SamzaSqlTask we can get rid of this. - Milinda ----------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33280/#review83250 ---

Re: Review Request 33280: [SAMZA-561] Basic streaming SQL query planning support

2015-05-12 Thread Milinda Pathirage
nerated in a config format > > and the StreamTask will be able to instantiate the OperatorRouter from a > > config. That way, the parser/planner can be only in the deployment console > > and does not need to be installed in each Samza container. But this co

Re: Review Request 34009: WIP: SAMZA-650 window store implementation

2015-05-12 Thread Milinda Pathirage
/StreamStreamJoin.java <https://reviews.apache.org/r/34009/#comment134390> Don't we need to make this final because we are passing tuples to an inner class? - Milinda Pathirage On May 9, 2015, 1:52 a.m., Yi Pan (Data Infrastruc

Re: Review Request 34009: WIP: SAMZA-650 window store implementation

2015-05-12 Thread Milinda Pathirage
samza-sql branch first and change the SAMZA-561 patch to work with latest API changes. - Milinda Pathirage On May 9, 2015, 1:52 a.m., Yi Pan (Data Infrastructure) wrote: > > --- > This is an automatically generated e-mail. To rep

Re: Review Request 34009: WIP: SAMZA-650 window store implementation

2015-05-12 Thread Milinda Pathirage
rCallbacksSqlTask.java <https://reviews.apache.org/r/34009/#comment134384> Why this class is named UserCallbacksSqlTask? - Milinda Pathirage On May 9, 2015, 1:52 a.m., Yi Pan (Data Infrastructure) wrote: > > --- > This is an aut

Re: Review Request 33749: WIP: SAMZA-650 window store implementation

2015-05-07 Thread Milinda Pathirage
t; When will this input disabling happens for a window operator? - Milinda Pathirage On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > http

Re: Review Request 33749: WIP: SAMZA-650 window store implementation

2015-05-07 Thread Milinda Pathirage
/StreamStreamJoiner.java <https://reviews.apache.org/r/33749/#comment133651> I think that stream-to-stream joining is not practical for possibly infinite streams. We may need to define some constraints or some other restrictions to make this practical. - Milinda Pathirage On May 4, 2

Re: Review Request 33749: WIP: SAMZA-650 window store implementation

2015-05-06 Thread Milinda Pathirage
e. But next will return null. So this doesn't exactly adhere to iterator semantics we know. - Milinda Pathirage On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote: > > --- > This is an automatically generated e-

Re: Review Request 33749: WIP: SAMZA-650 window store implementation

2015-05-06 Thread Milinda Pathirage
/MessageStore.java <https://reviews.apache.org/r/33749/#comment133461> Invalid parameter in doc comment. - Milinda Pathirage On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote: > > --- > This is an automatica

Re: What next for streaming SQL?

2015-04-30 Thread Milinda Pathirage
;> As of calcite-1.2, the streaming extensions are in Calcite’s master >> branch. (See >> https://github.com/apache/incubator-calcite/blob/master/doc/STREAM.md.) >> We are a couple of weeks away from the next Calcite release. If you need >> some work done in Calcite, now would

Re: What next for streaming SQL?

2015-04-28 Thread Milinda Pathirage
ing extensions are in Calcite’s master > branch. (See > https://github.com/apache/incubator-calcite/blob/master/doc/STREAM.md.) > We are a couple of weeks away from the next Calcite release. If you need > some work done in Calcite, now would be a good time. > > Julian > >

Re: Review Request 33142: [SAMZA-561] Review in progress

2015-04-21 Thread Milinda Pathirage
> On April 14, 2015, 10:14 p.m., Yi Pan (Data Infrastructure) wrote: > > samza-sql/src/main/java/org/apache/samza/sql/metadata/Stream.java, line 37 > > <https://reviews.apache.org/r/33142/diff/1/?file=926293#file926293line37> > > > > What's the meaning

Re: Review Request 33280: [SAMZA-561] Basic streaming SQL query planning support

2015-04-18 Thread Milinda Pathirage
vsc PRE-CREATION samza-test/src/main/resources/orders.json PRE-CREATION Diff: https://reviews.apache.org/r/33280/diff/ Testing (updated) --- * ./bin/check-all.sh passed. * Integration tests passed including new streaming sql integration test. Thanks, Milinda Pathirage

Re: Review Request 33280: [SAMZA-561] Basic streaming SQL query planning support

2015-04-18 Thread Milinda Pathirage
ests passed including new streaming sql integration test. Thanks, Milinda Pathirage

Review Request 33280: [SAMZA-561] Basic streaming SQL query planning support

2015-04-16 Thread Milinda Pathirage
ain/resources/orders.avsc PRE-CREATION samza-test/src/main/resources/orders.json PRE-CREATION Diff: https://reviews.apache.org/r/33280/diff/ Testing --- Integration tests passed including new streaming sql integration test. Thanks, Milinda Pathirage

Re: Review Request 33142: [SAMZA-561] Review in progress

2015-04-16 Thread Milinda Pathirage
e) wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/33142/ > --- > > (Updated April 13, 2015, 9:04 p.m.) > >

Re: Review Request 33142: [SAMZA-561] Review in progress

2015-04-16 Thread Milinda Pathirage
- > > (Updated April 13, 2015, 9:04 p.m.) > > > Review request for samza and Milinda Pathirage. > > > Bugs: SAMZA-561 > https://issues.apache.org/jira/browse/SAMZA-561 > > > Repository: samza >

Re: Review Request 33219: [SAMZA-649] Create samza-sql-calcite module for Calcite SQL front end

2015-04-15 Thread Milinda Pathirage
--- ./bin/check-all.sh passed. Thanks, Milinda Pathirage

Re: Review Request 33219: [SAMZA-649] Create samza-sql-calcite module for Calcite SQL front end

2015-04-15 Thread Milinda Pathirage
-- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33219/#review80228 ----------- On April 15, 2015, 2:49 p.m., Milinda Pathirage wrote: > >

Review Request 33219: [SAMZA-649] Create samza-sql-calcite module for Calcite SQL front end

2015-04-15 Thread Milinda Pathirage
-core/src/test/java/org/apache/samza/sql/test/metadata/TestAvroSchemaConverter.java b4ac5f5 settings.gradle 5cbb755 Diff: https://reviews.apache.org/r/33219/diff/ Testing --- ./bin/check-all.sh passed. Thanks, Milinda Pathirage

Re: Updating samza-sql branch to Java 1.7

2015-04-15 Thread Milinda Pathirage
> > > On 14 April 2015 at 12:47, Milinda Pathirage > > wrote: > > > Hi Devs, > > > > > > Calcite dropped support for Java 1.6 in 1.1.0-incubating. I want to use > > > Calcite 1.2.0-incubating-SNAPSHOT in samza-sql branch. Is it okay

Updating samza-sql branch to Java 1.7

2015-04-14 Thread Milinda Pathirage
Hi Devs, Calcite dropped support for Java 1.6 in 1.1.0-incubating. I want to use Calcite 1.2.0-incubating-SNAPSHOT in samza-sql branch. Is it okay to update samza-sql branch to Java 1.7? Thanks Milinda -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Computing

Re: Review Request 33170: Renamed samza-sql to samza-sql-core

2015-04-14 Thread Milinda Pathirage
er. - Milinda --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33170/#review80043 --- On April 14, 2015, 3:14 p.m., Milinda Pathirage wrote: > > ---

Review Request 33170: Renamed samza-sql to samza-sql-core

2015-04-14 Thread Milinda Pathirage
settings.gradle 08e548c Diff: https://reviews.apache.org/r/33170/diff/ Testing --- Thanks, Milinda Pathirage

Stream SQL Query Planner Update

2015-04-06 Thread Milinda Pathirage
other comments on improvements are welcome. Thanks Milinda -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Computing | Data to Insight Center Indiana University twitter: milindalakmal skype: milinda.pathirage blog: http://milinda.pathirage.org

Re: Kafka partition key

2015-03-26 Thread Milinda Pathirage
> > > > > This email and any attachments may contain confidential and privileged > > > material for the sole use of the intended recipient. Any review, > copying, > > > or distribution of this email (or any attachments) by others is > > prohibited. > > > If you are not the intended recipient, please contact the sender > > > immediately and permanently delete this email and any attachments. No > > > employee or agent of TiVo Inc. is authorized to conclude any binding > > > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo > > > Inc. may only be made by a signed written agreement. > > > > > > -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Computing | Data to Insight Center Indiana University twitter: milindalakmal skype: milinda.pathirage blog: http://milinda.pathirage.org

Re: Extraneous wiki page

2015-03-10 Thread Milinda Pathirage
can't delete it or > > move it out of the space: > > > > > https://cwiki.apache.org/confluence/display/SAMZA/Social+Media+Guidelines > > > > Could someone with sufficient karma go ahead and remove that for me? It's > > already been re-created in the

Re: Handling defaults and windowed aggregates in stream queries

2015-03-06 Thread Milinda Pathirage
I think my previous comment about maintaining start and end offsets as the window state will not work when there are delays. We may need to keep multiple such offsets. But this may not be a clean solution. On Thu, Mar 5, 2015 at 2:42 PM, Milinda Pathirage wrote: > Hi Yi, > > Pleas

Re: Handling defaults and windowed aggregates in stream queries

2015-03-05 Thread Milinda Pathirage
of the physical window operator which provides: > a) a varying/growing materialized view; b) a driver that tells the > aggregation/join to compute/re-compute results on-top-of the materialized > view. > > > I will think more about this model and may have more questions about this in f

Re: Handling defaults and windowed aggregates in stream queries

2015-03-04 Thread Milinda Pathirage
t; current row) > on orders.orderId = shipments.orderId (query 7) > and shipments.rowtime between orders.rowtime and orders.rowtime + > interval '1' hour > > So, my hope is that end-users will rarely need to use an explicit window. > > In the algebra, we will

Re: Handling defaults and windowed aggregates in stream queries

2015-03-02 Thread Milinda Pathirage
ce between > a rule and a re-write? > > Thanks! > > On Mon, Mar 2, 2015 at 7:14 AM, Milinda Pathirage > wrote: > > > @Chris > > Yes, I was referring to that mail. Actually I was wrong about the ‘Now’ > > window, it should be a ‘Unbounded’ window for

Re: Handling defaults and windowed aggregates in stream queries

2015-03-02 Thread Milinda Pathirage
x27;t it > much clear if the following is used? > LogicalProject(EXPR$0=[CASE(>(COUNT($2), 0), CAST($SUM0($2)):INTEGER, > null)]) >LogicalWindow(ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING) > > On Thu, Feb 26, 2015 at 12:18 PM, Milinda Pathirage > > wrote: > > &g

Review Request 31556: Avro schema to Calcite RelDataType conveter (SAMZA-582)

2015-02-27 Thread Milinda Pathirage
://reviews.apache.org/r/31556/diff/ Testing --- ./bin/check-all.sh: successful Thanks, Milinda Pathirage

Handling defaults and windowed aggregates in stream queries

2015-02-26 Thread Milinda Pathirage
th our operator layer. May be there are better or easier ways to do this. So your comments are always welcome. Thanks Milinda [1] http://mail-archives.apache.org/mod_mbox/incubator-calcite-dev/201502.mbox/browser -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Com

Re: Review Request 31290: SAMAZ-544

2015-02-25 Thread Milinda Pathirage
> On Feb. 25, 2015, 9:43 p.m., Milinda Pathirage wrote: > > Ship It! I think I found an issue. But I am not sure whether this is caused by my environment. I'm getting following error: zopkio [ERROR] incorrect dir structure testfile:/tmp/samza-integraiton2/scripts/tests.py exist

Re: Review Request 31290: SAMAZ-544

2015-02-25 Thread Milinda Pathirage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31290/#review74096 --- Ship it! Ship It! - Milinda Pathirage On Feb. 23, 2015, 12:23

Re: Review Request 31405: Serde Implementation or AvroData (SAMZA-575)

2015-02-25 Thread Milinda Pathirage
at org.apache.samza.system.kafka.TestKafkaSystemAdmin.testShouldGetOldestNewestAndNextOffsets(TestKafkaSystemAdmin.scala:239) Thanks, Milinda Pathirage

Re: Review Request 31405: Serde Implementation or AvroData (SAMZA-575)

2015-02-25 Thread Milinda Pathirage
iew74060 --- On Feb. 25, 2015, 5:54 a.m., Milinda Pathirage wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/3

Re: Test failure in samza-sql branch

2015-02-25 Thread Milinda Pathirage
I ran a bin/check-all.sh after the merge this morning, and didn't see > any failures. > > Cheers, > Chris > > On Tuesday, February 24, 2015, Milinda Pathirage > wrote: > > > Hi Devs, > > > > Has anyone seen following test

Review Request 31405: Serde Implementation or AvroData (SAMZA-575)

2015-02-24 Thread Milinda Pathirage
at org.junit.Assert.assertEquals(Assert.java:126) at org.junit.Assert.assertEquals(Assert.java:145) at org.apache.samza.system.kafka.TestKafkaSystemAdmin.testShouldGetOldestNewestAndNextOffsets(TestKafkaSystemAdmin.scala:239) Thanks, Milinda Pathirage

Test failure in samza-sql branch

2015-02-24 Thread Milinda Pathirage
at org.junit.Assert.assertEquals(Assert.java:126) at org.junit.Assert.assertEquals(Assert.java:145) at org.apache.samza.system.kafka.TestKafkaSystemAdmin.testShouldGetOldestNewestAndNextOffsets(TestKafkaSystemAdmin.scala:239) I got this on samza-sql branch. Thanks Milinda -- Milinda Pathirag

Re: Integration test deployment script doesn't work when there are two jobs

2015-02-24 Thread Milinda Pathirage
Looks like rebasing is already done. I didn't see the commit messages. Milinda On Tue, Feb 24, 2015 at 3:08 PM, Milinda Pathirage wrote: > Hi Chris, > > Thanks for the info. I'll rebase to the latest 0.9.0 branch. I think its > better if we can rebase the samza-sql bra

Re: Integration test deployment script doesn't work when there are two jobs

2015-02-24 Thread Milinda Pathirage
hat this requires that jobs shut > themselves down when they're "done". See SAMZA-544, or the negate-number > example (on master) on how this is done. > > I can also rebase the samza-sql Apache branch as well. > > Cheers, > Chris > > On Tue, Feb 24, 2015 at

Re: Integration test deployment script doesn't work when there are two jobs

2015-02-24 Thread Milinda Pathirage
Hi Chris, You can find my changes at https://github.com/milinda/samza/tree/samza-integration-tests Thanks Milinda On Tue, Feb 24, 2015 at 12:04 PM, Milinda Pathirage wrote: > Hi Chris, > > Will push it to a branch in my Samza fork and send you the link. > > Thanks > Milinda

Re: Integration test deployment script doesn't work when there are two jobs

2015-02-24 Thread Milinda Pathirage
? > > Cheers, > Chris > > On Tue, Feb 24, 2015 at 8:58 AM, Milinda Pathirage > wrote: > > > Hi Devs, > > > > I tried to add a new integration test to samza-test and found out that > the > > second test always fail if that test depends on a different

Integration test deployment script doesn't work when there are two jobs

2015-02-24 Thread Milinda Pathirage
umber test worked without any issues) when I had both those jobs in deployment.py. But when I remove the 'negate_number' job from deployment script, my test passed without any issues. Do any of you have experienced this situation before? Thanks Milinda -- Milinda Pathirage Ph

Re: Process of adding a new integration test

2015-02-18 Thread Milinda Pathirage
trial > and error. It'd be good to use your experience to better document stuff, so > those in the future can add integration tests easily. > > Cheers, > Chris > > On Wed, Feb 18, 2015 at 8:35 AM, Milinda Pathirage > wrote: > > > Hi, > > > > What

Process of adding a new integration test

2015-02-18 Thread Milinda Pathirage
Hi, What is the process I need to follow to add a new integration test to the samza-test? Is there a easy way, if I just need to initialize a task and send some messages to it? Thanks Milinda -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Computing | Data to

Re: Stream SQL for Samza Query Language Guide and Design

2015-02-17 Thread Milinda Pathirage
will make it versioned, and editable. Comments can happen in > the JIRA. I'm OK either way, but the wiki has been a little cumbersome in > the past. > > Cheers, > Chris > > On Mon, Feb 16, 2015 at 1:59 PM, Milinda Pathirage > wrote: > > > Hi Chris and Yi, >

Stream SQL for Samza Query Language Guide and Design

2015-02-16 Thread Milinda Pathirage
can do shared editing? Thanks Milinda -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Computing | Data to Insight Center Indiana University twitter: milindalakmal skype: milinda.pathirage blog: http://milinda.pathirage.org

Re: [DISCUSS] SQL workflow

2015-02-12 Thread Milinda Pathirage
the branch (, Jakob said pedantically). > > > > On 12 February 2015 at 08:25, Milinda Pathirage > wrote: > >> I am +1 for this approach. > >> > >> Milinda > >> > >> On Thu, Feb 12, 2015 at 11:22 AM, Chris Riccomini < > criccom...@a

Re: [DISCUSS] SQL workflow

2015-02-12 Thread Milinda Pathirage
Wed, Feb 11, 2015 at 10:08 AM, Chris Riccomini < > criccom...@apache.org > > > > > > wrote: > > > > > >> Hey Milinda, > > >> > > >> I've just committed SAMZA-484, so you should be able to re-base and > get > > >> a

Re: Review Request 30667: Samza Calcite Integration Prototype (SAMZA-483)

2015-02-11 Thread Milinda Pathirage
stemConsumer-0" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MockSystemConsumer-9" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MockSystemConsumer-10" Thanks, Milinda Pathirage

Re: [DISCUSS] SQL workflow

2015-02-11 Thread Milinda Pathirage
ini wrote: > Hey Milinda, > > That'd be great. Do you have any dependency on SAMZA-484? I haven't had a > chance to commit that one yet. > > Cheers, > Chris > > On Wed, Feb 11, 2015 at 9:23 AM, Milinda Pathirage > wrote: > > > /need/want > &

Re: [DISCUSS] SQL workflow

2015-02-11 Thread Milinda Pathirage
/need/want Milinda On Wed, Feb 11, 2015 at 12:22 PM, Milinda Pathirage wrote: > Hi Chris, > > Do you need me to create the SAMZA-483 patch against latest master with > SAMZA-482 patch? I think that will make it easier to review the patch? > > Thanks > Milinda > > O

Re: [DISCUSS] SQL workflow

2015-02-11 Thread Milinda Pathirage
eans... > > > > > > > > > > > > Thanks, > > > > > > jg > > > > > > > > > > > > On 4 February 2015 at 17:11, Chris Riccomini < > > criccom...@apache.org> > > > > > > wrote: > >

Re: Review Request 30667: Samza Calcite Integration Prototype (SAMZA-483)

2015-02-08 Thread Milinda Pathirage
ask.java PRE-CREATION settings.gradle bb07a3b Diff: https://reviews.apache.org/r/30667/diff/ Testing --- Thanks, Milinda Pathirage

Review Request 30667: Samza Calcite Integration Prototype (SAMZA-483)

2015-02-05 Thread Milinda Pathirage
org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION settings.gradle bb07a3b Diff: https://reviews.apache.org/r/30667/diff/ Testing --- Thanks, Milinda Pathirage

Re: [DISCUSS] SQL workflow

2015-02-04 Thread Milinda Pathirage
his means that > > there > > > will probably be a fair amount of commits/JIRAs on this module as we > > > iterate, but I think that's OK. > > > > > > Does this sound good to everyone? > > > > > > Cheers, > > > Chris > > > > > > -- Milinda Pathirage PhD Student | Research Assistant School of Informatics and Computing | Data to Insight Center Indiana University twitter: milindalakmal skype: milinda.pathirage blog: http://milinda.pathirage.org

  1   2   >