Hi Wail,
so you are actually not looking for for a way to limit the size, but
really
for a way to paginate the whole result, is that right?
IIRC the question of GC for the results materialized by the result
distribution is not nicely solved anyway, so that might not bring
additional
issues for now ... but we also shouldn't introduce new heavy
requirements on
this.
Cheers,
Till
On 16 Apr 2016, at 3:41, Wail Alkowaileet wrote:
Hi Till,
Just for a clarification, I have implemented a remote result iterator
on
Asterix-Spark connector. The problem I had was when the result
partitions
are in between Asterix and Spark, there are no disk-spilling
mechanism. So
when the partitions are too big I get OOM. Now the result are
delivered in
a limited number of frames (as you explained) and pull more frames
when a
threshold is consumed.
The same technique can adopted in the GUI mode. However, this way can
introduce dangling results which would live for 24 hours. Probably, we
should have an "abort" mechanism when the session is ended/closed.
On Sat, Apr 16, 2016 at 8:35 AM, Mike Carey <[email protected]> wrote:
+1 for the separation and also for providing a place where we can
deliver
warnings - I remember longing for a few recently (we might have one
or more
JIRA issues mentioning that).....
On 4/15/16 8:16 PM, Till Westmann wrote:
Yes, indeed the warning are feature-creep and not essential. We have
been
talking about compiler warnings for along time, but we don’t have
them so
far. The warnings that we could show for the HTTP API are warnings
about
unused or ignored parameters but, again, they are not essential.
It'll be
perfectly fine to add that at a later time when there's a concrete
need/use
for them.
Also, thinking about it a bit more, I'm even more convinced that
it's the
better approach to have different fields for results and result
handles as
the client can know if it needs another hop just by looking a the
response,
without knowing what the request was.
Cheers,
Till
On 15 Apr 2016, at 17:32, Ildar Absalyamov wrote:
Till,
All the comments make sense to me except for warnings. I was
struggling
to remember anything in Asterix, which would resemble the warnings
described by you.
As a future extension we could support that, but in this document I
was
trying to reflect current state of Asterix functionality + some
extensions
which were on horizon for a long time and are really needed.
On Apr 15, 2016, at 14:37, Till Westmann <[email protected]> wrote:
Hi Ildar,
thanks for writing all of this up!
A few comments/proposals:
- Request:
- For the "format" parameter, I think that it would be nice to
support
both, the parameter and the Accept header, as it’s sometimes
much
more
convenient to pass a parameter than a HTTP header. However, I
think
that
the HTTP header should override the parameter if they conflict.
- "execute-query" could be renamed to "execute-statement" to be
consistent
with the "statement" parameter.
- Response:
- I'm wondering if "results" should be able to get a URI for the
handle or
if we should have another field in the response for that (e.g.
"handle"). The advantage of 2 fields would be that the consumer
knows
how to parse each field (either as a URI or as an array).
- Inside of the "metrics" object I would only expect simple
numbers.
For
the plans we could have another top-level field ("plan"?) an
object
that
contains the different plans ..
- It would also be nice to add a new top-level field for
warnings. That
could be used to report warnings from the engine that evaluates
the
statement. And it could also be used to report unused
parameters
(assuming that the default behavior for a parameter that is
passed
in,
but not understood by the server is simply ignored).
Thoughts?
Cheers,
Till
On 14 Apr 2016, at 15:23, Ildar Absalyamov wrote:
Hi Devs,
Recently there have been a number of conversations about the
future of
our REST (aka HTTP) API. I summarized these discussions in an
outline of
the new API design:
https://cwiki.apache.org/confluence/display/ASTERIXDB/New+HTTP+API+Design
<
https://cwiki.apache.org/confluence/display/ASTERIXDB/New+HTTP+API+Design>.
The need to refactor existing API came from different directions
(and
from different people), and is explained in motivation section.
Thus I
believe it’s about the time to take an effort and improve
existing API, so
that it will not drag us down in the future. However during the
transition
step I believe it would be better to keep exiting API endpoints,
so that we
would not break people’s current experimental setup.
It would be good to know feedback from the folks, who have been
contributing to that part of the systems recently.
Best regards,
Ildar
Best regards,
Ildar
--
*Regards,*
Wail Alkowaileet
