Re: Multiple streaming aggregations in structured streaming

[Michael Armbrust]

Doing this generally is pretty hard.  We will likely support algebraic
aggregate eventually, but this is not currently slotted for 2.2.  Instead I
think we will add something like mapWithState that lets users compute
arbitrary stateful things.  What is your use case?


On Wed, Nov 16, 2016 at 6:58 PM, wszxyh  wrote:

> Hi
>
> Multiple streaming aggregations are not yet supported. When will it be
> supported? Is it in the plan?
>
> Thanks
>
>
>
>

Re: [build system] massive jenkins infrastructure changes forthcoming

[shane knapp]

On Thu, Nov 17, 2016 at 4:52 PM, Reynold Xin  wrote:
> Thanks for the headsup, Shane.
>
no problem!

i'm really looking forward to starting w/a much cleaner slate that
what we have now.

not only are we locked in a jenkins/plugin version dependency hell
that keeps us from updating easily and regularly (esp WRT security
issues), but our jenkins logs completely useless for debugging due to
stack trace spam.

no joke, about 80%+ of the logs are stack traces.  :(

anyways, i will post more updates as they come in.

shane

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Failed to run spark jobs on mesos due to "hadoop" not found.

[Meethu Mathew]

Hi,

Add HADOOP_HOME=/path/to/hadoop/folder in /etc/default/mesos-slave in all
mesos agents and restart mesos

Regards,
Meethu Mathew


On Thu, Nov 10, 2016 at 4:57 PM, Yu Wei  wrote:

> Hi Guys,
>
> I failed to launch spark jobs on mesos. Actually I submitted the job to
> cluster successfully.
>
> But the job failed to run.
>
> I1110 18:25:11.095507   301 fetcher.cpp:498] Fetcher Info:
> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/1f8e621b-3cbf-4b86-a1c1-
> 9e2cf77265ee-S7\/root","items":[{"action":"BYPASS_CACHE","
> uri":{"extract":true,"value":"hdfs:\/\/192.168.111.74:9090\/
> bigdata\/package\/spark-examples_2.11-2.0.1.jar"}}],"
> sandbox_directory":"\/var\/lib\/mesos\/agent\/slaves\/
> 1f8e621b-3cbf-4b86-a1c1-9e2cf77265ee-S7\/frameworks\/
> 1f8e621b-3cbf-4b86-a1c1-9e2cf77265ee-0002\/executors\/
> driver-20161110182510-0001\/runs\/b561328e-9110-4583-b740-
> 98f9653e7fc2","user":"root"}
> I1110 18:25:11.099799   301 fetcher.cpp:409] Fetching URI 'hdfs://
> 192.168.111.74:9090/bigdata/package/spark-examples_2.11-2.0.1.jar'
> I1110 18:25:11.099820   301 fetcher.cpp:250] Fetching directly into the
> sandbox directory
> I1110 18:25:11.099862   301 fetcher.cpp:187] Fetching URI 'hdfs://
> 192.168.111.74:9090/bigdata/package/spark-examples_2.11-2.0.1.jar'
> E1110 18:25:11.101842   301 shell.hpp:106] Command 'hadoop version 2>&1'
> failed; this is the output:
> sh: hadoop: command not found
> Failed to fetch 'hdfs://192.168.111.74:9090/bigdata/package/spark-
> examples_2.11-2.0.1.jar': Failed to create HDFS client: Failed to execute
> 'hadoop version 2>&1'; the command was either not found or exited with a
> non-zero exit status: 127
> Failed to synchronize with agent (it's probably exited
>
> Actually I installed hadoop on each agent node.
>
>
> Any advice?
>
>
> Thanks,
>
> Jared, (韦煜）
> Software developer
> Interested in open source software, big data, Linux
>

Re: issues with github pull request notification emails missing

[Sean Owen]

I have seen the same issue from time to time where I couldn't see a
person's alias in the popup. Happened yesterday for me with @joshrosen. No
idea why.

I was also missing emails for a while to spam, but it seemed like a Gmail
problem. It said I had marked messages from about 6 different people as
spam, which doesn't seem like something i could have done with a mis-click.
So, worth looking out for if you're missing emails and using gmail.

Also, I noted that if you just add an "Approve" in the new review system
that does not generate an email. Which is maybe a good thing.

I do still see a big problem with Gmail and these messages not being
threaded. One PR's messages will typically split somewhat arbitrarily over
5-6 threads, which makes managing discussions really quite annoying. I
don't know if it's somehow related.

On Thu, Nov 17, 2016 at 10:10 PM Xiao Li  wrote:

Just FYI, normally, when we ping a people, the github can show the full
name after we type the github id. Below is an example:

[image: Screenshot 2016-11-17 14.05.55.png]

Starting from last week, Reynold's full name is not shown. Does github
update their hash functions?

[image: Screenshot 2016-11-17 14.05.38.png]

Thanks,

Xiao Li



2016-11-16 23:30 GMT-08:00 Holden Karau :

+1 it seems like I'm missing a number of my GitHub email notifications
lately (although since I run my own mail server and forward I've been
assuming it's my own fault).


I've also had issues with having greatly delayed notifications on some of
my own pull requests but that might be unrelated.

On Thu, Nov 17, 2016 at 8:20 AM Reynold Xin  wrote:

I've noticed that a lot of github pull request notifications no longer come
to my inbox. In the past I'd get an email for every reply to a pull request
that I subscribed to (i.e. commented on). Lately I noticed for a lot of
them I didn't get any emails, but if I opened the pull requests directly on
github, I'd see the new replies. I've looked at spam folder and none of the
missing notifications are there. So it's either github not sending the
notifications, or the emails are lost in transit.

The way it manifests is that I often comment on a pull request, and then I
don't know whether the contributor (author) has updated it or not. From the
contributor's point of view, it looks like I've been ignoring the pull
request.

I think this started happening when github switched over to the new code
review mode (
https://github.com/blog/2256-a-whole-new-github-universe-announcing-new-tools-forums-and-features
)


Did anybody else notice this issue?