Re: Is there "EXCEPT ALL" in Spark SQL?

[hareesh makam]

There is Except in DataFrame API.

df1.except(df2)

Same can be used in SQL as well.

public DataFrame

except(DataFrame

other)

Returns a new DataFrame

containing
rows in this frame but not in another frame. This is equivalent to EXCEPT in
SQL.


-Hareesh


On 6 July 2017 at 12:22, jeff saremi  wrote:

> I tried this query in 1.6 and it failed:
>
> SELECT * FROM Table1 EXCEPT ALL SELECT * FROM Table2
>
>
> Exception in thread "main" java.lang.RuntimeException: [1.32] failure:
> ``('' expected but `all' found
>
>
> thanks
>
> Jeff
>

Re: Spark streaming app starts processing when kill that app

[hareesh makam]

If you are running your master on a single core, it might be an issue of
Starvation.
assuming you are running it locally, try setting master to local[2] or
higher.

Check the first example at
https://spark.apache.org/docs/latest/streaming-programming-guide.html

- Hareesh

On 3 May 2016 at 12:35, Shams ul Haque  wrote:

> Hi all,
>
> I am facing strange issue when running Spark Streaming app.
>
> What i was doing is, When i submit my app by *spark-submit *it works fine
> and also visible in Spark UI. But it doesn't process any data coming from
> kafka. And when i kill that app by pressing Ctrl + C on terminal, then it
> start processing all data received from Kafka and then get shutdown.
>
> I am trying to figure out why is this happening. Please help me if you
> know anything.
>
> Thanks and regards
> Shams ul Haque
>

Re: Unable to compile from source

[hareesh makam]

Are you behind a proxy?

Or

Try disabling the SSL check while building.

http://stackoverflow.com/questions/21252800/maven-trusting-all-certs-unlimited-java-policy

Check above link to know how to disable SSL check.

- hareesh.
On Jan 8, 2016 4:54 PM, "Gaini Rajeshwar" 
wrote:

> Hi All,
>
> I am new to apache spark.
>
> I have downloaded *Spark 1.6.0 (Jan 04 2016) source code version*.
>
> I did run the following command following command as per spark
> documentation .
>
> build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package
>
> When run above command, i am getting the following error
>
> [ERROR] Failed to execute goal on project spark-catalyst_2.10: Could not 
> resolve dependencies for project 
> org.apache.spark:spark-catalyst_2.10:jar:1.6.0: Failed to collect 
> dependencies at org.codehaus.janino:janino:jar:2.7.8: Failed to read artifact 
> descriptor for org.codehaus.janino:janino:jar:2.7.8: Could not transfer 
> artifact org.codehaus.janino:janino:pom:2.7.8 from/to central 
> (https://repo1.maven.org/maven2): Remote host closed connection during 
> handshake: SSL peer shut down incorrectly -> [Help 1]
>
> Can anyone help with this ?
>
>
>