Re: [Building] Building with JDK11

[Sean Owen]

Why do you need Java 11 bytecode though?
Java 8 bytecode runs fine on Java 11. The settings in the build are really
there for testing, not because it's required to use Java 11.

On Mon, Jul 18, 2022 at 10:29 PM Gera Shegalov  wrote:

> Bytecode version is controlled by javac "-target" option for Java, and by
> scalac "-target:" for Scala
> JDK can cross-compile between known versions.
>
> Spark uses 1.8 as source and target by default controlled by the Maven
> property java.version
> .
> But it's also hard-coded with -target:jvm-1.8
> 
> for Scala. Higher JDK versions can run lower version bytecode.
>
> if you want to try 11, replace occurences -target:jvm-1.8 by
> --target:jvm-${java.version} in pom.xml you should be able to produce 11
> bytecode by adding -Djava.version=11 to your Maven build command.
>
> ./build/mvn -Djava.version=11  ...
>
> However, I did not try beyond a quick compile on core and cannot say
> anything about fallout implications at run time.
>

Re: [Building] Building with JDK11

[Gera Shegalov]

Bytecode version is controlled by javac "-target" option for Java, and by
scalac "-target:" for Scala
JDK can cross-compile between known versions.

Spark uses 1.8 as source and target by default controlled by the Maven
property java.version
.
But it's also hard-coded with -target:jvm-1.8

for Scala. Higher JDK versions can run lower version bytecode.

if you want to try 11, replace occurences -target:jvm-1.8 by
--target:jvm-${java.version} in pom.xml you should be able to produce 11
bytecode by adding -Djava.version=11 to your Maven build command.

./build/mvn -Djava.version=11  ...

However, I did not try beyond a quick compile on core and cannot say
anything about fallout implications at run time.

On Mon, Jul 18, 2022 at 3:49 PM Szymon Kuryło 
wrote:

> Ok, so I used a docker image `adoptopenjdk/openjdk11:latest` as a builder,
> and still got version 52 classes.
>
> pon., 18 lip 2022 o 09:51 Stephen Coy  napisał(a):
>
>> Hi Sergey,
>>
>> I’m willing to be corrected, but I think the use of a JAVA_HOME
>> environment variable was something that was started by and continues to be
>> perpetuated by Apache Tomcat.
>>
>> … or maybe Apache Ant, but modern versions of Ant do not need it either.
>>
>> It is not needed for modern releases of Apache Maven.
>>
>> Cheers,
>>
>> Steve C
>>
>> On 18 Jul 2022, at 4:12 pm, Sergey B.  wrote:
>>
>> Hi Steve,
>>
>> Can you shed some light why do they need $JAVA_HOME at all if
>> everything is already in place?
>>
>> Regards,
>> - Sergey
>>
>> On Mon, Jul 18, 2022 at 4:31 AM Stephen Coy <
>> s...@infomedia.com.au.invalid> wrote:
>>
>>> Hi Szymon,
>>>
>>> There seems to be a common misconception that setting JAVA_HOME will set
>>> the version of Java that is used.
>>>
>>> This is not true, because in most environments you need to have a PATH
>>> environment variable set up that points at the version of Java that you
>>> want to use.
>>>
>>> You can set JAVA_HOME to anything at all and `java -version` will always
>>> return the same result.
>>>
>>> The way that you configure PATH varies from OS to OS:
>>>
>>>
>>>- MacOS use `/usr/libexec/java_home -v11`
>>>- On linux use `sudo alternatives --config java`
>>>- On Windows I have no idea
>>>
>>>
>>> When you do this the `mvn` command will compute the value of JAVA_HOME
>>> for its own use; there is no need to explicitly set it yourself (unless
>>> other Java applications that you use insist u[on it).
>>>
>>>
>>> Cheers,
>>>
>>> Steve C
>>>
>>> On 16 Jul 2022, at 7:24 am, Szymon Kuryło 
>>> wrote:
>>>
>>> Hello,
>>>
>>> I'm trying to build a Java 11 Spark distro using the
>>> dev/make-distribution.sh script.
>>> I have set JAVA_HOME to point to JDK11 location, I've also set the
>>> java.version property in pom.xml to 11, effectively also setting
>>> `maven.compile.source` and `maven.compile.target`.
>>> When inspecting classes from the `dist` directory with `javap -v`, I
>>> find that the class major version is 52, which is specific to JDK8. Am
>>> I missing something? Is there a reliable way to set the JDK used in the
>>> build process?
>>>
>>> Thanks,
>>> Szymon K.
>>>
>>>
>>> This email contains confidential information of and is the copyright of
>>> Infomedia. It must not be forwarded, amended or disclosed without consent
>>> of the sender. If you received this message by mistake, please advise the
>>> sender and delete all copies. Security of transmission on the internet
>>> cannot be guaranteed, could be infected, intercepted, or corrupted and you
>>> should ensure you have suitable antivirus protection in place. By sending
>>> us your or any third party personal details, you consent to (or confirm you
>>> have obtained consent from such third parties) to Infomedia’s privacy
>>> policy. http://www.infomedia.com.au/privacy-policy/
>>>
>>
>>

Re: [Building] Building with JDK11

[Szymon Kuryło]

Ok, so I used a docker image `adoptopenjdk/openjdk11:latest` as a builder,
and still got version 52 classes.

pon., 18 lip 2022 o 09:51 Stephen Coy  napisał(a):

> Hi Sergey,
>
> I’m willing to be corrected, but I think the use of a JAVA_HOME
> environment variable was something that was started by and continues to be
> perpetuated by Apache Tomcat.
>
> … or maybe Apache Ant, but modern versions of Ant do not need it either.
>
> It is not needed for modern releases of Apache Maven.
>
> Cheers,
>
> Steve C
>
> On 18 Jul 2022, at 4:12 pm, Sergey B.  wrote:
>
> Hi Steve,
>
> Can you shed some light why do they need $JAVA_HOME at all if
> everything is already in place?
>
> Regards,
> - Sergey
>
> On Mon, Jul 18, 2022 at 4:31 AM Stephen Coy 
> wrote:
>
>> Hi Szymon,
>>
>> There seems to be a common misconception that setting JAVA_HOME will set
>> the version of Java that is used.
>>
>> This is not true, because in most environments you need to have a PATH
>> environment variable set up that points at the version of Java that you
>> want to use.
>>
>> You can set JAVA_HOME to anything at all and `java -version` will always
>> return the same result.
>>
>> The way that you configure PATH varies from OS to OS:
>>
>>
>>- MacOS use `/usr/libexec/java_home -v11`
>>- On linux use `sudo alternatives --config java`
>>- On Windows I have no idea
>>
>>
>> When you do this the `mvn` command will compute the value of JAVA_HOME
>> for its own use; there is no need to explicitly set it yourself (unless
>> other Java applications that you use insist u[on it).
>>
>>
>> Cheers,
>>
>> Steve C
>>
>> On 16 Jul 2022, at 7:24 am, Szymon Kuryło  wrote:
>>
>> Hello,
>>
>> I'm trying to build a Java 11 Spark distro using the
>> dev/make-distribution.sh script.
>> I have set JAVA_HOME to point to JDK11 location, I've also set the
>> java.version property in pom.xml to 11, effectively also setting
>> `maven.compile.source` and `maven.compile.target`.
>> When inspecting classes from the `dist` directory with `javap -v`, I find
>> that the class major version is 52, which is specific to JDK8. Am I missing
>> something? Is there a reliable way to set the JDK used in the build process?
>>
>> Thanks,
>> Szymon K.
>>
>>
>> This email contains confidential information of and is the copyright of
>> Infomedia. It must not be forwarded, amended or disclosed without consent
>> of the sender. If you received this message by mistake, please advise the
>> sender and delete all copies. Security of transmission on the internet
>> cannot be guaranteed, could be infected, intercepted, or corrupted and you
>> should ensure you have suitable antivirus protection in place. By sending
>> us your or any third party personal details, you consent to (or confirm you
>> have obtained consent from such third parties) to Infomedia’s privacy
>> policy. http://www.infomedia.com.au/privacy-policy/
>>
>
>

Re: very simple UI on webpage to display x/y plots+histogram of data stored in hive

[ayan guha]

If possible, start with a Jupyter or databricks notebook

On Tue, 19 Jul 2022 at 7:35 am, Joris Billen 
wrote:

> Thank you - looks like it COULD do it.
> Have to try if I can have a simple UI, user selects one out of 100
> options, and receives the correct x/y plot and correct histogram of data
> stored in hive and retrieved with spark into pandas…
>
> Many thanks for your suggestion!
>
>
> On 18 Jul 2022, at 15:08, Sean Owen  wrote:
>
> You pull your data via Spark to a pandas DF and do whatever you want
>
>
> --
Best Regards,
Ayan Guha

Re: very simple UI on webpage to display x/y plots+histogram of data stored in hive

[Joris Billen]

Thank you - looks like it COULD do it.
Have to try if I can have a simple UI, user selects one out of 100 options, and 
receives the correct x/y plot and correct histogram of data stored in hive and 
retrieved with spark into pandas…

Many thanks for your suggestion!

On 18 Jul 2022, at 15:08, Sean Owen mailto:sro...@gmail.com>> 
wrote:

You pull your data via Spark to a pandas DF and do whatever you want

Re: very simple UI on webpage to display x/y plots+histogram of data stored in hive

[Sean Owen]

Sure, look at any python-based plotting package. plot.ly does this nicely.
You pull your data via Spark to a pandas DF and do whatever you want.

On Mon, Jul 18, 2022 at 1:42 PM Joris Billen 
wrote:

> Hi,
> I am making a very short demo and would like to make the most rudimentary
> UI (withouth knowing anything about front end) that would show a x/y plot
> of data stored in HIVE (that I typically query with spark) together with a
> histogram (something one would typically created in a jupyter notebook).
> Without being a front-end developer, does something exists to publish
> something on a html page, where the user selects something (for instance
> out of a list a certain date), and then in the browser, the store x/y plot
> based on hive tables and a histogram is returned?
>
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>

very simple UI on webpage to display x/y plots+histogram of data stored in hive

[Joris Billen]

Hi, 
I am making a very short demo and would like to make the most rudimentary UI 
(withouth knowing anything about front end) that would show a x/y plot of data 
stored in HIVE (that I typically query with spark) together with a histogram 
(something one would typically created in a jupyter notebook).
Without being a front-end developer, does something exists to publish something 
on a html page, where the user selects something (for instance out of a list a 
certain date), and then in the browser, the store x/y plot based on hive tables 
and a histogram is returned?


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Issue while building spark project

[Sean Owen]

Increase the stack size for the JVM when Maven / SBT run. The build sets
this but you may still need something like "-Xss4m" in your MAVEN_OPTS

On Mon, Jul 18, 2022 at 11:18 AM rajat kumar 
wrote:

> Hello ,
>
> Can anyone pls help me in below error. It is a maven project. It is coming
> while building it
>
> [ERROR] error: java.lang.StackOverflowError
> [INFO] at
> scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4885)
>

Issue while building spark project

[rajat kumar]

Hello ,

Can anyone pls help me in below error. It is a maven project. It is coming
while building it

[ERROR] error: java.lang.StackOverflowError
[INFO] at
scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4885)

Re: [Building] Building with JDK11

[Stephen Coy]

Hi Sergey,

I’m willing to be corrected, but I think the use of a JAVA_HOME environment 
variable was something that was started by and continues to be perpetuated by 
Apache Tomcat.

… or maybe Apache Ant, but modern versions of Ant do not need it either.

It is not needed for modern releases of Apache Maven.

Cheers,

Steve C

On 18 Jul 2022, at 4:12 pm, Sergey B. 
mailto:sergey.bushma...@gmail.com>> wrote:

Hi Steve,

Can you shed some light why do they need $JAVA_HOME at all if everything is 
already in place?

Regards,
- Sergey

On Mon, Jul 18, 2022 at 4:31 AM Stephen Coy 
mailto:s...@infomedia.com.au.invalid>> wrote:
Hi Szymon,

There seems to be a common misconception that setting JAVA_HOME will set the 
version of Java that is used.

This is not true, because in most environments you need to have a PATH 
environment variable set up that points at the version of Java that you want to 
use.

You can set JAVA_HOME to anything at all and `java -version` will always return 
the same result.

The way that you configure PATH varies from OS to OS:


  *   MacOS use `/usr/libexec/java_home -v11`
  *   On linux use `sudo alternatives --config java`
  *   On Windows I have no idea

When you do this the `mvn` command will compute the value of JAVA_HOME for its 
own use; there is no need to explicitly set it yourself (unless other Java 
applications that you use insist u[on it).


Cheers,

Steve C

On 16 Jul 2022, at 7:24 am, Szymon Kuryło 
mailto:szymonkur...@gmail.com>> wrote:

Hello,

I'm trying to build a Java 11 Spark distro using the dev/make-distribution.sh 
script.
I have set JAVA_HOME to point to JDK11 location, I've also set the java.version 
property in pom.xml to 11, effectively also setting `maven.compile.source` and 
`maven.compile.target`.
When inspecting classes from the `dist` directory with `javap -v`, I find that 
the class major version is 52, which is specific to JDK8. Am I missing 
something? Is there a reliable way to set the JDK used in the build process?

Thanks,
Szymon K.

This email contains confidential information of and is the copyright of 
Infomedia. It must not be forwarded, amended or disclosed without consent of 
the sender. If you received this message by mistake, please advise the sender 
and delete all copies. Security of transmission on the internet cannot be 
guaranteed, could be infected, intercepted, or corrupted and you should ensure 
you have suitable antivirus protection in place. By sending us your or any 
third party personal details, you consent to (or confirm you have obtained 
consent from such third parties) to Infomedia’s privacy policy. 
http://www.infomedia.com.au/privacy-policy/

Re: [Building] Building with JDK11

[Sergey B.]

Hi Steve,

Can you shed some light why do they need $JAVA_HOME at all if everything is
already in place?

Regards,
- Sergey

On Mon, Jul 18, 2022 at 4:31 AM Stephen Coy 
wrote:

> Hi Szymon,
>
> There seems to be a common misconception that setting JAVA_HOME will set
> the version of Java that is used.
>
> This is not true, because in most environments you need to have a PATH
> environment variable set up that points at the version of Java that you
> want to use.
>
> You can set JAVA_HOME to anything at all and `java -version` will always
> return the same result.
>
> The way that you configure PATH varies from OS to OS:
>
>
>- MacOS use `/usr/libexec/java_home -v11`
>- On linux use `sudo alternatives --config java`
>- On Windows I have no idea
>
>
> When you do this the `mvn` command will compute the value of JAVA_HOME for
> its own use; there is no need to explicitly set it yourself (unless other
> Java applications that you use insist u[on it).
>
>
> Cheers,
>
> Steve C
>
> On 16 Jul 2022, at 7:24 am, Szymon Kuryło  wrote:
>
> Hello,
>
> I'm trying to build a Java 11 Spark distro using the
> dev/make-distribution.sh script.
> I have set JAVA_HOME to point to JDK11 location, I've also set the
> java.version property in pom.xml to 11, effectively also setting
> `maven.compile.source` and `maven.compile.target`.
> When inspecting classes from the `dist` directory with `javap -v`, I find
> that the class major version is 52, which is specific to JDK8. Am I missing
> something? Is there a reliable way to set the JDK used in the build process?
>
> Thanks,
> Szymon K.
>
>
> This email contains confidential information of and is the copyright of
> Infomedia. It must not be forwarded, amended or disclosed without consent
> of the sender. If you received this message by mistake, please advise the
> sender and delete all copies. Security of transmission on the internet
> cannot be guaranteed, could be infected, intercepted, or corrupted and you
> should ensure you have suitable antivirus protection in place. By sending
> us your or any third party personal details, you consent to (or confirm you
> have obtained consent from such third parties) to Infomedia’s privacy
> policy. http://www.infomedia.com.au/privacy-policy/
>