Re: BUG: take with SparkSession.master[url]

I have configured in IntelliJ as external jars spark-3.0.0-preview2-bin-hadoop2.7/jar not pulling anything from maven. Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org On Fri, 27 Mar 2020 at 05:45, Wenchen Fan wrote: > Which

Re: BUG: take with SparkSession.master[url]

Which Spark/Scala version do you use? On Fri, Mar 27, 2020 at 1:24 PM Zahid Rahman wrote: > > with the following sparksession configuration > > val spark = SparkSession.builder().master("local[*]").appName("Spark Session > take").getOrCreate(); > > this line works > > flights.filter(flight_row

BUG: take with SparkSession.master[url]

with the following sparksession configuration val spark = SparkSession.builder().master("local[*]").appName("Spark Session take").getOrCreate(); this line works flights.filter(flight_row => flight_row.ORIGIN_COUNTRY_NAME != "Canada").map(flight_row => flight_row).take(5) however if change the

Unsubscribe

On Thu, Mar 26, 2020, 8:35 PM Andrew Melo wrote: > Hello all, > > Is there a way to register classes within a datasourcev2 implementation in > the Kryo serializer? > > I've attempted the following in both the constructor and static block of > my toplevel class: > > SparkContext context =

Supporting Kryo registration in DSv2

Hello all, Is there a way to register classes within a datasourcev2 implementation in the Kryo serializer? I've attempted the following in both the constructor and static block of my toplevel class: SparkContext context = SparkContext.getOrCreate(); SparkConf conf =

Re: can we all help use our expertise to create an IT solution for Covid-19

Hi all, Many thanks for your contribution. A number of colleagues have proposed existing sites devoted to analyzing this virus mainly in US and North America. Also donating computers etc. although worthy gestures will not be enough. What I had in mind (and apologies if this looks pedantic) is

Re: can we all help use our expertise to create an IT solution for Covid-19

Not sure if this helps, but there is a project where you can donate spare compute resources https://foldingathome.org/start-folding/ Maybe if labs need them we can donate cloud credits as well? On Thu, Mar 26, 2020 at 1:08 PM Denny Lee wrote: > There are a number of really good datasets

Re: can we all help use our expertise to create an IT solution for Covid-19

There are a number of really good datasets already available including (but not limited to): - South Korea COVID-19 Dataset - 2019 Novel Coronavirus COVID-19 (2019-nCoV) Data Repository by Johns Hopkins CSSE

Re: results of taken(3) not appearing in console window

Thanks. I added that as well. I also needed to add a hard coded import previously spark.implicits._ val flightDf = spark.read.parquet("/data/flight-data/parquet/2010-summary.parquet/") flights.show(3) +-+---+-+ |DEST_COUNTRY_NAME|ORIGIN_COUNTRY_NAME|count|

Re: results of taken(3) not appearing in console window

Thank you. Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org On Thu, 26 Mar 2020 at 19:18, Reynold Xin wrote: > bcc dev, +user > > You need to print out the result. Take itself doesn't print. You only got > the results printed to the

Re: can we all help use our expertise to create an IT solution for Covid-19

Actually I thought these sites exist look at John's hopkins and worldometers On Thu, Mar 26, 2020, 2:27 PM Zahid Rahman wrote: > > "We can then donate this to WHO or others and we can make it very modular > though microservices etc." > > I have no interest because there are 8 million muslims

Re: results of taken(3) not appearing in console window

bcc dev, +user You need to print out the result. Take itself doesn't print. You only got the results printed to the console because the Scala REPL automatically prints the returned value from take. On Thu, Mar 26, 2020 at 12:15 PM, Zahid Rahman < zahidr1...@gmail.com > wrote: > > I am

Re: can we all help use our expertise to create an IT solution for Covid-19

Thank you for your remarks. Points taken. Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com *Disclaimer:*

Re: can we all help use our expertise to create an IT solution for Covid-19

"We can then donate this to WHO or others and we can make it very modular though microservices etc." I have no interest because there are 8 million muslims locked up in their home for 8 months by the Hindutwa (Indians) You didn't take any notice of them. Now you are locked up in your home and you

Re: can we all help use our expertise to create an IT solution for Covid-19

Thanks but nobody claimed we can fix it. However, we can all contribute to it. When it utilizes the cloud then it become a global digitization issue. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: can we all help use our expertise to create an IT solution for Covid-19

People in tech should be more humble and admit this is not something they can fix. There's already plenty of visualizations, dashboards etc showing the spread of the virus. This is not even a big data problem, so Spark would have limited use. On Thu, Mar 26, 2020 at 10:37 AM Sol Rodriguez wrote:

Re: can we all help use our expertise to create an IT solution for Covid-19

IMO it's not about technology, it's about data... if we don't have access to the data there's no point throwing "microservices" and "kafka" at the problem. You might find that the most effective analysis might be delivered through an excel sheet ;) So before technology I'd suggest to get access to

Re: can we all help use our expertise to create an IT solution for Covid-19

Have you taken a look at this ( https://coronavirus.1point3acres.com/en/test )? They have a visualizer with a very basic analysis of the outbreak. On Thu, Mar 26, 2020 at 8:54 AM Mich Talebzadeh wrote: > Thanks. > > Agreed, computers are not the end but means to an end. We all have to > start

Re: can we all help use our expertise to create an IT solution for Covid-19

Thanks. Agreed, computers are not the end but means to an end. We all have to start from somewhere. It all helps. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: can we all help use our expertise to create an IT solution for Covid-19

Hello Mich, I will be more than happy to contribute to this. Thanks, Hemant On Thu, Mar 26, 2020 at 7:11 PM Mich Talebzadeh wrote: > Hi all, > > Do you think we can create a global solution in the cloud using > volunteers like us and third party employees. What I have in mind is to > create

can we all help use our expertise to create an IT solution for Covid-19

Hi all, Do you think we can create a global solution in the cloud using volunteers like us and third party employees. What I have in mind is to create a comprehensive real time solution to get data from various countries, universities pushed into a fast database through Kafka and Spark and used