Dynamic allocation not working

2019-07-08 Thread Amit Sharma
Hi All, i have set the dynamic allocation propertt = true in my script file and also shuffle property in script as well as on all worker nodes spark-env file. I am using spark kafka streaming. I checked that as request comes no of cores allocation increase but even after request is completed no of

Seeking help of UDF number-float converting

2019-07-08 Thread Danni Wu
Hello all: I am using UDF to convert schema to JSON, and based on the JSON schema, when a schema has key “type” is “number”, I need to convert the input data to float, such as if an “income” type is number, and the input data is “100”, the output should be “100.0”. But the problem is if an origi