Curious you are using"jdbc:sqlserve" to connect oracle, why?
Also kindly reminder scrubbing your user id password.
Sent from my iPhone
> On Jan 16, 2018, at 03:00, Onur EKİNCİ <oeki...@innova.com.tr> wrote:
>
> Hi,
>
> We are trying to get data from an Oracle database into Kinetica database
> through Apache Spark.
>
> We installed Spark in standalone mode. We executed the following commands.
> However, we have tried everything but we couldnt manage to run jobs in
> parallel. We use 2 IBM servers each of which has 128cores and 1TB memory.
>
> We also added in the spark-defaults.conf :
> spark.executor.memory=64g
> spark.executor.cores=32
> spark.default.parallelism=32
> spark.cores.max=64
> spark.scheduler.mode=FAIR
> spark.sql.shuffle.partions=32
>
>
> On the machine: 10.20.10.228
> ./start-master.sh --webui-port 8585
>
> ./start-slave.sh --webui-port 8586 spark://10.20.10.228:7077
>
>
> On the machine 10.20.10.229:
> ./start-slave.sh --webui-port 8586 spark://10.20.10.228:7077
>
>
> On the machine: 10.20.10.228:
>
> We start the Spark shell:
>
> spark-shell --master spark://10.20.10.228:7077
>
> Then we make configurations:
>
> val df = spark.read.format("jdbc").option("url",
> "jdbc:sqlserver://10.20.10.148:1433;databaseName=testdb").option("dbtable",
> "dbo.temp_muh_hareket").option("user", "gpudb").option("password",
> "Kinetica2017!").load()
> import com.kinetica.spark._
> val lp = new LoaderParams("http://10.20.10.228:9191";,
> "jdbc:simba://10.20.10.228:9292;ParentSet=MASTER", "muh_hareket_20",
> false,"",100000,true,true,"admin","Kinetica2017!",4, true, true, 1)
> SparkKineticaLoader.KineticaWriter(df,lp);
>
>
> The above commands successfully work. The data transfer completes. However,
> jobs work serially not in parallel. Also executors work serially and take
> turns. They donw work in parallel.
>
> How can we make jobs work in parallel?
>
>
> <image002.jpg>
>
> <image006.jpg>
>
> <image008.jpg>
> <image012.jpg>
>
> <image013.jpg>
>
>
>
>
>
>
> <image014.jpg>
>
>
> I really appreciate your help. We have done everything that we could.
>
>
> Onur EKİNCİ
> Bilgi Yönetimi Yöneticisi
> Knowledge Management Manager
>
> m:+90 553 044 2341 d:+90 212 329 7000
>
> İTÜ Ayazağa Kampüsü, Teknokent ARI4 Binası 34469 Maslak İstanbul - Google Maps
> <imza_2d4cbd2e-9f86-452e-8fa7-5851198cb9af.png>
>
>
>
> Yasal Uyarı :
> Bu elektronik posta işbu linki kullanarak ulaşabileceğiniz Koşul ve Şartlar
> dokümanına tabidir :
> http://www.innova.com.tr/disclaimer-yasal-uyari.asp
