subject:"\[jira\] \[Commented\] \(SPARK\-25523\) Multi thread execute sparkSession.read\(\).jdbc\(url, table, properties\) problem"

[jira] [Commented] (SPARK-25523) Multi thread execute sparkSession.read().jdbc(url, table, properties) problem

2018-09-27 Thread Hyukjin Kwon (JIRA)



[ 
https://issues.apache.org/jira/browse/SPARK-25523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16630018#comment-16630018
 ] 

Hyukjin Kwon commented on SPARK-25523:
--

Can you narrow down the problem? Looks hard to follow. It's even on Windows. 
Arguably there are less Windows users. If that's unable to narrow down the 
problem, I would just leave this {{Cannot Produce}} for now.

> Multi thread execute sparkSession.read().jdbc(url, table, properties) problem
> -
>
> Key: SPARK-25523
> URL: https://issues.apache.org/jira/browse/SPARK-25523
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.0
> Environment: h3. [IntelliJ 
> _IDEA_|http://www.baidu.com/link?url=7ZLtsOfyqR1YxLqcTU0Q-hqXWV_PsY6IzIzZoKhiXZZ4AcLrpQ4DoTG30yIN-Gs8]
>  
> local mode
>  
>Reporter: huanghuai
>Priority: Major
>
> public static void test2() throws Exception{
>  String ckUrlPrefix="jdbc:clickhouse://";
>  String quote = "`";
>  JdbcDialects.registerDialect(new JdbcDialect() {
>  @Override
>  public boolean canHandle(String url)
> { return url.startsWith(ckUrlPrefix); }
> @Override
>  public String quoteIdentifier(String colName)
> { return quote + colName + quote; }
> });
> SparkSession spark = initSpark();
>  String ckUrl = "jdbc:clickhouse://192.168.2.148:8123/default";
>  Properties ckProp = new Properties();
>  ckProp.put("user", "default");
>  ckProp.put("password", "");
> String prestoUrl = "jdbc:presto://192.168.2.148:9002/mysql-xxx/xxx";
>  Properties prestoUrlProp = new Properties();
>  prestoUrlProp.put("user", "root");
>  prestoUrlProp.put("password", "");
> // new Thread(()->{
> // spark.read()
> // .jdbc(ckUrl, "ontime", ckProp).show();
> // }).start();
> System.out.println("--");
> new Thread(()->{
> spark.read()
> .jdbc(prestoUrl, "tx_user", prestoUrlProp).show();
> }).start();
> System.out.println("--");
> new Thread(()->{
> Dataset load = spark.read()
> .format("com.vertica.spark.datasource.DefaultSource")
> .option("host", "192.168.1.102")
> .option("port", 5433)
> .option("user", "dbadmin")
> .option("password", "manager")
> .option("db", "test")
> .option("dbschema", "public")
> .option("table", "customers")
> .load();
> load.printSchema();
> load.show();
> }).start();
>  System.out.println("--");
>  }
> public static SparkSession initSpark() throws Exception
> { return SparkSession.builder() .master("spark://dsjkfb1:7077") 
> //spark://dsjkfb1:7077 .appName("Test") .config("spark.executor.instances",3) 
> .config("spark.executor.cores",2) .config("spark.cores.max",6) 
> //.config("spark.default.parallelism",1) 
> .config("spark.submit.deployMode","client") 
> .config("spark.driver.memory","2G") .config("spark.executor.memory","3G") 
> .config("spark.driver.maxResultSize", "2G") .config("spark.local.dir", 
> "d:\\tmp") .config("spark.driver.host", "192.168.2.148") 
> .config("spark.scheduler.mode", "FAIR") .config("spark.jars", 
> "F:\\project\\xxx\\vertica-jdbc-7.0.1-0.jar," + 
> "F:\\project\\xxx\\clickhouse-jdbc-0.1.40.jar," + 
> "F:\\project\\xxx\\vertica-spark-connector-9.1-2.1.jar," + 
> "F:\\project\\xxx\\presto-jdbc-0.189-mining.jar")  .getOrCreate(); }
>  
>  
> {color:#ff}*  The above is code 
> --*{color}
> {color:#ff}*question: If i open vertica jdbc , thread will pending 
> forever.*{color}
> {color:#ff}*And driver loging like this:*{color}
>  
> 2018-09-26 10:32:51 INFO SharedState:54 - Setting 
> hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir 
> ('file:/C:/Users/admin/Desktop/test-project/sparktest/spark-warehouse/').
>  2018-09-26 10:32:51 INFO SharedState:54 - Warehouse path is 
> 'file:/C:/Users/admin/Desktop/test-project/sparktest/spark-warehouse/'.
>  2018-09-26 10:32:51 INFO ContextHandler:781 - Started 
> o.s.j.s.ServletContextHandler@2f70d6e2\{/SQL,null,AVAILABLE,@Spark}
>  2018-09-26 10:32:51 INFO ContextHandler:781 - Started 
> o.s.j.s.ServletContextHandler@1d66833d\{/SQL/json,null,AVAILABLE,@Spark}
>  2018-09-26 10:32:51 INFO ContextHandler:781 - Started 
> o.s.j.s.ServletContextHandler@65af6f3a\{/SQL/execution,null,AVAILABLE,@Spark}
>  2018-09-26 10:32:51 INFO ContextHandler:781 - Started 
> o.s.j.s.ServletContextHandler@55012968\{/SQL/execution/json,null,AVAILABLE,@Spark}
>  2018-09-26 10:32:51 INFO ContextHandler:781 - Started 
> o.s.j.s.ServletContextHandler@59e3f5aa\{/static/sql,null,AVAILABLE,@Spark}
>  2018-09-26 10:32:52 INFO StateStoreCoordinatorRef:54 - Registered 
> StateStoreCoordinator endpoint
>  2018-09-26 10:32:52 INFO CoarseGrainedSchedulerBa

[jira] [Commented] (SPARK-25523) Multi thread execute sparkSession.read().jdbc(url, table, properties) problem

2018-09-27 Thread huanghuai (JIRA)



[ 
https://issues.apache.org/jira/browse/SPARK-25523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16630421#comment-16630421
 ] 

huanghuai commented on SPARK-25523:
---

Sorry, I know this question is hard to understand, but my project is right on 
windows 10 , and when a http request coming, it will create a sparkSession and 
run  a spark application, I am also helpless.

 

I will try my best to solve this problem , If it's not solved in a few days 
later, I will close this issue. I know ,this problem is hard to you to 
reproduce.

> Multi thread execute sparkSession.read().jdbc(url, table, properties) problem
> -
>
> Key: SPARK-25523
> URL: https://issues.apache.org/jira/browse/SPARK-25523
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.0
> Environment: h3. [IntelliJ 
> _IDEA_|http://www.baidu.com/link?url=7ZLtsOfyqR1YxLqcTU0Q-hqXWV_PsY6IzIzZoKhiXZZ4AcLrpQ4DoTG30yIN-Gs8]
>  
> local mode
>  
>Reporter: huanghuai
>Priority: Major
>
> public static void test2() throws Exception{
>  String ckUrlPrefix="jdbc:clickhouse://";
>  String quote = "`";
>  JdbcDialects.registerDialect(new JdbcDialect() {
>  @Override
>  public boolean canHandle(String url)
> { return url.startsWith(ckUrlPrefix); }
> @Override
>  public String quoteIdentifier(String colName)
> { return quote + colName + quote; }
> });
> SparkSession spark = initSpark();
>  String ckUrl = "jdbc:clickhouse://192.168.2.148:8123/default";
>  Properties ckProp = new Properties();
>  ckProp.put("user", "default");
>  ckProp.put("password", "");
> String prestoUrl = "jdbc:presto://192.168.2.148:9002/mysql-xxx/xxx";
>  Properties prestoUrlProp = new Properties();
>  prestoUrlProp.put("user", "root");
>  prestoUrlProp.put("password", "");
> // new Thread(()->{
> // spark.read()
> // .jdbc(ckUrl, "ontime", ckProp).show();
> // }).start();
> System.out.println("--");
> new Thread(()->{
> spark.read()
> .jdbc(prestoUrl, "tx_user", prestoUrlProp).show();
> }).start();
> System.out.println("--");
> new Thread(()->{
> Dataset load = spark.read()
> .format("com.vertica.spark.datasource.DefaultSource")
> .option("host", "192.168.1.102")
> .option("port", 5433)
> .option("user", "dbadmin")
> .option("password", "manager")
> .option("db", "test")
> .option("dbschema", "public")
> .option("table", "customers")
> .load();
> load.printSchema();
> load.show();
> }).start();
>  System.out.println("--");
>  }
> public static SparkSession initSpark() throws Exception
> { return SparkSession.builder() .master("spark://dsjkfb1:7077") 
> //spark://dsjkfb1:7077 .appName("Test") .config("spark.executor.instances",3) 
> .config("spark.executor.cores",2) .config("spark.cores.max",6) 
> //.config("spark.default.parallelism",1) 
> .config("spark.submit.deployMode","client") 
> .config("spark.driver.memory","2G") .config("spark.executor.memory","3G") 
> .config("spark.driver.maxResultSize", "2G") .config("spark.local.dir", 
> "d:\\tmp") .config("spark.driver.host", "192.168.2.148") 
> .config("spark.scheduler.mode", "FAIR") .config("spark.jars", 
> "F:\\project\\xxx\\vertica-jdbc-7.0.1-0.jar," + 
> "F:\\project\\xxx\\clickhouse-jdbc-0.1.40.jar," + 
> "F:\\project\\xxx\\vertica-spark-connector-9.1-2.1.jar," + 
> "F:\\project\\xxx\\presto-jdbc-0.189-mining.jar")  .getOrCreate(); }
>  
>  
> {color:#ff}*  The above is code 
> --*{color}
> {color:#ff}*question: If i open vertica jdbc , thread will pending 
> forever.*{color}
> {color:#ff}*And driver loging like this:*{color}
>  
> 2018-09-26 10:32:51 INFO SharedState:54 - Setting 
> hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir 
> ('file:/C:/Users/admin/Desktop/test-project/sparktest/spark-warehouse/').
>  2018-09-26 10:32:51 INFO SharedState:54 - Warehouse path is 
> 'file:/C:/Users/admin/Desktop/test-project/sparktest/spark-warehouse/'.
>  2018-09-26 10:32:51 INFO ContextHandler:781 - Started 
> o.s.j.s.ServletContextHandler@2f70d6e2\{/SQL,null,AVAILABLE,@Spark}
>  2018-09-26 10:32:51 INFO ContextHandler:781 - Started 
> o.s.j.s.ServletContextHandler@1d66833d\{/SQL/json,null,AVAILABLE,@Spark}
>  2018-09-26 10:32:51 INFO ContextHandler:781 - Started 
> o.s.j.s.ServletContextHandler@65af6f3a\{/SQL/execution,null,AVAILABLE,@Spark}
>  2018-09-26 10:32:51 INFO ContextHandler:781 - Started 
> o.s.j.s.ServletContextHandler@55012968\{/SQL/execution/json,null,AVAILABLE,@Spark}
>  2018-09-26 10:32:51 INFO ContextHandler:781 - Started 
> o.s.j.s.ServletContextHandler@59e3f5aa\{/static/sql,null,AVAILABLE,@Spark}
>  2018

[jira] [Commented] (SPARK-25523) Multi thread execute sparkSession.read().jdbc(url, table, properties) problem

[jira] [Commented] (SPARK-25523) Multi thread execute sparkSession.read().jdbc(url, table, properties) problem

2 matches

Site Navigation

Mail list logo

Footer information