[GitHub] [incubator-seatunnel] zuoliansheng opened a new issue #1099: 使用seatunnel完成从mysql到hive抽数，部分时间字段为NULL

Wed, 19 Jan 2022 00:43:27 -0800 


zuoliansheng opened a new issue #1099:
URL: https://github.com/apache/incubator-seatunnel/issues/1099


   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22)
 and found no similar issues.
   
   
   ### What happened
   
   使用seatunnel完成从mysql到hive抽数，部分时间字段为NULL
   
   ### SeaTunnel Version
   
   1.5.4-2.11.8
   
   ### SeaTunnel Config
   
   ```conf
   spark {
       spark.app.name = ${hive_table}"_wd"
       spark.default.parallelism = 4
       spark.executor.instances = 2
       spark.executor.cores = 1
       spark.executor.memory = "2g"
       spark.sql.catalogImplementation = "hive"
   }
   input {
        mysql{
           url = "jdbc:mysql://"${alarm_addr}"/"alarm
           table = "(SELECT 
`id`,`bike_id`,`bike_sn`,`bike_version`,`type`,`content`,`start_time`,`times`,`end_time`,`status`,`duration`,`finish_type`,`create_time`,`update_time`,`city_id`,`data`,`is_test`
  from "${table}" where  ((update_time >= '"${ds}"' and update_time 
<'"${tomorrow_half_hour_ds}"') or (create_time >= '"${ds}"' and create_time 
<'"${tomorrow_ds}"')) ) as A"
           result_table_name = ${table}"_log"
           user = ${alarm_user}
           password = ${alarm_passwd}
           jdbc.characterEncoding = "UTF-8"
           jdbc.defaultFetchSize = 2000
           jdbc.useCursorFetch = true
           jdbc.zeroDateTimeBehavior="convertToNull"
           jdbc.tinyInt1isBit = false
           jdbc.dontTrackOpenResources = true
      }
   }
   filter {
        org.interestinglab.waterdrop.filter.ReplaceColumns {
           pattern = "[\\n\\r]"
           replacement = ""
           result_table_name = ${table}"_tmp"
       }
   
   }
   output {
       Hive {
           sql = "insert overwrite table "${hive_database}"."${hive_table}" 
partition(event_day='"${ds_nodash}"') select * from "${table}"_tmp"
       }
   }
   ```
   
   
   ### Running Command
   
   ```shell
   /data/apps/waterdrop/bin/start-waterdrop.sh --master yarn --deploy-mode 
client --config ods_ebike_op_alarm_di.conf
   ```
   
   
   ### Error Exception
   
   ```log
   无报错
   ```
   
   
   ### Flink or Spark Version
   
   spark 2.3.0
   hive 3.1.0
   
   
   
   
   ### Java or Scala Version
   
   jdk1.8
   
   ### Screenshots
   
   mysql中的数据截图：
   
![image](https://user-images.githubusercontent.com/36074095/150093424-fbd32ddd-cd94-415c-930d-0386b75474d2.png)
   hive中的数据截图：
   
![image](https://user-images.githubusercontent.com/36074095/150093926-b4f6f6a4-a189-4e04-8ff1-58012d8b7170.png)
   
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to