Hi Prince,
Please share what errors you are facing while running Data quality checks
against Postgres . Also ensure you are able to connect to Postgres from your
docker and that you have access and appropriate username, password to connect
to Postgres DB.
Here is the sample Griffin DQ json that works for me.
{
"name": "<data_quality_check_name>",
"process.type": "batch",
"data.sources": [
{
"name": "source",
"baseline": true,
"connector": {
"type": "jdbc",
"config": {
"database": "<db_name>",
"tablename": "<table_name>",
"url": <postgress_url>,
"user": <user_name>,
"password": <user_password>,
"driver": <driver_name>,
"where": "<filter_condition>"
}
}
},
{
"name": "target",
"baseline": true,
"connector": {
"type": "parquet",
"config": {
"paths": [
"<path>"
],
"skipOnError": "false",
"recordFilter": "<filter_condition>"
}
}
}
],
"evaluate.rule": {
"rules": [
{
"dsl.type": "griffin-dsl",
"dq.type": "completeness",
"out.dataframe.name": "source_completeness",
"rule": "<columns_list>",
"details": {
"source": "source"
},
"out": [
{
"type": "metric",
"name": "source_completeness_metrics"
},
{
"type": "record",
"name": "source_completeness"
}
]
},
{
"dsl.type": "griffin-dsl",
"dq.type": "completeness",
"out.dataframe.name": "target_completeness",
"rule": "columns_list",
"details": {
"source": "target"
},
"out": [
{
"type": "metric",
"name": "target_completeness_metrics"
},
{
"type": "record",
"name": "target_completeness"
}
]
}
]
},
"sinks": [
<your_sink>
]
}
Please check and let know if you need anything.
Thanks,
Rashmi Nalwad
________________________________
From: Prince Jain <[email protected]>
Sent: Thursday, March 30, 2023 11:51 PM
To: [email protected] <[email protected]>
Subject: Issues while Connecting to DB
This email is from an external sender.
Hi,
I am trying to use apache griffin using docker where I am trying to connect my
postgres database in order to have data quality checks on the same, but failed
to do so.
There is also no proper documentation which I can refer to. I would be very
helpful if you could provide some kind of documentation or proper steps in
order to do the above work
Thanks & Regards
Prince Jain
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
