Hi Lionel, I created a custom config.json and defined a custom rule to run<dsl.type is spark-sql> and submitted it through spark. The spark submit runs fine without any issues. On HDFS, I could see the directory with the custom rule name but I am unable to see the _METRIC file where the results will be persisted, I only see a _START file. What am I missing here?
HDFS -> ../persist/CheckAlphaNumeric/1527143022495/_START Config.json -> { "name": "CheckAlphaNumeric", "process.type": "batch", "data.sources": [ { "name": "src", "connectors": [ { "type": "hive", "version": "1.2", "config": { "database": "griffined", "table.name": "check_table" } } ] } ], "evaluateRule": { "rules": [ { "dsl.type": "spark-sql", "dq.type": "profiling", "name": "checkalphnumeric", "rule": "SELECT count(name) FROM src WHERE name REGEXP '^[a-zA-Z0-9]+$'", "metric": { "name": "check_rules" } } ] } } ________________________________ Any comments or statements made in this email are not necessarily those of Tavant Technologies. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete the material from any computer. All emails sent from or to Tavant Technologies may be subject to our monitoring procedures.