I have seen a use-case where query fails for 12 GB single json file having
structure '‘{ "key":[obj1, obj2, obj3..objn]}’'. Here json file has a key
element and value is array of json object 'obj'. There were around 175K objects
in this array and each obj is again complex json object with nested array
elements. From what I understood, Drill reads entire file content as single
json record (which actually is) and fails with DATA_READ ERROR.
The solution was to re organize the data to either of following structure. Also
to break single file into multiple smaller file for better parallelism.
Structure 2: File has array of json object like below
[ {obj1},{obj2}..,{objn}]
Structure 3: File has json objects as below
{obj1}
{obj1}
..
{objn}
I was checking if this is the case here..
Thanks,
Arjun
________________________________
From: Yun Liu <y....@castsoftware.com>
Sent: Saturday, November 4, 2017 2:27 AM
To: user@drill.apache.org
Subject: RE: Drill Capacity
Hi Arjun,
Column 4 has the most data and a bit long here. The other 3 columns has maybe a
word or 2. Thanks for your patience.
[ {
"type" : "quality-rules",
"reference" : {
"href" : "",
"name" : "Avoid unreferenced Tables",
"key" : "1634",
"critical" : false
},
"result" : {
"grade" : 2,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 5,
"successfulChecks" : 47,
"ratio" : 0.9038461538461539
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 1,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 2.0769230769230775,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 5,
"successfulChecks" : 47,
"ratio" : 0.9038461538461539
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 1,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "",
"name" : "Namespace naming convention - case control",
"key" : "3550",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 31,
"failedChecks" : 0,
"successfulChecks" : 31,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 31,
"failedChecks" : 0,
"successfulChecks" : 31,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "2",
"name" : "Interface naming convention - case and character set control",
"key" : "3554",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 10,
"failedChecks" : 0,
"successfulChecks" : 10,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 10,
"failedChecks" : 0,
"successfulChecks" : 10,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "",
"name" : "Enumerations naming convention - case and character set control",
"key" : "3558",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 7,
"failedChecks" : 0,
"successfulChecks" : 7,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 7,
"failedChecks" : 0,
"successfulChecks" : 7,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "",
"name" : "Enumeration Items naming convention - case and character set
control",
"key" : "3560",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 65,
"failedChecks" : 0,
"successfulChecks" : 65,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 65,
"failedChecks" : 0,
"successfulChecks" : 65,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "",
"name" : "Private Fields naming convention - case and character set
control",
"key" : "3562",
"critical" : false
},
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 130,
"failedChecks" : 90,
"successfulChecks" : 40,
"ratio" : 0.3076923076923077
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 130,
"failedChecks" : 90,
"successfulChecks" : 40,
"ratio" : 0.3076923076923077
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "",
"name" : "Methods naming convention - case and character set control",
"key" : "3566",
"critical" : false
},
"result" : {
"grade" : 3.759803921568629,
"violationRatio" : {
"totalChecks" : 561,
"failedChecks" : 11,
"successfulChecks" : 550,
"ratio" : 0.9803921568627451
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 3.759803921568629,
"violationRatio" : {
"totalChecks" : 561,
"failedChecks" : 11,
"successfulChecks" : 550,
"ratio" : 0.9803921568627451
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "",
"name" : "Avoid using Keywords as names",
"key" : "3570",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 1596,
"failedChecks" : 4,
"successfulChecks" : 1592,
"ratio" : 0.9974937343358395
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 1596,
"failedChecks" : 4,
"successfulChecks" : 1592,
"ratio" : 0.9974937343358395
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "3574/snapshots/3722",
"name" : "Properties naming convention - case and character set control",
"key" : "3574",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 352,
"failedChecks" : 0,
"successfulChecks" : 352,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 352,
"failedChecks" : 0,
"successfulChecks" : 352,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "3576/snapshots/3722",
"name" : "Avoid declaring public Fields",
"key" : "3576",
"critical" : true
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 130,
"failedChecks" : 0,
"successfulChecks" : 130,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 130,
"failedChecks" : 0,
"successfulChecks" : 130,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "3578/snapshots/3722",
"name" : "Avoid large Classes - too many Constructors",
"key" : "3578",
"critical" : false
},
"result" : {
"grade" : 3.9691011235955074,
"violationRatio" : {
"totalChecks" : 178,
"failedChecks" : 2,
"successfulChecks" : 176,
"ratio" : 0.9887640449438202
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 3.9691011235955074,
"violationRatio" : {
"totalChecks" : 178,
"failedChecks" : 2,
"successfulChecks" : 176,
"ratio" : 0.9887640449438202
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "3580/snapshots/3722",
"name" : "Avoid large Classes - too many Methods",
"key" : "3580",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 178,
"failedChecks" : 1,
"successfulChecks" : 177,
"ratio" : 0.9943820224719101
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 178,
"failedChecks" : 1,
"successfulChecks" : 177,
"ratio" : 0.9943820224719101
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "3586/snapshots/3722",
"name" : "Avoid large Methods - too many Lines of Code",
"key" : "3586",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 496,
"failedChecks" : 4,
"successfulChecks" : 492,
"ratio" : 0.9919354838709677
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 496,
"failedChecks" : 4,
"successfulChecks" : 492,
"ratio" : 0.9919354838709677
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "3612/snapshots/3722",
"name" : "Close SQL connection ASAP",
"key" : "3612",
"critical" : true
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 561,
"failedChecks" : 0,
"successfulChecks" : 561,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 561,
"failedChecks" : 0,
"successfulChecks" : 561,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "3614/snapshots/3722",
"name" : "Avoid using String.Empty for empty string tests",
"key" : "3614",
"critical" : false
},
"result" : {
"grade" : 3.971603563474389,
"violationRatio" : {
"totalChecks" : 449,
"failedChecks" : 5,
"successfulChecks" : 444,
"ratio" : 0.9888641425389755
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 3.971603563474389,
"violationRatio" : {
"totalChecks" : 449,
"failedChecks" : 5,
"successfulChecks" : 444,
"ratio" : 0.9888641425389755
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "3626/snapshots/3722",
"name" : "Avoid Interfaces with a very low comment/code ratio",
"key" : "3626",
"critical" : false
},
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 10,
"failedChecks" : 8,
"successfulChecks" : 2,
"ratio" : 0.2
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 10,
"failedChecks" : 8,
"successfulChecks" : 2,
"ratio" : 0.2
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "3630/snapshots/3722",
"name" : "Avoid having Classes implementing too many Interfaces",
"key" : "3630",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 178,
"failedChecks" : 0,
"successfulChecks" : 178,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 178,
"failedChecks" : 0,
"successfulChecks" : 178,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "4062/snapshots/3722",
"name" : "Avoid Functions and Procedures doing an Insert, Update or Delete
without managing a transaction",
"key" : "4062",
"critical" : false
},
"result" : {
"grade" : 1.3841463414634148,
"violationRatio" : {
"totalChecks" : 205,
"failedChecks" : 71,
"successfulChecks" : 134,
"ratio" : 0.6536585365853659
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 12,
"removedViolations" : 2
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 1.3841463414634148,
"violationRatio" : {
"totalChecks" : 205,
"failedChecks" : 71,
"successfulChecks" : 134,
"ratio" : 0.6536585365853659
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 12,
"removedViolations" : 2
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "4064/snapshots/3722",
"name" : "Avoid Procedures using an Insert, Update, Delete, Create Table or
Select without including error management",
"key" : "4064",
"critical" : true
},
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 203,
"failedChecks" : 172,
"successfulChecks" : 31,
"ratio" : 0.15270935960591134
},
"evolutionSummary" : {
"addedCriticalViolations" : 23,
"removedCriticalViolations" : 2,
"addedViolations" : 23,
"removedViolations" : 2
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 203,
"failedChecks" : 172,
"successfulChecks" : 31,
"ratio" : 0.15270935960591134
},
"evolutionSummary" : {
"addedCriticalViolations" : 23,
"removedCriticalViolations" : 2,
"addedViolations" : 23,
"removedViolations" : 2
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "4066/snapshots/3722",
"name" : "Avoid Stored Procedures not returning a status value",
"key" : "4066",
"critical" : false
},
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 203,
"failedChecks" : 184,
"successfulChecks" : 19,
"ratio" : 0.09359605911330049
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 23,
"removedViolations" : 2
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 203,
"failedChecks" : 184,
"successfulChecks" : 19,
"ratio" : 0.09359605911330049
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 23,
"removedViolations" : 2
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "4070/snapshots/3722",
"name" : "Avoid use of \"truncate table\"",
"key" : "4070",
"critical" : true
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 206,
"failedChecks" : 0,
"successfulChecks" : 206,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 206,
"failedChecks" : 0,
"successfulChecks" : 206,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "4076/snapshots/3722",
"name" : "Avoid using temporary Objects",
"key" : "4076",
"critical" : false
},
"result" : {
"grade" : 3.274390243902438,
"violationRatio" : {
"totalChecks" : 205,
"failedChecks" : 8,
"successfulChecks" : 197,
"ratio" : 0.9609756097560975
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 3.274390243902438,
"violationRatio" : {
"totalChecks" : 205,
"failedChecks" : 8,
"successfulChecks" : 197,
"ratio" : 0.9609756097560975
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "4084/snapshots/3722",
"name" : "Avoid nested Stored Procedures using temporary Tables",
"key" : "4084",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 203,
"failedChecks" : 0,
"successfulChecks" : 203,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 203,
"failedChecks" : 0,
"successfulChecks" : 203,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7126/snapshots/3722",
"name" : "Avoid Artifacts with high Commented-out Code Lines/Code Lines
ratio",
"key" : "7126",
"critical" : false
},
"result" : {
"grade" : 3.7591514143094855,
"violationRatio" : {
"totalChecks" : 3005,
"failedChecks" : 59,
"successfulChecks" : 2946,
"ratio" : 0.9803660565723794
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 4,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 3.8659342622365145,
"violationRatio" : {
"totalChecks" : 2799,
"failedChecks" : 43,
"successfulChecks" : 2756,
"ratio" : 0.9846373704894605
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
}, {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 2.4466019417475735,
"violationRatio" : {
"totalChecks" : 206,
"failedChecks" : 16,
"successfulChecks" : 190,
"ratio" : 0.9223300970873787
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 4,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7130/snapshots/3722",
"name" : "Avoid Artifacts with High Depth of Nested Subqueries",
"key" : "7130",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 206,
"failedChecks" : 0,
"successfulChecks" : 206,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 206,
"failedChecks" : 0,
"successfulChecks" : 206,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7156/snapshots/3722",
"name" : "Avoid Too Many Copy Pasted Artifacts",
"key" : "7156",
"critical" : false
},
"result" : {
"grade" : 1.6478102189781023,
"violationRatio" : {
"totalChecks" : 959,
"failedChecks" : 231,
"successfulChecks" : 728,
"ratio" : 0.7591240875912408
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 7,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 1.671151439299124,
"violationRatio" : {
"totalChecks" : 799,
"failedChecks" : 185,
"successfulChecks" : 614,
"ratio" : 0.7684605757196495
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
}, {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 1.53125,
"violationRatio" : {
"totalChecks" : 160,
"failedChecks" : 46,
"successfulChecks" : 114,
"ratio" : 0.7125
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 7,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7194/snapshots/3722",
"name" : "Avoid large number of String concatenation",
"key" : "7194",
"critical" : false
},
"result" : {
"grade" : 3.796370967741936,
"violationRatio" : {
"totalChecks" : 496,
"failedChecks" : 9,
"successfulChecks" : 487,
"ratio" : 0.9818548387096774
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 3.796370967741936,
"violationRatio" : {
"totalChecks" : 496,
"failedChecks" : 9,
"successfulChecks" : 487,
"ratio" : 0.9818548387096774
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7198/snapshots/3722",
"name" : "Avoid String concatenation in loops",
"key" : "7198",
"critical" : false
},
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 496,
"failedChecks" : 15,
"successfulChecks" : 481,
"ratio" : 0.969758064516129
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 496,
"failedChecks" : 15,
"successfulChecks" : 481,
"ratio" : 0.969758064516129
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7204/snapshots/3722",
"name" : "Avoid method invocation in a loop termination expression",
"key" : "7204",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 561,
"failedChecks" : 2,
"successfulChecks" : 559,
"ratio" : 0.9964349376114082
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 561,
"failedChecks" : 2,
"successfulChecks" : 559,
"ratio" : 0.9964349376114082
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7208/snapshots/3722",
"name" : "Avoid the use of is inside loops",
"key" : "7208",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 496,
"failedChecks" : 0,
"successfulChecks" : 496,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 496,
"failedChecks" : 0,
"successfulChecks" : 496,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7212/snapshots/3722",
"name" : "Avoid instantiations inside loops",
"key" : "7212",
"critical" : true
},
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 496,
"failedChecks" : 23,
"successfulChecks" : 473,
"ratio" : 0.9536290322580645
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 496,
"failedChecks" : 23,
"successfulChecks" : 473,
"ratio" : 0.9536290322580645
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7260/snapshots/3722",
"name" : "User Interface elements must not use directly the database",
"key" : "7260",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 561,
"failedChecks" : 0,
"successfulChecks" : 561,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 561,
"failedChecks" : 0,
"successfulChecks" : 561,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7262/snapshots/3722",
"name" : "Avoid Namespaces with High Efferent Coupling (CE)",
"key" : "7262",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 0,
"successfulChecks" : 52,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 0,
"successfulChecks" : 52,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7264/snapshots/3722",
"name" : "Avoid namespaces with High Afferent Coupling (CA)",
"key" : "7264",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 0,
"successfulChecks" : 52,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 0,
"successfulChecks" : 52,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7266/snapshots/3722",
"name" : "Call 'base.Dispose()' or 'MyBase.Finalize()' in the \"finally\"
block of 'Dispose(bool)' methods",
"key" : "7266",
"critical" : false
},
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 3,
"failedChecks" : 3,
"successfulChecks" : 0,
"ratio" : 0.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 3,
"failedChecks" : 3,
"successfulChecks" : 0,
"ratio" : 0.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7268/snapshots/3722",
"name" : "Dispose() methods should call GC.SuppressFinalize",
"key" : "7268",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 5,
"failedChecks" : 0,
"successfulChecks" : 5,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 5,
"failedChecks" : 0,
"successfulChecks" : 5,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7270/snapshots/3722",
"name" : "Declare as Static all methods not using instance members",
"key" : "7270",
"critical" : false
},
"result" : {
"grade" : 3.5369875222816383,
"violationRatio" : {
"totalChecks" : 561,
"failedChecks" : 16,
"successfulChecks" : 545,
"ratio" : 0.9714795008912656
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 3.5369875222816383,
"violationRatio" : {
"totalChecks" : 561,
"failedChecks" : 16,
"successfulChecks" : 545,
"ratio" : 0.9714795008912656
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7272/snapshots/3722",
"name" : "Provide a private default Constructor for utility Classes",
"key" : "7272",
"critical" : false
},
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 6,
"failedChecks" : 6,
"successfulChecks" : 0,
"ratio" : 0.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 6,
"failedChecks" : 6,
"successfulChecks" : 0,
"ratio" : 0.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7294/snapshots/3722",
"name" : "Avoid cyclical calls and inheritances between namespaces content",
"key" : "7294",
"critical" : true
},
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 31,
"failedChecks" : 5,
"successfulChecks" : 26,
"ratio" : 0.8387096774193549
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 31,
"failedChecks" : 5,
"successfulChecks" : 26,
"ratio" : 0.8387096774193549
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7344/snapshots/3722",
"name" : "Avoid \"SELECT *\" queries",
"key" : "7344",
"critical" : false
},
"result" : {
"grade" : 2.932038834951456,
"violationRatio" : {
"totalChecks" : 206,
"failedChecks" : 11,
"successfulChecks" : 195,
"ratio" : 0.9466019417475728
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 2.932038834951456,
"violationRatio" : {
"totalChecks" : 206,
"failedChecks" : 11,
"successfulChecks" : 195,
"ratio" : 0.9466019417475728
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7346/snapshots/3722",
"name" : "Avoid redundant indexes",
"key" : "7346",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 0,
"successfulChecks" : 52,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 0,
"successfulChecks" : 52,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7348/snapshots/3722",
"name" : "Avoid too many Indexes on one Table",
"key" : "7348",
"critical" : false
},
"result" : {
"grade" : 3.76923076923077,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 1,
"successfulChecks" : 51,
"ratio" : 0.9807692307692307
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 3.76923076923077,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 1,
"successfulChecks" : 51,
"ratio" : 0.9807692307692307
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7352/snapshots/3722",
"name" : "Avoid calling properties that clone values in loops",
"key" : "7352",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 561,
"failedChecks" : 0,
"successfulChecks" : 561,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 561,
"failedChecks" : 0,
"successfulChecks" : 561,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7386/snapshots/3722",
"name" : "Avoid Tables without a clustered Index",
"key" : "7386",
"critical" : false
},
"result" : {
"grade" : 1.721153846153846,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 11,
"successfulChecks" : 41,
"ratio" : 0.7884615384615384
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 4,
"removedViolations" : 1
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 1.721153846153846,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 11,
"successfulChecks" : 41,
"ratio" : 0.7884615384615384
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 4,
"removedViolations" : 1
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7388/snapshots/3722",
"name" : "Avoid artifacts having recursive calls",
"key" : "7388",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 3005,
"failedChecks" : 19,
"successfulChecks" : 2986,
"ratio" : 0.9936772046589019
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 2799,
"failedChecks" : 19,
"successfulChecks" : 2780,
"ratio" : 0.9932118613790639
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
}, {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 206,
"failedChecks" : 0,
"successfulChecks" : 206,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7390/snapshots/3722",
"name" : "Avoid having multiple Artifacts inserting data on the same SQL
Table",
"key" : "7390",
"critical" : false
},
"result" : {
"grade" : 1.673076923076923,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 12,
"successfulChecks" : 40,
"ratio" : 0.7692307692307693
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 1.673076923076923,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 12,
"successfulChecks" : 40,
"ratio" : 0.7692307692307693
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7392/snapshots/3722",
"name" : "Avoid having multiple artifacts deleting data on the same SQL
table",
"key" : "7392",
"critical" : false
},
"result" : {
"grade" : 1.8653846153846154,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 8,
"successfulChecks" : 44,
"ratio" : 0.8461538461538461
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 1.8653846153846154,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 8,
"successfulChecks" : 44,
"ratio" : 0.8461538461538461
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7394/snapshots/3722",
"name" : "Avoid having multiple Artifacts updating data on the same SQL
Table",
"key" : "7394",
"critical" : false
},
"result" : {
"grade" : 2.4615384615384612,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 4,
"successfulChecks" : 48,
"ratio" : 0.9230769230769231
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 2.4615384615384612,
"violationRatio" : {
"totalChecks" : 52,
"failedChecks" : 4,
"successfulChecks" : 48,
"ratio" : 0.9230769230769231
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7418/snapshots/3722",
"name" : "Avoid SQL queries using functions on indexed columns in the WHERE
clause",
"key" : "7418",
"critical" : false
},
"result" : {
"grade" : 3.28846153846154,
"violationRatio" : {
"totalChecks" : 182,
"failedChecks" : 7,
"successfulChecks" : 175,
"ratio" : 0.9615384615384616
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 5,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 3.28846153846154,
"violationRatio" : {
"totalChecks" : 182,
"failedChecks" : 7,
"successfulChecks" : 175,
"ratio" : 0.9615384615384616
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 5,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7420/snapshots/3722",
"name" : "Avoid SQL queries with implicit conversions in the WHERE clause",
"key" : "7420",
"critical" : false
},
"result" : {
"grade" : 3.425824175824175,
"violationRatio" : {
"totalChecks" : 182,
"failedChecks" : 6,
"successfulChecks" : 176,
"ratio" : 0.967032967032967
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 1,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 3.425824175824175,
"violationRatio" : {
"totalChecks" : 182,
"failedChecks" : 6,
"successfulChecks" : 176,
"ratio" : 0.967032967032967
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 1,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7424/snapshots/3722",
"name" : "Avoid using SQL queries inside a loop",
"key" : "7424",
"critical" : true
},
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 221,
"failedChecks" : 11,
"successfulChecks" : 210,
"ratio" : 0.9502262443438914
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 15,
"failedChecks" : 0,
"successfulChecks" : 15,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
}, {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 206,
"failedChecks" : 11,
"successfulChecks" : 195,
"ratio" : 0.9466019417475728
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7428/snapshots/3722",
"name" : "Avoid SQL queries not using the first column of a composite index
in the WHERE clause",
"key" : "7428",
"critical" : false
},
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 24,
"failedChecks" : 24,
"successfulChecks" : 0,
"ratio" : 0.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 3,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 24,
"failedChecks" : 24,
"successfulChecks" : 0,
"ratio" : 0.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 3,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7458/snapshots/3722",
"name" : "Avoid large Interfaces - too many Methods",
"key" : "7458",
"critical" : false
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 10,
"failedChecks" : 0,
"successfulChecks" : 10,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 10,
"failedChecks" : 0,
"successfulChecks" : 10,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7460/snapshots/3722",
"name" : "Avoid using untyped DataSet",
"key" : "7460",
"critical" : true
},
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 17,
"failedChecks" : 17,
"successfulChecks" : 0,
"ratio" : 0.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 1.0,
"violationRatio" : {
"totalChecks" : 17,
"failedChecks" : 17,
"successfulChecks" : 0,
"ratio" : 0.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7470/snapshots/3722",
"name" : "Avoid doing select on Datatable in loop",
"key" : "7470",
"critical" : true
},
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 2,
"failedChecks" : 0,
"successfulChecks" : 2,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 2,
"failedChecks" : 0,
"successfulChecks" : 2,
"ratio" : 1.0
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7760/snapshots/3722",
"name" : "Avoid triggers, functions and procedures with a very low
comment/code ratio",
"key" : "7760",
"critical" : false
},
"result" : {
"grade" : 1.4695121951219512,
"violationRatio" : {
"totalChecks" : 205,
"failedChecks" : 64,
"successfulChecks" : 141,
"ratio" : 0.6878048780487804
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 2,
"removedViolations" : 1
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 1.4695121951219512,
"violationRatio" : {
"totalChecks" : 205,
"failedChecks" : 64,
"successfulChecks" : 141,
"ratio" : 0.6878048780487804
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 2,
"removedViolations" : 1
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7762/snapshots/3722",
"name" : "Avoid undocumented Triggers, Functions and Procedures",
"key" : "7762",
"critical" : false
},
"result" : {
"grade" : 1.6524390243902438,
"violationRatio" : {
"totalChecks" : 205,
"failedChecks" : 49,
"successfulChecks" : 156,
"ratio" : 0.7609756097560976
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 1,
"removedViolations" : 1
}
},
"technologyResults" : [ {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 1.6524390243902438,
"violationRatio" : {
"totalChecks" : 205,
"failedChecks" : 49,
"successfulChecks" : 156,
"ratio" : 0.7609756097560976
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 1,
"removedViolations" : 1
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7766/snapshots/3722",
"name" : "Avoid Artifacts with High Cyclomatic Complexity",
"key" : "7766",
"critical" : false
},
"result" : {
"grade" : 1.8886054421768708,
"violationRatio" : {
"totalChecks" : 2940,
"failedChecks" : 425,
"successfulChecks" : 2515,
"ratio" : 0.8554421768707483
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 1,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 1.8632040965618142,
"violationRatio" : {
"totalChecks" : 2734,
"failedChecks" : 423,
"successfulChecks" : 2311,
"ratio" : 0.8452816386247257
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
}
}, {
"technology" : "Microsoft T-SQL",
"result" : {
"grade" : 4.0,
"violationRatio" : {
"totalChecks" : 206,
"failedChecks" : 2,
"successfulChecks" : 204,
"ratio" : 0.9902912621359223
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 1,
"removedViolations" : 0
}
}
} ]
}, {
"type" : "quality-rules",
"reference" : {
"href" : "7768/snapshots/3722",
"name" : "Avoid Artifacts with High Depth of Code",
"key" : "7768",
"critical" : false
},
"result" : {
"grade" : 2.4081632653061233,
"violationRatio" : {
"totalChecks" : 2940,
"failedChecks" : 234,
"successfulChecks" : 2706,
"ratio" : 0.9204081632653062
},
"evolutionSummary" : {
"addedCriticalViolations" : 0,
"removedCriticalViolations" : 0,
"addedViolations" : 0,
"removedViolations" : 0
}
},
"technologyResults" : [ {
"technology" : ".NET",
"result" : {
"grade" : 2.2882223847842,
"violationRatio" : {
"totalChecks" : 2734,
"failedChecks" : 234,
"successfulChecks" : 2500,
"ratio" : 0.9144111192392099
},
"evolutionSummary" : {
"addedCritica
Thanks,
Yun
-----Original Message-----
From: Arjun kr [mailto:arjun...@outlook.com]
Sent: Friday, November 3, 2017 4:41 PM
To: user@drill.apache.org
Subject: Re: Drill Capacity
Hi Yun,
Could you please provide more details on your json data structure for 400 MB
json file.
Structure 1:
'{ "key":[obj1, obj2, obj3..objn]}'
Structure 2:
[ {obj1},{obj2}..,{objn}]
Structure 3:
{obj1}
{obj1}
..
{objn}
Thanks,
Arjun
________________________________
From: Yun Liu <y....@castsoftware.com>
Sent: Saturday, November 4, 2017 1:49 AM
To: user@drill.apache.org
Subject: RE: Drill Capacity
Hi Paul,
Thanks for you detailed explanation. First off- I have 2 issues and I wanted to
clear it out before continuing.
Current setting: planner.memory.max_query_memory_per_node = 10GB, HEAP = 12G,
Direct memory = 32G, Perm 1024M, and planner.width.max_per_node = 5
Issue # 1:
When loading a json file with 400MB I keep getting a DATA_READ ERROR.
Each record in the file is about 64KB. Since it's a json file, there are only 4
fields per each record. Not sure how many records this file contains as it's
too large to open with any tools, but I am guessing about 3k rows.
With all the recommendations provided by various experts, nothing has worked.
Issue 2#:
While processing a query with is a join of 2 functional .json files, I am
getting a RESOURCE ERROR: One or more nodes ran out of memory while executing
the query. These 2 json files alone process fine but when joined together,
Drill throws me that error.
Json#1 is 11k KB, has 8 fields with 74091 rows
Json#2 is 752kb, has 8 fields with 4245 rows
Besides breaking them up to smaller files, not sure what else I could do.
Thanks for the help so far!
Yun
-----Original Message-----
From: Paul Rogers [mailto:prog...@mapr.com]
Sent: Thursday, November 2, 2017 11:06 PM
To: user@drill.apache.org
Subject: Re: Drill Capacity
Hi Yun,
I'm going to give you multiple ways to understand the issue based on the
information you've provided. I generally like to see the full logs to diagnose
such problems, but we'll start with what you've provided thus far.
How large is each record in your file? How many fields? How many bytes?
(Alternatively, how big is a single input file and how many records does it
contain?)
You mention the limit of 64K columns in CSV. This makes me wonder if you have a
"jumbo" record. If each individual record is large, then there won't be enough
space in the sort to take even a single batch of records, and you'll get the
sv2 error that you saw.
We can guess the size, however, from the info you provided:
batchGroups.size 1
spilledBatchGroups.size 0
allocated memory 42768000
allocator limit 41943040
This says you have a batch in memory and are trying to allocate some memory
(the "sv2"). The allocated memory number tells us that each batch size is
probably ~43 MB. But, the sort only has 42 MB to play with. The sort needs at
least two batches in memory to make progress, hence the out-of-memory errors.
It would be nice to confirm this from the logs, but unfortunately, Drill does
not normally log the size of each batch. As it turns out, however, the
"managed" version that Boaz mentioned added more logging around this problem:
it will tell you how large it thinks each batch is, and will warn if you have,
say, a 43 MB batch but only 42 MB in which to sort.
(If you do want to use the "managed" version of the sort, I suggest you try
Drill 1.12 when it is released as that version contains additional fixes to
handle constrained memory.)
Also, at present, The JSON record reader loads 4096 records into each batch. If
your file has at least that many records, then we can guess each record is
about 43 MB / 4096 =~ 10K in size. (You can confirm, as noted above, by
dividing total file size by record count.)
We are doing work to handle such large batches, but the work is not yet
available in a release. Unfortunately, in the meanwhile, we also don't let you
control the batch size. But, we can provide another solution.
Let's explain why the message you provided said that the "allocator limit" was
42 MB. Drill does the following to allocate memory to the sort:
* Take the "max query memory per node" (default of 2 GB regardless of actual
direct memory),
* Divide by the number of sort operators in the plan (as shown in the
visualized query profile)
* Divide by the "planner width" which is, by default, 70% of the number of
cores on your system.
In your case, if you are using the default 2 GB total, but getting 41 MB per
sort, the divisor is 50. Maybe you have 2 sorts and 32 cores? (2 * 32 * 70% =~
45.) Or some other combination.
We can't reduce the number of sorts; that's determined by your query. But, we
can play with the other numbers.
First, we can increase the memory per query:
ALTER SESSION SET `planner.memory.max_query_memory_per_node` = 4,294,967,296
That is, 4 GB. This obviously means you must have at least 6 GB of direct
memory; more is better.
And/or, we can reduce the number of fragments:
ALTER SESSION SET `planner.width.max_per_node` = <a number>
The value is a bit tricky. Drill normally creates a number of fragments equal
to 70% of the number of CPUs on your system. Let's say you have 32 cores. If
so, change the max_per_node to, say, 10 or even 5. This will mean fewer sorts
and so more memory per sort, helping compensate for the "jumbo" batches in your
query. Pick a number based on your actual number of cores.
As an alternative, as Ted suggested, you could create a larger number of
smaller files as this would solve the batch size problem while also getting the
parallelization benefits that Kunal mentioned.
That is three separate possible solutions. Try them one by one or (carefully)
together.
- Paul
>> On 11/2/17, 12:31 PM, "Yun Liu" <y....@castsoftware.com> wrote:
>>
>> Hi Kunal and Andries,
>>
>> Thanks for your reply. We need json in this case because Drill
>> only supports up to 65536 columns in a csv file.
