The Mechanism: Incompetence Is Self-Concealing
The insight at the heart of the Dunning-Kruger effect is that the skills required to perform a task well are largely the same skills required to evaluate performance on that task. If you lack the competence to do something well, you also lack the competence to recognise that your performance is poor. Incompetence is therefore self-concealing: the person who doesn't know what they don't know cannot perceive the gap.
As Dunning later wrote: "The skills you need to produce a right answer are exactly the skills you need to recognize what a right answer is."
This is distinct from simple overconfidence — it is a specific prediction that people at the bottom of the skill distribution will be the most wildly miscalibrated.