Not all passwords provide equal protection. Some formats are more resistant to brute-force attacks than others. As an example, Microsoft Office 2013 and 2016 employ a smart encryption scheme that is very slow to decrypt. Even the fastest available GPU units found in NVIDIA’s latest GeForce GTX 1080 will only allow trying some 7100 passwords per second.
One solution is employing a custom dictionary, possibly containing the user’s passwords that were easier to break. Observing the common pattern in those other passwords may allow creating a custom mask that could greatly reduce the number of possible combinations.