In the realm of password recovery, benchmarking the speed of attacks holds significant importance. It is a customary practice to gauge the speed of attacks on various data formats using diverse hardware configurations. These tests yield results that are visually represented through graphs clearly demonstrating the performance of our products. However, these graphical representations merely scratch the surface of a much broader scope. Today, we delve deeper into the objectives and methodologies behind our password cracking speed tests.

The importance of benchmarks

While aesthetically pleasing graphs and benchmarks showcasing our products’ performance are readily available to our blog readers and website visitors, these nice looking graphs are not the sole objective. Internally, we do a much more thorough and comprehensive testing than could be fit ingo any reasonable graphical representation.

When benchmarking password recovery speeds, we pursue a diverse set of objectives.

Optimizing algorithms:

One of the key purposes of conducting performance tests is to gather data on the performance improvements achieved by internal optimizations. We compare the performance of the current (optimized) algorithm with its previous iteration, aiming to identify any enhancements but also inconsistencies across different hardware configurations. These benchmarks are performed using various hardware setups, focusing solely on a single algorithm. Such tests not only allow us to highlight the improved efficiency resulting from algorithm optimization but also reveal potential performance inconsistencies across different hardware configurations.

Introducing new data formats:

When introducing support for new data formats, we strive to compare the figures achieved in our tools against theoretical limits on as many different hardware configurations as possible. We share the results with our audience when announcing the support for a new data format. These tests enable us to assess the performance and effectiveness of our algorithms in handling different data formats, showcasing the extent of optimization achieved.

Supporting new GPU architectures:

With the introduction of new GPU architectures like NVIDIA’s Ada Lovelace or Intel’s Xe, it becomes crucial to adapt and test our algorithms on these hardware platforms. We conduct tests on graphics cards utilizing the new architecture, focusing on maximizing performance across multiple data formats. The measured performance is then compared against theoretical expectations, including performance on graphics cards built on already supported architectures. This approach ensures that our code performs optimally on the latest hardware. Whenever possible, we supplement the results with newly captured benchmarks on various models and architectures.

Benchmarking against competitors:

In all scenarios, we aim to compare our performance data with that of competing products. A significant deviation (over 20%) in either direction prompts the need for further investigations. Substantially lower performance may indicate inadequate optimization, while significantly higher performance demands additional scrutiny to ensure that no passwords of varying lengths are overlooked during an attack.

Testing methodology

To maintain consistency and comparability of results, we have established a rigorous testing methodology. Our testing tool is the internally used in-house developed command-line utility, epr_bench.exe. Except when introducing support for new data formats, our tests are based on the same set of files encrypted with robust passwords (more on that later).

We always test using a full brute-force attack method with a fixed password length and a specific character set limitation. The brute-force attack allows us to measure the pure attack speed on a particular  GPU or CPU model. Other attack methods, such as mask attacks, dictionary attacks, or more complex hybrid attacks, require additional computations that may limit the utilization of the graphics card.

The results of our testing are outputted as a CSV. Each entry in the file contains the test’s date and time, the model and parameters of the graphics card and CPU, and the acceleration algorithm used (e.g., CPU, NVIDIA, OpenCL). Additionally, it includes details such as the name and format of the test file, and the plugin used for testing. These entries are manually verified and stored in our internal database.

1682344293;"wz_aes256_sfx_456456456.exe";0;nvidia;0;"NVIDIA GeForce RTX 4070 (368 SP @ 2475 MHz, 4095 MB)";"12th Gen Intel(R) Core(TM) i9-12900K";24;1;1;"ZIP AES256";"ZIP Archive Files";"espr_zip.dll";"4.50.2531";399441920;59973;6660000;
1682344170;"v.1.2_sha256_aes256_pbkdf2-1024_123456789.odt";0;nvidia;0;"NVIDIA GeForce RTX 4070 (368 SP @ 2475 MHz, 4095 MB)";"12th Gen Intel(R) Core(TM) i9-12900K";24;1;1;"OpenDocument v1.2 (AES 256)";"OpenDocument";"espr_opendoc.dll";"4.50.2497";194445312;58629;3310000;

Our database allows us to extract specific datasets for analysis, enabling us to determine, for example, the relative performance of different graphics card models on the same set of test data or to compare the performance of our algorithms before and after optimization. The collected data serves as the foundation for generating informative graphs, which illustrate the performance characteristics and trends.

Special considerations

Occasionally, we emphasize specific aspects that are not part of the regular testing process. For instance, during the period of graphics card scarcity caused by cryptocurrency miners, we analyzed the results in terms of the price-performance ratio. Currently, with the decline in graphics card prices, we prioritize absolute performance and energy efficiency above everything else, the latter measured in the number of passwords per second per watt of power consumption. While we take these graphs with a grain of salt, as we rely on manufacturers’ specifications for power consumption rather than real-time measurements, they still allow us to compare different graphics cards with acceptable accuracy.

Beyond graphics cards: exploring password recovery speeds on complex hardware configurations

While graphics cards play a vital role in password cracking, CPUs are still important, especially for formats that demand a high usage of CPU resources alongside hardware acceleration. The performance and architecture of the CPU, along with proper implementation in our products, significantly impact the overall cracking speed. This is particularly evident in the challenges we faced when dealing with the hybrid architecture of Intel’s Alder Lake processors and their hybrid architecture. It is worth noting that certain algorithms, like Scrypt, cannot be implemented on a graphics card due to their inherent nature.

The technical nuances

Testing password recovery speeds is never simple. One significant aspect is the vast disparity (by 6-7 orders of magnitude) in attack speeds across different formats. This disparity poses challenges for both situations. Very slow recovery speeds require ample time to evaluate the stable recovery speed, while very fast attacks require adjusting the length of the test password to align with the expected recovery speed. Extremely high cracking speeds introduce another issue: the CPU may struggle to generate password packets at the same rate that GPU-based graphics cards can verify them. To overcome this limitation, we develop additional code that enables password packet generation directly on the graphics card for such formats.

Running the benchmark

Let’s focus on another aspect of our testing approach. We do not have the technical capability to test all supported data formats on all possible graphics cards using our products. If you are a licensed user of Elcomsoft Distributed Password Recovery and you need to determine the cracking speed for specific data but cannot find the corresponding benchmarks on our website, you have the option to conduct the test yourself. Follow these steps:

1. Launch Elcomsoft Distributed Password Recovery and add the encrypted file to the task queue.
2. Configure the attack as follows:
   • Attack type: Brute force
   • Password length restriction: Set both the minimum and maximum password length to 8 characters.
   • Character set restriction: Use lowercase Latin letters and digits only.

In the settings, select the graphics card for which you want to measure performance as the sole device (this is important because systems equipped with both a discrete graphics card and an integrated graphics core in the CPU will use both devices by default).

Next, start the attack and wait for 3 to 5 minutes. The program requires time to “ramp up.” During this period, the tool compiles and loads the code onto the GPU. The packet size, i.e., the number of passwords transmitted to the graphics card in one iteration, is dynamically calculated, and other necessary initializations are performed. After a few minutes, the performance will reach its maximum. The average speed is displayed in real time in Elcomsoft Distributed Password Recovery, but we recommend opening the agent window to view real-time performance.

Accurate performance data is extremely important. Even a modest 10% difference in cracking speed allows for better selection of optimal attacks. It enables either a larger range of password combinations to be attempted within the time constraints or faster completion of the task.

Conclusion

Testing the speed of password recovery serves multiple purposes beyond graphs and benchmarks. By conducting comprehensive performance measurements, we validate the optimizations we make, assess the compatibility with new GPU architectures, and benchmark against competitors. These tests enable us to deliver cutting-edge products with enhanced efficiency and provide our customers with reliable high-performance solutions.