LLMs have shown impressive reasoning capabilities, but solving complex problems often requires long inference-time computations. While humans tackle difficult problems by collaborating flexibly — dividing work, exploring multiple approaches, and adjusting strategies on the fly — current LLM frameworks enforce rigid collaboration patterns that don’t adapt well to all tasks.
A new paper called Hogwild! Inference offers a new approach to LLM parallelism that allows multiple model instances to run simultaneously with a shared attention cache, enabling them to develop their own collaboration strategy. This design lets LLM “workers” see each other’s progress in real-time and decide how best to collaborate without requiring specialized training.
Figure 1: An intuitive explanation of Hogwild! Inference, with 2 workers generating in parallel and 3 shared cache blocks. Each color denotes a cache block. Images from the paper.
Background: Existing Parallel Reasoning Approaches
Current approaches to parallel reasoning with LLMs fall into two main categories:
Discussion & aggregation methods like Self-Consistency allow multiple LLMs to reason independently and then vote on the solution. Extensions of this approach introduce communication rounds or specialized roles like Debugger, Examiner, or Judge. While these methods can improve reasoning accuracy, they don’t necessarily speed up the process since each agent must solve the entire problem sequentially.
Read more