Thread Pool
aka Replicated Workers or Worker-crew model.
Problem
- Frequent creation and destruction of threads for short-lived tasks can be expensive.
Solution
- Maintain multiple threads waiting for a task to be assigned.
- Queue the tasks and assign them to worker threads
- Add tasks to completed tasks queue.
Benefits
- Thread creation-destruction overhead is contained to initial creation of the pool => may result in better performance and stability
Optimal Pool Size
- Threads consume memory
- Context-switching also takes time
=> If the number of threads are excess, the above two will lead to memory wastage and performance penalties.
- Thread creation + more threads = cost of creating unused threads
- Thread creation + less threads = longer waiting time
- Thread destruction + more threads = cost of creating them again later
- Thread destruction + less threads = May starve other processes of resources (more unused threads in memory)
Thus, the pool size has to be very carefully tuned to a sweet-spot. This can be dynamically determined during the application's lifecycle based on the number of waiting tasks.
CPU Intensive tasks
Tasks which block the CPU cores. For example, intensive hashing or cryptographic algorithms. In such cases, if you have even thousands of tasks, the number of threads running at a time will be equal to the number of cores on your CPU. So if you have too many threads, CPU will simply keep rotating between all these tasks and won't be able to actually complete any of these quick.
In such cases, Ideal Pool Size = CPU Core count
Other consideration: How many other applications (or executors/threads are running on the same CPU)
I/O Intensive tasks
Tasks which don't really need the CPU, like input/output from user, or calls to a remote service. In such cases, even if you create few threads, CPU will keep waiting for a response to come back, staying idle.
Ideal pool size will be higher in such cases.
Exact number will depend on the rate of task submissions and average task wait time. Too many threads will increase memory consumption too.
