17. Concurrent Execution
- 17.1.threading - Thread-based parallelism
- 17.2.multiprocessing - Process-based parallelism
- 17.3. Theconcurrentpackage
- 17.4.concurrent.futures - Launching parallel tasks
- 17.5.subprocess - Subprocess management
- 17.6.sched - Event scheduler
- 17.7.queue - A synchronized queue class
- 17.8.dummy_threading - Drop-in replacement for thethreadingmodule
- 17.9._thread - Low-level threading API
- 17.10._dummy_thread - Drop-in replacement for the_threadmodule
threading example
import threading
counter = 0
def worker():
global counter
counter += 1
print('The count is %d' % counter)
for i in range(10):
threading.Thread(target=worker).start()
Queue - A synchronized queue class
The queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics.
The module implements three types of queue, which differ only in the order in which the entries are retrieved. In aFIFOqueue, the first tasks added are the first retrieved. In aLIFOqueue, the most recently added entry is the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the heapq module) and the lowest valued entry is retrieved first.
PriorityQueue
try:
import Queue as Q # ver. < 3.0
except ImportError:
import queue as Q
q = Q.PriorityQueue()
q.put(10)
q.put(1)
q.put(5)
while not q.empty():
print q.get(),
1 5 10
import queue as Q
q = Q.PriorityQueue()
q.put(10)
q.put(5)
q.put(2)
q.put(20)
while not q.empty():
print(q.get())
2
5
10
20
## Priority Queue can store objects such as tuples
try:
import Queue as Q # ver. < 3.0
except ImportError:
import queue as Q
q = Q.PriorityQueue()
q.put((10,'ten'))
q.put((1,'one'))
q.put((5,'five'))
while not q.empty():
print q.get(),
(1, 'one') (5, 'five') (10, 'ten')
# print a priority queue
print(q.queue)
https://docs.python.org/3/library/queue.html
Python Parallel Processing
https://www.machinelearningplus.com/python/parallel-processing-python
Subprocess
Can be used to call other compiled programs
import subprocess
subprocess.call(['./helloscript.go'])
Bottleneck | Example | Optimize with |
---|---|---|
IO | Network connection, file operation | Multithreading |
CPU | Complex math problem, search | Multiprocessing |
https://zacs.site/blog/linear-python.html
sched
The sched module defines a class which implements a general purpose event scheduler
https://docs.python.org/3/library/sched.html