17. Concurrent Execution

threading example

import threading

counter = 0

def worker():

global counter

counter += 1

print('The count is %d' % counter)

for i in range(10):

threading.Thread(target=worker).start()

Queue - A synchronized queue class

The queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics.

The module implements three types of queue, which differ only in the order in which the entries are retrieved. In aFIFOqueue, the first tasks added are the first retrieved. In aLIFOqueue, the most recently added entry is the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the heapq module) and the lowest valued entry is retrieved first.

PriorityQueue

try:
import Queue as Q # ver. < 3.0
except ImportError:
import queue as Q

q = Q.PriorityQueue()
q.put(10)
q.put(1)
q.put(5)
while not q.empty():
print q.get(),

1 5 10

import queue as Q

q = Q.PriorityQueue()

q.put(10)

q.put(5)

q.put(2)

q.put(20)

while not q.empty():

print(q.get())

2

5

10

20

## Priority Queue can store objects such as tuples

try:
import Queue as Q # ver. < 3.0
except ImportError:
import queue as Q

q = Q.PriorityQueue()
q.put((10,'ten'))
q.put((1,'one'))
q.put((5,'five'))
while not q.empty():
print q.get(),

(1, 'one') (5, 'five') (10, 'ten')

# print a priority queue

print(q.queue)

https://docs.python.org/3/library/queue.html

Python Parallel Processing

https://www.machinelearningplus.com/python/parallel-processing-python

https://blog.floydhub.com/multiprocessing-vs-threading-in-python-what-every-data-scientist-needs-to-know

Subprocess

Can be used to call other compiled programs

import subprocess

subprocess.call(['./helloscript.go'])

Bottleneck	Example	Optimize with
IO	Network connection, file operation	Multithreading
CPU	Complex math problem, search	Multiprocessing

https://zacs.site/blog/linear-python.html

sched

The sched module defines a class which implements a general purpose event scheduler

https://docs.python.org/3/library/sched.html

Links

Ray | Faster Python

threading example​

Queue - A synchronized queue class​

PriorityQueue​

Python Parallel Processing​

Subprocess​

Can be used to call other compiled programs​

sched​

Links​