hello in this video, I will be taking a look at multiprocessing versus threading in Python. Now, I hear these two statements quite a lot with regards to multithreading in Python. The first one being that Python is multithreaded, and the second statement being that Python is not multithreaded. So which one's right? Well, I think the correct statement is actually that Python is multithreaded, but it is not simultaneously multithreaded. That is to say, two threads cannot execute simultaneously. So what is a thread? Well, to understand that, first we need to ask what a process is. A process is an instance of a program. This may be an instance of Firefox or several instances of Microsoft Word, for example. A process has memory set aside for its code and its data. A process then has at least one thread, which has its own registers and stack. This thread has access to the data of the process. The thread is the smallest sequence of instructions that can be managed by the OS. That is to say, the OS determines when and for how long the thread

can run a process. can spawn another thread. i'll be multi-threaded with each thread having its own register and stack. that is to say its own space in memory. but most importantly, it has access to the process memory space. thus, a thread can access the data held within the process. even if the python implementation creates user level threads instead of kernel threads. the user using the thread package will be able to access the data from the process. if another instance of the same program is launched, it does so with its own code and data space in the memory. these processes can all have their own threads. most importantly, however, is that the data is not accessible between processes. the data space from one process is not accessible to another process. in order for the two processes to share data, they must do so via queues or pipes. likewise, threads can also utilize cues and pipes to share data. but pipe some cues have some overhead to consider. having a look at threads and processes in a bit more detail in python. so here as an example, we have two processes. process zero

and process. One each process will have two threads: thread zero and thread one. If we consider what happens on a four-core CPU, we see that threads in Python are concurrent but not parallel. That is to say, each thread is subdivided in time, and a little bit of progress is made for each thread, but no two threads from the same process are ever executing at the same time. Both threads and process can jump around on the physical CPU cores. This behavior can be suppressed by asking the OS for an affinity to a certain CPU core, but that is up to the OS as to whether to do so. In any case, each thread, every given process, cannot execute simultaneously. That is the most important thing to remember here: that in Python, no two threads from the same process can operate simultaneously. Processes on the event can execute simultaneously. As you see, threads zero and thread one from both of the individual processes can run simultaneously because they belong to individual processes. In order for a thread to run, it must acquire a protected global interpreter lock within the

process by ensuring only a single thread can execute at any one time. Thread safety is guaranteed in Python, and by thread safety, we mean that there can be no deadlocks. This actually does improve a single-threaded performance, but it prevents simultaneous multi-threading within a process. That is to reiterate, two threads cannot execute simultaneously, meaning intense CPU tasks cannot be parallelized. But on the bright side, I/O limited threads are hardly affected at all since they spend most of their time waiting for data to come in or waiting for a turn signal for data going out. So if we go back to our previous example and instead of looking at CPU-bound tasks, we look at I/O-bound tasks, where here thread 0 in each process is an I/O-bound task, we can see that here threads in both processes have some I/O operations. These I/O operations are blocking, that is to say, once the I/O request is made, the thread must wait for the return value. This could be, for example, reading or writing to a file, reading or writing from a serial port.

or fetching a URL in Python. Once an I/O request is encountered, the interpreter releases the Global Interpreter Lock, allowing other threads time to execute. If thread 1 is a CPU-limited task doing lots of things, then being interrupted occasionally to do some I/O is acceptable, especially if the alternative would be to do some long process operation, then request and have to wait for the data to come in before continuing. With Python threading, the work can be done while the next task of work is being requested. And so now we can go ahead and compare threads and processes in Python using a few different metrics. So, for the setup cost in terms of both time and resources, threads are less costly than processes. For the handling, the CPython interpreter handles the threads where the OS handles the processes. For memory access, threads have access to all the data of the host process and can use the same state space to share with other threads, whereas processes have separate data spaces. For execution, both are concurrent but importantly in Python, threads are not parallel. Two threads cannot execute simultaneously in time.

processes. This is sometimes hard to explain, so let's look at an example. So here we have a function that takes in a timestamp t naught and an array. The array is set up so that it has chunks of time in 50,000 nanosecond intervals, that is 50 microsecond intervals, in the first column, and zero for the second column. The function enters an infinite loop where it simply evaluates the time since the timestamp using the performance counter. It adds one to the row in which the performance count is binned into. For example, if the performance counter returns 75,000 nanoseconds, then the second row of the second column is incremented once. The time exceeds the maximum time in the array, the function returns. In the end, the array will be populated as a histogram with the number of times the loop was executed within the 50 microsecond time increment. And as you can see, this is an intensive function that is always doing something, and importantly, none of the operations are multi-threaded behind the scenes.

so it is indeed a single car limited. We can then go ahead and import the threading library, and we can create a number of threads, and we can give them all a deep copy of this array. And then we run each thread, and the deep copies of the array will then be populated with the data from this function f. So what does that look like? So here are the results for four threads. I should say here that the CPU I'm running is an 8-car CPU with simultaneous multi-threading enabled, hence up to 16 kernel threads can execute simultaneously. We're only using four. The horizontal bars on the left indicate if the histogram was populated for that particular time interval. So this is an activity plot for the threads. Whenever there's color, it means the thread was doing some work. We can see that at any one point, only one thread is executing. The threads are all in discrete chunks of 15 milliseconds, which is no coincidence. CPython will allow a thread to run up to a blocking I/O call or 15 milliseconds, whichever happens.

sooner before another thread is given as a chance to execute. Very rarely, a thread will be reassigned to run after it's forced to stop after 50 milliseconds, which results in a small pause over the three seconds to which this activity plot was generated. This represents 200 lots of thread allocation chunks. The amount of time each thread has executed for is plotted on the right. You can see that the threads are not evenly executed despite all doing the same thing. Overall, only around three seconds of work has been done across all the threads, and the total number of function calls across all the threads was around two million. When switching to eight threads, the situation is very similar, but we see a lot more asymmetry in the time that the threads have been allocated. During this three-second window, you see a few dips in some of the chunks. This is where the OS jumped in and paused the thread briefly in order to do something, such as move the thread to another physical or virtual processor. This results in at least a 50 microsecond window and the no function

iterations were completed, and again, despite having eight threads, they collectively only do about three seconds of work in total spread across these eight threads. And we see a very large asymmetry where some of the threads are really not getting a lot of execution time during this three-second window. In particular, threads two and seven look like they've been given the wrong end of the stick, not actually performing a lot of work as compared to, say, threads zero and one. The total number of function calls this time across all threads was again around two million, so around the same number of actual function calls were performed despite doubling the number of threads allocated. When using 16 threads instead, we see a very large amount of asymmetry for the first three-second test. Again and again, what we'll see is overall only around three seconds of work will be done across all 16 threads with a large degree of asymmetry. Now across all of the threads, you can see that the threads with a lower index tend to be getting a lot more of this.

initial chunk of time than any of the threads. With a higher index, in fact, as you can see, currently none of threads 12, 13, 14, and 15 have actually had any time allocated to execute, and so the total number of function calls for the 16 threads again was around 2 million. So even doubling further from the 8 million, there will still be no improvement to see the actual work done by the threads. And so, how can we actually improve this? If we had this CPU-limited task, how would we go about actually increasing our performance using our multicore systems? So we can go ahead and look at using multiprocessing, which is a standard Python library used to do what most people think of when they actually say multithreading. By using multiprocessing to process several processes simultaneously. So if we can take our code and we can make a few changes. The first thing we must change, in particular for use on Windows, is to change the offset time to time.time. This is because the performance counter is relative to the

start time of the process. Since we are spawning multiple processes at different times, we cannot use the perth counter for the offset. We also can't use the time.clock as this has been depreciated and will be removed in Python 3.8. time.process time is also out as this is process specific. Luckily, the time.time function, on at least my machine, has a resolution of around one millisecond and is not process specific in reference. And the one millisecond is an acceptable offset. So, we're using time.time synchronization to synchronize to around one millisecond and time.perf counter for high performance binning. Also, we must make sure that we return the array this time because processes don't share memory. We are sending a separate copy of the array to the process, which then it must send back. So, what we do is we go ahead and create a number of deep copies of this array, and then we create a pool of workers using the multi-processing pool class, which will automatically send our array using the starmap function, and then also

handle the return array back to us in order of the out sending. We make sure we send a reliable timestamp, which is our tnot, to the individual process for the synchronization. There is a lot of overhead associated process, including the communication. I do have another video on my channel where I look into this, but for this video, we are only interested in how the execution is happening. So we'll only be looking at the performance of the processes. So if we look at the activity plot again, but this time with our four processes, we can see now something very different. We can see clearly that each process is executing simultaneously, so each process has activity happening at the same time as the others. Also, because they are running in parallel, their posting times are near identical. All four processes start at roughly the same time. Thanks to the fact that we're sending a relatively small array to each process and the process does a lot of work on this array, we are actually doing around four times the work than we could have done with multi-threading. So this is actually speeding up our

performance in fact. Over the course of this three seconds, we actually perform nine million function iterations, which is more than four times as many as the threading interface provided at around two million. You do get the occasional hiccup where the OS did something like switch a process to another car, etc., but overall, it's a massive improvement over a single-threaded application. When moving to eight processes, we start to see something very interesting. Each process has a different start time that exceeds the accuracy of the timestamp, meaning they do indeed start at different times. And this is because we still only have a single process, the main process, with a few threads handling packing and sending of the data to each worker process. And as we know, these threads can only execute in a non-parallel fashion. So, multiprocessing is better suited for very intense CPU-limited tasks where each process has to do a lot of work without spending an appreciable amount of its time on input and output. We see that since my CPU has eight cars, if simultaneous multi-threading, I can use eight of these cars with almost full utilization.

and Windows still has enough processing power to do its stuff behind the scenes. We do occasionally see a hiccup where some background process required a lot of multi-core CPU power and so competed with these Python processes for CPU time. But in general, we are actually doing around eight times as much work as we could have done with multi-threading. Overall, 16.7 million function iterations were made, which is around 85% more than for four processes but eight times more than using the eight threads. So, the improvement is still appreciable. If we then go on to our 16-process example again but this time using multiprocessing, things get very interesting. Remember, my CPU has eight physical cores with simultaneous multi-threading enabled, giving 16 simultaneous compute units. Here we can see that the start times of the processes are all quite spread. This spread is over about 180 milliseconds, which dwarfs the previous 10 milliseconds for the eight threads. We see now, during these three seconds, we have some processes that run relatively interrupted, but processes like 9, 10, 13, and 14

have been constantly interrupted resulting in lower progress overall despite getting a head start. This is not surprising as the OS has to manage all these processes, and occasionally something more important comes along that the OS must deal with. Its scheduling algorithm will decide to kick off some of our threads in order to do something else, but in general, we are still doing around 14 times as much work as we could have done with the multi-threading. So it's still a massive improvement, and the total number of function calls made here was around 20.6 million, which is only 23 percent more than for eight prices, despite doubling the amount of prices, which is quite strange. Or is it because the actual ST technologies, even though it's eight physical calls and 16 simultaneous calls thanks to simultaneous multiple technology, you don't actually get double the performance. The typical performance increases around 25, which is what we're seeing here. And so just comparing the activity plot for both packages, we can see more clearly that the multiprocessing is doing more in the

same amount of time than the threading do, and so if we look at the actual work done by both multi-threading and multi-processing, instead of just the activity blocks, we see that adding more threads did not result in any more work being done during those three seconds, whereas multi-processing achieved excellent scaling up to eight simultaneous processes, which is the same as the physical number of CPU cores, and then also another extra 25 percent from utilizing simultaneous multiple technology for the next eight cores, all while bearing in mind that the operating system still needs some CPU time to do things. So, what have we learned? If we take a look at the two different types of tasks that threading and multi-processing really cater to, if the task is CPU-limited, and by that we mean a large amount of CPU work is required, like the examples we saw in this video, multi-processing is the way to go. If the task is I/O-limited, such as waiting for serial data to come in, waiting for URLs to resolve, etc., multi-threading is the way to go. A program can do other

things while waiting for the i o such as refreshing a gui or processing the previous chunk of data. In fact, threading is excellent for guis or interactive figures. You can check the gui input 60 times a second, for example, which requires very little cpu time. And so, it makes the gpu very responsive because you're checking whether or not the users interacted with the gui many times a second. But the actual processing power associated with that is a very small chunk of the actual total processing power available for that single thread. And so, any background tasks, such as actually changing the gui or doing any data operations in the background, basically just as fast. And a word of warning, incorrectly using either one of these multi-processing or threading libraries can actually cause your program to run slower as both of them are associated with some overhead. They're not for free. And also, be careful with some types of cpu limited tasks. The back end of modules like numpy and scipy are heavily optimized and utilize simultaneous multi-threading in their native c code. They are written

in these functions will not benefit from multi-processing if they use all available calls. Just touching on this idea that the multi-processing can actually be slower. A bit more in a previous video. I showed how multi-processing may not offer a speed increase due to the large overhead associated with packing, sending, and receiving tasks to and from each worker in a process. It is entirely possible to get a massive performance hit when using multi-person incorrectly. This can often occur when you have a simple function that doesn't actually do a lot, and you're sending a lot of data to that simple function. The overhead associated with sending that data to the process dwarfs saving in computational time from having that done simultaneously. The ones that benefit the most are the ones where a lot of operations must be performed on a relatively small amount of data. And so in summary, what have we seen? We've seen that Python is most certainly multi-threaded. It's called current, not parallel. And that the C Python implementation will consider switching threads every 15

milliseconds or so. When IO operation is encountered, threads are a great way to divide up work in a discrete time fashion, and it's very useful things like GUIs where you only have to check occasionally for some interaction, or for programs where there is a lot of time spent waiting for data to come in or waiting for these IO operations. Multithreading does not restrict him in single car. The OS may switch the Python process between physical and virtual CPU cars. In a lot of sense, you're up to the mercy of what the OS decides to do, and multiprocessing is the standard path and way to increase pricing power if needed. It's part of the standard Python library and most numerical libraries such as NumPy, SciPy, TensorFlow, etc., are simultaneously multithreaded behind the scenes, so you won't necessarily get any speed increase when using multiprocessing. And here are some of the things I did not go through in this video, but that you should watch out for. Since threading allows process data to be accessed, data may need protected with a lock.

locks ensure that if a thread is interrupted halfway through changing or accessing some data, another thread cannot modify it during that interruption. E.g., if some data was updated by several rice operations, reading an intermediate value could return nonsense. Blocks can be used to protect these operations. If using a pipe to share data, they may become corrupted into test simultaneously types are not thread safe. Queue, on the other hand, can be accessed simultaneously and by multiple users and by multiple processes. That is to say, Q is thread and process safe. However, queues are more costly than a pipe to set up, so use the one most appropriate for your application. And so, thank you for watching. I hope you found this video informative, and stay tuned for more content like this.

threading vs multiprocessing in python

threading vs multiprocessing in python