In this post, we address a common challenge: efficiently processing large-scale inputs through algorithmic pipelines that include deep learning models. A typical solution is to run multiple independent jobs, each responsible for processing a single input. This setup is often managed with job orchestration frameworks (e.g.,\u00a0Kubernetes<\/a>). However, when deep learning models are involved, this approach can become inefficient as loading and executing the same model in each individual process can lead to resource contention and scaling limitations. As AI models become increasingly prevalent in algorithmic pipelines, it is crucial that we revisit the design of such solutions.<\/p>\n In this post we evaluate the benefits of centralized Inference<\/a> serving, where a dedicated inference server handles prediction requests from multiple parallel jobs. We define a toy experiment in which we run an image-processing pipeline based on a\u00a0ResNet-152<\/a>\u00a0image classifier on 1,000 individual images. We compare the runtime performance and resource utilization of the following two implementations:<\/p>\n To keep the experiment focused, we make several simplifying assumptions:<\/p>\n The code is shared for demonstrative purposes only. Please do not interpret our choice of TorchServe \u2014 or any other component of our demonstration \u2014 as an endorsement of its use.<\/p>\n We conduct our experiments on an\u00a0Amazon EC2 c5.2xlarge<\/a>\u00a0instance, with 8 vCPUs and 16 GiB of memory, running a\u00a0PyTorch Deep Learning AMI<\/a>\u00a0(DLAMI). We activate the PyTorch environment using the following command:<\/p>\n We begin by creating a ResNet-152 model checkpoint. Using\u00a0TorchScript<\/a>, we serialize both the model definition and its weights into a single file:<\/p>\n Our inference function performs the following steps:<\/p>\n We define a constant MAX_THREADS hyperparameter that we use to restrict the number of threads used for model inference in each process. This is to prevent resource contention between the multiple jobs.<\/p>\n We define a function that spawns parallel processes, each processing a single image input. This function:<\/p>\n While the optimal number of maximum concurrent processes is best determined empirically, we can estimate an upper bound based on the 16 GiB of system memory and the size of the resnet-152.pt file, 231 MB.<\/p>\n The table below summarizes the runtime results for several configurations:<\/p>\n Although memory becomes fully saturated at 50 concurrent processes, we observe that maximum throughput is achieved at 8 concurrent jobs\u200a\u2014\u200aone per vCPU. This indicates that beyond this point, resource contention outweighs any potential gains from additional parallelism.<\/p>\n Running parallel jobs that each load and execute the model independently introduces significant inefficiencies and waste:<\/p>\n A more efficient alternative is to centralize inference execution using a dedicated model inference server. This approach would eliminate redundant model loading and reduce overall system resource utilization.<\/p>\n In the next section we will set up an AI model inference server and assess its impact on resource utilization and runtime performance.<\/p>\n Note:<\/strong>\u00a0We could have modified our multiprocessing-based approach to share a single model across processes (e.g., using\u00a0torch.multiprocessing<\/a>\u00a0or another solution based on\u00a0shared memory<\/a>). However, the inference server demonstration better aligns with real-world production environments, where jobs often run in isolated containers.<\/p>\n The TorchServe setup described in this section loosely follows the\u00a0resnet tutorial<\/a>. Please refer to the official\u00a0TorchServe<\/a>\u00a0documentation for more in-depth guidelines.<\/p>\n The PyTorch environment of our DLAMI comes preinstalled with TorchServe executables. If you are running in a different environment run the following installation command:<\/p>\n The TorchServe Model Archiver packages the model and its associated files into a \u201c.mar<\/em>\u201d file archive, the format required for deployment on TorchServe. We create a TorchServe model archive file based on our model checkpoint file and using the\u00a0default image_classifier handler<\/a>:<\/p>\n We create a TorchServe config.properties file to define how TorchServe should operate:<\/p>\n After completing these steps, our working directory should look like this:<\/p>\n In a separate shell we start our TorchServe inference server:<\/p>\n We define an alternative prediction function that calls our inference service:<\/p>\n Now that inference requests are being processed by a central server, we can scale up parallel processing. Unlike the earlier approach where each process loaded and executed its own model, we have sufficient CPU resources to allow for many more concurrent processes. Here we choose 100 processes in accordance with the default\u00a0job_queue_size\u00a0<\/em>capacity of the inference server:<\/p>\n The performance results are captured in the table below. Keep in mind that the comparative results can vary greatly based on the details of the AI model and the runtime environment.<\/p>\n By using a centralized inference server, not only have we have increased overall throughput by more than 2X, but we have freed significant CPU resources for other computation tasks.<\/p>\n Now that we have effectively demonstrated the benefits of a centralized inference serving solution, we can explore several ways to enhance and optimize the setup. Recall that our experiment was intentionally simplified to focus on demonstrating the utility of inference serving. In real-world deployments, additional enhancements may be required to tailor the solution to your specific needs.<\/p>\n In the next sections we explore two simple ways to enhance our TorchServe-based inference server implementation. We leave the discussion on other enhancements to future posts.<\/p>\n Many model inference service solutions support the option of grouping inference requests into batches. This usually results in increased throughput, especially when the model is running on a GPU.<\/p>\n We extend our TorchServe\u00a0config.properties<\/em>\u00a0file to support batch inference with a batch size of up to 8 samples. Please see the\u00a0official documentation<\/a>\u00a0for details on batch inference with TorchServe.<\/p>\n We append the results in the table below:<\/p>\n Enabling batched inference increases the throughput by an additional 26.5%.<\/p>\n Many model inference service solutions will support creating multiple inference workers for each AI model. This enables fine-tuning the number of inference workers based on expected load. Some solutions support auto-scaling of the number of inference workers.<\/p>\n We extend our own TorchServe setup by increasing the\u00a0 Importantly, we must limit the number of threads allocated to each worker to prevent resource contention. This is controlled by the\u00a0 The modified TorchServe startup sequence appears below:<\/p>\n In the table below we append the results of running with 2, 4, and 8 inference workers:<\/p>\n By configuring TorchServe to use multiple inference workers, we are able to increase the throughput by an additional 36%. This amounts to a 3.75X improvement over the baseline experiment.<\/p>\n This experiment highlights the potential impact of inference server deployment on multi-job deep learning workloads. Our findings suggest that using an inference server can improve system resource utilization, enable higher concurrency, and significantly increase overall throughput. Keep in mind that the precise benefits will greatly depend on the details of the workload and the runtime environment.<\/p>\n Designing the inference serving architecture is just one part of optimizing AI model execution. Please see some of our\u00a0many posts<\/a>\u00a0covering a wide range AI model optimization techniques.<\/p>\n The post The Case for Centralized AI Model Inference Serving<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\n\n
\n
Toy Experiment<\/h2>\n
source \/opt\/pytorch\/bin\/activate<\/code><\/pre>\nStep 1: Creating a TorchScript Model Checkpoint<\/h3>\n
import torch\nfrom torchvision.models import resnet152, ResNet152_Weights\n\nmodel = resnet152(weights=ResNet152_Weights.DEFAULT)\nmodel = torch.jit.script(model)\nmodel.save(\"resnet-152.pt\")<\/code><\/pre>\nStep 2: Model Inference Function<\/h3>\n
\n
import os, time, psutil\nimport multiprocessing as mp\nimport torch\nimport torch.nn.functional as F\nimport torchvision.transforms as transforms\nfrom PIL import Image\n\n\ndef predict(image_id):\n # Limit each process to 1 thread\n MAX_THREADS = 1\n os.environ[\"OMP_NUM_THREADS\"] = str(MAX_THREADS)\n os.environ[\"MKL_NUM_THREADS\"] = str(MAX_THREADS)\n torch.set_num_threads(MAX_THREADS)\n\n # load the model\n model = torch.jit.load('resnet-152.pt').eval()\n\n # Define image preprocessing steps\n transform = transforms.Compose([\n transforms.Resize(256),\n transforms.CenterCrop(224),\n transforms.ToTensor(),\n transforms.Normalize(mean=[0.485, 0.456, 0.406], \n std=[0.229, 0.224, 0.225])\n ])\n\n # load the image\n image = Image.open('kitten.jpg').convert(\"RGB\")\n \n # preproc\n image = transform(image).unsqueeze(0)\n\n # perform inference\n with torch.no_grad():\n output = model(image)\n\n # postproc\n probabilities = F.softmax(output[0], dim=0)\n probs, classes = torch.topk(probabilities, 5, dim=0)\n probs = probs.tolist()\n classes = classes.tolist()\n\n return dict(zip(classes, probs))\n<\/code><\/pre>\nStep 3: Running Parallel Inference Jobs<\/h3>\n
\n
def process_image(image_id):\n print(f\"Processing image {image_id} (PID: {os.getpid()})\")\n predict(image_id)\n\ndef spawn_jobs(total_images, max_concurrent):\n start_time = time.time()\n max_mem_utilization = 0.\n max_utilization = 0.\n\n processes = []\n index = 0\n while index < total_images or processes:\n\n while len(processes) < max_concurrent and index < total_images:\n # Start a new process\n p = mp.Process(target=process_image, args=(index,))\n index += 1\n p.start()\n processes.append(p)\n\n # sample memory utilization\n mem_usage = psutil.virtual_memory().percent\n max_mem_utilization = max(max_mem_utilization, mem_usage)\n cpu_util = psutil.cpu_percent(interval=0.1)\n max_utilization = max(max_utilization, cpu_util)\n\n # Remove completed processes from list\n processes = [p for p in processes if p.is_alive()]\n\n total_time = time.time() - start_time\n print(f\"nTotal Processing Time: {total_time:.2f} seconds\")\n print(f\"Max CPU Utilization: {max_utilization:.2f}%\")\n print(f\"Max Memory Utilization: {max_mem_utilization:.2f}%\")\n\nspawn_jobs(total_images=1000, max_concurrent=32)<\/code><\/pre>\nEstimating the Maximum Number of Processes<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/1muTsRotRMCuLoHOGk6DYFw.png?ssl=1\")
The Inefficiencies of Independent Model Execution<\/h2>\n
\n
TorchServe Setup<\/h2>\n
Installation<\/h3>\n
pip install torchserve torch-model-archiver<\/code><\/pre>\nCreating a Model Archive<\/h3>\n
mkdir model_store\ntorch-model-archiver \n --model-name resnet-152 \n --serialized-file resnet-152.pt \n --handler image_classifier \n --version 1.0 \n --export-path model_store<\/code><\/pre>\nTorchServe Configuration<\/h3>\n
model_store=model_store\nload_models=resnet-152.mar\nmodels={\n \"resnet-152\": {\n \"1.0\": {\n \"marName\": \"resnet-152.mar\"\n }\n }\n}\n\n# Number of workers per model\ndefault_workers_per_model=1\n\n# Job queue size (default is 100)\njob_queue_size=100<\/code><\/pre>\n\u251c\u2500\u2500 config.properties\n\u05ab\u251c\u2500\u2500 kitten.jpg\n\u251c\u2500\u2500 model_store\n\u2502 \u251c\u2500\u2500 resnet-152.mar\n\u251c\u2500\u2500 multi_job.py<\/code><\/pre>\nStarting TorchServe<\/h3>\n
source \/opt\/pytorch\/bin\/activate\ntorchserve \n --start \n --disable-token-auth \n --enable-model-api \n --ts-config config.properties<\/code><\/pre>\nInference Request Implementation<\/h3>\n
import requests\n\ndef predict_client(image_id):\n with open('kitten.jpg', 'rb') as f:\n image = f.read()\n response = requests.post(\n \"http:\/\/127.0.0.1:8080\/predictions\/resnet-152\",\n data=image,\n headers={'Content-Type': 'application\/octet-stream'}\n )\n\n if response.status_code == 200:\n return response.json()\n else:\n print(f\"Error from inference server: {response.text}\")<\/code><\/pre>\nScaling Up the Number of Concurrent Jobs<\/h3>\n
spawn_jobs(total_images=1000, max_concurrent=100)<\/code><\/pre>\nResults<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/1oNoZtyFhBvv7TPdqtZ19Dg.png?ssl=1\")
Next Steps<\/h2>\n
\n
Batch Inference with TorchServe<\/h2>\n
model_store=model_store\nload_models=resnet-152.mar\nmodels={\n \"resnet-152\": {\n \"1.0\": {\n \"marName\": \"resnet-152.mar\",\n \"batchSize\": 8,\n \"maxBatchDelay\": 100,\n \"responseTimeout\": 200\n }\n }\n}\n\n# Number of workers per model\ndefault_workers_per_model=1\n\n# Job queue size (default is 100)\njob_queue_size=100<\/code><\/pre>\nResults<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/1xGheBKZ2wFDOi0BgSroiWQ.png?ssl=1\")
Multi-Worker Inference with TorchServe<\/h2>\n
default_workers_per_model<\/code>\u00a0setting that controls the number of inference workers assigned to our image classification model.<\/p>\nnumber_of_netty_threads<\/code>\u00a0<\/em>setting and by the\u00a0OMP_NUM_THREADS<\/code>\u00a0and\u00a0MKL_NUM_THREADS<\/code>\u00a0environment variables. Here we have set the number of threads to equal the number of vCPUs (8) divided by the number of workers.<\/p>\nmodel_store=model_store\nload_models=resnet-152.mar\nmodels={\n \"resnet-152\": {\n \"1.0\": {\n \"marName\": \"resnet-152.mar\"\n \"batchSize\": 8,\n \"maxBatchDelay\": 100,\n \"responseTimeout\": 200\n }\n }\n}\n\n# Number of workers per model\ndefault_workers_per_model=2 \n\n# Job queue size (default is 100)\njob_queue_size=100\n\n# Number of threads per worker\nnumber_of_netty_threads=4<\/code><\/pre>\nexport OMP_NUM_THREADS=4\nexport MKL_NUM_THREADS=4\ntorchserve \n --start \n --disable-token-auth \n --enable-model-api \n --ts-config config.properties<\/code><\/pre>\nResults<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/1ZRCedYA53M_YZLpbIBRiig.png?ssl=1\")
Summary<\/h2>\n