![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AK3RRU2maaQdGOIvT9tMFIw.png?ssl=1\")
Machine Learning for Real-Time Bidding<\/h4>\n
The past few years were a revolution for the mobile advertising and gaming industries, with the broad adoption of neural networks for advertising tasks, including click prediction. This migration occurred prior to the success of Large Language Models (LLMs) and other AI innovations, but is building on the momentum of this wave. The mobile gaming industry is spending billions on User Acquisition every year, and top players in this space such as Applovin have market caps of over $100B. In this post, we\u2019ll discuss a conventional ML approach for click prediction, offer motivations for the migration to deep learning for this task, provide a hands-on example of the benefits of this approach using a data set from Kaggle<\/a>, and detail some of the enhancements that this approach provides.<\/p>\n Most large tech companies in the AdTech space are likely using deep learning for predicting user behavior. Social Media platforms have embraced the migration from classic machine learning (ML) to deep learning, as indicated by this Reddit post<\/a> and this LinkedIn post<\/a>. In the mobile gaming space Moloco<\/a>, Liftoff<\/a>, and Applovin <\/a>have all shared details on their migration to deep learning or hardware acceleration to improve their user acquisition platforms. Most Demand Side Platforms (DSPs) are now looking to leverage neural networks to improve the value that their platforms provide for mobile user acquisition.<\/p>\n We\u2019ll start by discussing logistic regression as an industry standard for predicting user actions, discuss some of the shortfalls of this approach, and then showcase deep learning as a solution for click prediction. We\u2019ll provide a deep dive on implementations for both a classic ML notebook<\/a> and deep learning notebook<\/a> for the task of predicting if a user is going to click on an ad. We won\u2019t dive into the state of the art, but we will highlight where deep learning provides many benefits.<\/p>\n All images in this post, with the exception of the header image, were created from by the author in the notebooks linked above. The Kaggle data set that we explore in this post has the CC0: Public Domain<\/a>\u00a0license.<\/p>\n One of the goal types that DSPs typically provide for user acquisition is a cost per click model, where the advertiser is charged each time that the platform serves an impression on a mobile device and the user clicks. We\u2019ll focus on this goal type to keep things simple, but most advertisers prefer goal types focused on driving installs or acquiring users that will spend money in their\u00a0app.<\/p>\n In programmatic bidding, a DSP is integrated with one or more ad exchanges, which provide inventory for the platform to bid on. Most exchanges use a version of the OpenRTB specification<\/a> to send bid requests to DSPs and get back responses in a standardized format. For each ad request from a Supply Side Platform (SSP), the exchange runs an auction and the DSP that responds with the highest price wins. The exchange then provides the winning bid response to the SSP, which may result in an ad impression on a mobile\u00a0device.<\/p>\n In order for a DSP to integrate with an ad exchange, there is an onboarding process to make sure that the DSP can meet the technical requirements of an exchange, which typically requires DSPs to respond to bid requests within 120 milliseconds. What makes this a huge challenge is that some exchanges provide over 1 million bid requests per second, and DSPs are usually integrating with several exchanges. For example, Moloco responds<\/a> to over 5 million requests per second (QPS) during peak capacity. Because of the latency requirements and massive scale of requests, it\u2019s challenging to use machine learning for user acquisition within a DSP, but it\u2019s also a requirement in order to meet advertiser goals.<\/p>\n In order to make money as a DSP you need to be able to deliver ad impressions that meet your advertiser goals, while also generating net revenue. To accomplish this, a DSP needs to bid less than the expected value that an impression will deliver, while also bidding high enough to exceed the bid floor of a request and win in auctions against other DSPs. A demand-side platform is billed per impression shown, which corresponds to a CPM (cost per impression) model. If the advertiser goal is a target cost per click (CPC), then the DSP needs to translate the CPC value to a CPM value for bidding. We can do this using machine learning and predicting the likelihood of a user to click on an impression, which we call p_ctr<\/em>. We can this calculate a bid price as\u00a0follows:<\/p>\n We use the likelihood of a click event to convert from cost per click to cost per impression and then apply a bid shade with a value of less than 1.0 to make sure that we are delivering more value for advertisers than we are paying to the ad exchange for serving the impression.<\/p>\n In order for a click prediction model to perform well for programmatic user acquisition, we want a model that has the following properties:<\/p>\n Many ad tech platforms started with logistic regression for click prediction, because they work well for the first 3 desired properties. Over time, it was discovered that deep learning models could perform better than logistic regression on the bias goal, with neural networks being better at discriminating between click and no-click impressions. Additionally, neural networks can use batch evaluation and align will with the fourth property of parallel evaluation.<\/p>\n DSPs were able to push logistic regression models pretty far, which is what we\u2019ll cover in the next section, but they do have some boundaries in their application to user acquisition. Deep neural networks (DNN) can overcome some of these issues, but present new challenges of their\u00a0own.<\/p>\n Ad Tech companies have been using logistic regression for more than a decade for click prediction. For example, Facebook presented using logit in combination with other models at ADKDD 2014<\/a>. There are many different ways of using logistic regression for click prediction, but I\u2019ll focus on a single approach I worked on in the past called Big Logistic. The general idea was to turn all of your features into tokens, create combinations of tokens to represent crosses or feature interactions, and then create a list of tokens that you use to convert your input features into a sparse vector representation. It\u2019s an approach where every feature is 1-hot encoded and all of the features are binary, which helps simplify hyperparameter tuning for the click model. It\u2019s an approach that can support numeric, categorical, and many-hot features as\u00a0inputs.<\/p>\n To determine what this approach looks like in practice, we\u2019ll provide a hands-on example of training a click prediction model using the CTR In Advertisement<\/a> Kaggle data set. The full notebook for feature encoding, model training and evaluation is available here<\/a>. I used Databricks, PySpark, and MLlib for this pipeline.<\/p>\n The dataset provides a training data set with labels and a test data set without labels. For this exercise we\u2019ll split the training file into train and test groups, so that we have labels available for all records. We create a 90\/10% split where the train set has 414k records and test has 46k records. The data set has 15 columns, which includes a label, 2 columns that we\u2019ll ignore (session_id and user_id) and 12 categorical values that we\u2019ll use as features in our model. A few sample records are shown in the table\u00a0above.<\/p>\n The first step we\u2019ll perform is tokenizing the data set, which is a form of 1-hot encoding. We convert each column to a string value by concatenating the feature name and feature value. For example, we would create the following tokens for the first row in the above\u00a0table:<\/p>\n [\u201cproduct_c\u201d, \u201ccampaign_id_359520\u201d, \u201cwebpage_id_13787\u201d,\u00a0..]<\/em><\/p>\n For null values, we use \u201cnull\u201d as the value, e.g. \u201cproduct_null\u201d. We also create all combinations of two features, which generates additional tokens:<\/p>\n [\u201cproduct_c*campaign_id_359520\u201d, \u201c\u201d, \u201cproduct_c*webpage_id_13787\u201d, \u201ccampaign_id_359520*webpage_id_13787\u201d,..]<\/em><\/p>\n We use a UDF on the PySpark dataframe to convert the 12 columns into a vector of strings. The resulting dataframe includes the token list and label, as shown\u00a0below.<\/p>\n We then create a top tokens list, assign an index to each token in this list, and use the mapping of token name to token index to encode the data. We limited our token list to values where we have at least 1000 examples, which resulted in roughly 2,500\u00a0tokens.<\/p>\n We then apply this token list to each record in the data set to convert from the token list to a sparse vector representation. If a record includes the token for an index, the value is set to 1, and if the token is missing the value is set to 0. This results in a data set that we can use with MLlib to train a logistic regression model.<\/p>\n We split the dataset into train and test groups, fit the model on the train data set, and then transform the test data set to get predictions.<\/p>\n This process resulted in the following offline metrics, which we\u2019ll compare to a deep learning model in the next\u00a0section.<\/p>\n The AUC metrics don\u2019t look great, but there isn\u2019t much signal in the data set with the features that we explored, and other participants in the Kaggle competition generally had lower ROC metrics. One other limitation of the data set is that the categorical values are low cardinality, with only a small number of distinct values. This resulted in a low parameter count, with only 2,500 features, which limited the bias of the\u00a0model.<\/p>\n Logistic regression works great for click prediction, but where we run into challenges is when dealing with high cardinality features. In mobile ad tech, the publisher app, where the ad is rendered, is a high cardinality feature, because there are millions of potential mobile apps that may render an ad. If we want to include the publisher app as a feature in our model, and are using 1-hot encoding, we are going to end up with a large parameter count. This is especially the case when we perform feature crosses between the publisher app and other high cardinality features, such as the device\u00a0model.<\/p>\n I\u2019ve worked with logistic regression click models that have more than 50 million parameters. At this scale, MLlib\u2019s implementation of logistic regression runs into training issues, because it densifies the vectors in its training loop. To avoid this bottleneck, I used the Fregata library<\/a>, which performs gradient descent using the sparse vector directly in a model averaging strategy.<\/p>\n The other issue with large click models is model inference. If you include too many parameters in your logit model, it may be slow to evaluate, significantly increasing your model serving\u00a0costs.<\/p>\n Deep learning is a good solution for click models, because it provides methods for working efficiently with very sparse features with high cardinality. One of the key layers that we\u2019ll use in our deep learning model is an embedding layer, which takes a categorical feature as an input and a dense vector as an output. With an embedding layer, we learn a vector for each of the entries in our vocabulary for a categorical feature, and the number of parameters is the size of the vocabulary times the output dense vector size, which we can control. Neural networks can reduce the parameter count by creating interactions between the dense layers output of embeddings, rather than making crosses between the sparse 1-hot encoded approach used in logistic regression.<\/p>\n Embedding layers are just one way that neural networks can provide improvements over logistic regression models, because deep learning frameworks provide a variety of layer types and architectures. We\u2019ll focus on embeddings for our sample model to keep things simplistic. We\u2019ll create a pipeline for encoding the data set into TensorFlow Records and then train a model using embeddings and cross layers to perform click prediction. The full notebook for data preparation, model training and evaluation is available here<\/a>.<\/p>\n The first step that we perform is generating a vocabulary for each of the features that we want to encode. For each feature, we find all values with more than 100 instances, and everything else is grouped into an out-of-vocab (OOV) value. We then encode all of the categorical features and combine them into a single tensor named int<\/em>, as shown\u00a0below.<\/p>\n We then save the Spark dataframe as TensorFlow records to cloud\u00a0storage.<\/p>\n We then copy the files to the driver node and create TensorFlow data sets for training and evaluating the\u00a0model.<\/p>\n We then create a Keras model, where the input layer is an embedding layer per categorical feature, we have two hidden cross layers, and a final output layer that is a sigmoid activation for the propensity prediction.<\/p>\n The resulting model has 7,951 parameters, which is about 3 times the size of our logistic regression model. If the categories had larger cardinalities, then we would expect the parameter count of the logit model to be higher. We train the model for 40\u00a0epochs:<\/p>\n We can now compare the offline metrics between our logistic regression and DNN\u00a0models:<\/p>\n We do see improvements to the log loss metric where lower is better and the AUC metrics where higher is better. The main improvement is to the precision-recall (PR) AUC metric, which may help the model perform better in auctions. One of the issues with the DNN model is that the model calibration is worse, and the DNN average predicted value is further off than the logistic regression model. We would need to do a bit more model tuning to improve the calibration of the\u00a0model.<\/p>\n We are now in the era of deep learning for ad tech and companies are using a variety of architectures to deliver advertiser goals for user acquisition. In this post, we showed how migrating from logistic regression to a simple neural network with embedding layers can provide better offline metrics for a click prediction model. Here are some additional ways we could leverage deep learning to improve click prediction:<\/p>\n Thanks for\u00a0reading!<\/p>\n Ben Weber is a machine learning engineer with over a decade of experience in gaming and ad tech with prior roles at Zynga, Microsoft, Amazon, and Electronic Arts.<\/p>\n Deep Learning for Click Prediction in Mobile AdTech<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\nCost Per Click\u00a0Modeling<\/h3>\n
cpm = target_cpc * p_ctr
bid_price = cpm * bid_shade<\/pre>\n\n
<\/strong>We want a click model that is highly discriminative and able to differentiate between impressions unlikely to result in a click and ones that are highly likely to result in a click. If a model does not have sufficient bias, it won\u2019t be able to compete with other DSPs in auctions.<\/li>\n
<\/strong>We want the predicted and actual conversion rates of the model to align well for the ad impressions the DSP purchases. This means we have a preference for models where the output can be interpreted as a probability of a conversion occurring. Poor calibration will result in inefficient spending. A sample calibration plot is shown\u00a0below.<\/li>\n
<\/strong>We want to reduce our compute cost when bidding on millions of requests per second and have models that are fast to inference.<\/li>\n
Ideally, we want to be able to run model inference in parallel to improve throughput. For a single bid request, a DSP may be considering hundreds of campaigns to bid for, and each one needs a p_ctr<\/em>\u00a0value.<\/li>\n<\/ol>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/966\/1%2AbeCmyaZdfGTMVz2GAY92Xw.png?ssl=1\")
The Big Logistic\u00a0Era<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2A8ydYYLsyCeM_zsM66ZP5EQ.png?ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2ADrOviK1E8E3taYh68Hhwtg.png?ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/522\/1%2ASC-Q8P6IjpLrO3XaXrCojA.png?ssl=1\")
<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2A7SzMSdMX0UN4nDzvenM68A.png?ssl=1\")
classifier = LogisticRegression(featuresCol = 'features',
labelCol = 'label', maxIter = 50, regParam = 0.01, elasticNetParam = 0)
lr_model = classifier.fit(train_df)
pred_df = lr_model.transform(test_df).cache()<\/pre>\nActual Conv: 0.06890
Predicted Conv: 0.06770
Log Loss: 0.24795
ROC AUC: 0.58808
PR AUC: 0.09054<\/pre>\nThe Deep Learning\u00a0Era<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/596\/1%2AZPo6bKjg3yG6q6TH0WAZNg.png?ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/534\/1%2ASWCU68o2Dj_aU7AIib5ASw.png?ssl=1\")
output_path = \"dbfs:\/mnt\/ben\/kaggle\/train\/\"
train_df.write.format(\"tfrecords\").mode(\"overwrite\").save(output_path)<\/pre>\ndef getRecords(paths):
features = {
'int': FixedLenFeature([len(vocab_sizes)], tf.int64),
'label': FixedLenFeature([1], tf.int64)
}
@tf.function
def _parse_example(x):
f = tf.io.parse_example(x, features)
return f, f.pop(\"label\")
dataset = tf.data.TFRecordDataset(paths)
dataset = dataset.batch(10000)
dataset = dataset.map(_parse_example)
return dataset
training_data = getRecords(train_paths)
test_data = getRecords(test_paths)<\/pre>\ncat_input = tf.keras.Input(shape=(len(vocab_sizes)),
name = \"int\", dtype='int64')
input_layers = [cat_input]
cross_inputs = []
for attribute in categories_index:
index = categories_index[attribute]
size = vocab_sizes[attribute]
category_input = cat_input[:,(index):(index+1)]
embedding = keras.layers.Flatten()
(keras.layers.Embedding(size, 5)(category_input))
cross_inputs.append(embedding)
cross_input = keras.layers.Concatenate()(cross_inputs)
cross_layer = tfrs.layers.dcn.Cross()
crossed_ouput = cross_layer(cross_input, cross_input)
cross_layer = tfrs.layers.dcn.Cross()
crossed_ouput = cross_layer(cross_input, crossed_ouput)
sigmoid_output=tf.keras.layers.Dense(1,activation=\"sigmoid\")(crossed_ouput)
model = tf.keras.Model(inputs=input_layers, outputs = [ sigmoid_output ])
model.summary()<\/pre>\nmetrics=[tf.keras.metrics.AUC(), tf.keras.metrics.AUC(curve=\"PR\")]
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
loss=tf.keras.losses.BinaryCrossentropy(), metrics=metrics)
history = model.fit(x = training_data, epochs = 40,
validation_data = test_data, verbose=0)<\/pre>\n Logit DNN
Actual Conv: 0.06890 0.06890
Predicted Conv: 0.06770 0.06574
Log Loss: 0.24795 0.24758
ROC AUC: 0.58808 0.59284
PR AUC: 0.09054 0.09249
<\/pre>\nWhat\u2019s Next?<\/strong><\/h3>\n
\n
<\/strong>We can use models such as BERT<\/a> to convert app store descriptions into vectors that we can use as input to the click\u00a0model.<\/li>\n
<\/strong>We could explore the DCN<\/a> and TabTransformer<\/a> architectures.<\/li>\n
<\/strong>We could use img2vec<\/a> to create input embeddings from creative\u00a0assets.<\/li>\n<\/ol>\n<\/p>\n
\n