TikTok has become a rising app and an important information source for journalists. Protests are organized on the platform — in June teens and K-Pop stans trended as they falsely registered for a Trump rally to artificially sell out the event. Live protest footage has also been recorded and shared on the app. In addition to political activity, the app has also been used by QAnon to spread conspiracy theories. When a conspiracy theory that Wayfair was selling children started to trend, Wayfair stock prices started to drop.
However despite the importance of the app, it’s notoriously hard to monitor. Unlike Twitter and Facebook, it’s primary media are videos. Though there are hashtags associated with each video, oftentimes they are only loosely related to the topic of the video. In fact, many people add trending hashtags regardless of whether or not they relate to their video in the hopes of making it someone’s For You Page, essentially a user’s home feed of TikTok.
This tutorial shows you how to create a deep learning model to recognize a specific face on the app. The face I have chosen in President Trump, however any face can be used. In fact, any image can be used. For example if you wanted to train the model to recognize confederate flags on the app, you can just swap the training data for images of confederate flags). I will then show you how this model can then be used by a separate python script that will call a TikTok scraping api, and which can be run in the background of any computer, monitoring for potential appearances of Trump on the app. This will allow journalists to be alerted every time the president is trending on the app.
This project has two steps:
- Creating the deep learning model
- Connecting the deep learning model to TikTok api
For Step 1, we will train and create the model from a python notebook on Google Colab. Google Colab is free and we don’t require a particularly large amount of cpu to train our model, so no need to sign up for the Colab Pro account.
Download the training images and mapping from this repository:
And upload them all into a folder called Data_deeplearning into your google drive. These training images are captures taken from TikTok videos that I found by searching for videos tagged with Trump. Notice some of them include trump and some of them do not. I’ve gone through the trouble of labeling them so you don’t have to! (Trust me this was by far the most boring and tedious part of this entire project.)
Then, create a Colab notebook in your google drive and mount your notebook onto your drive, then change directory into your Data_deeplearning folder. This will allow your notebook to access all the images in that folder.
from google.colab import drive
import osdrive.mount('/content/drive')path = "/content/drive/MyDrive/Data_deeplearning"os.chdir( path )
We now need to convert the images into numpy arrays so that our deep learning model can read them later on. To do this, we will read in the file “mapping_trump.csv” in the DeepLearningTikTok directory using pandas, then using that dataframe we will iterate over each image and convert it from an image to a numpy array. We will do the same thing for the labels. Note that from the label you can see 1 means Trump is in the picture and 0 means there is no Trump in that picture.
X = [ ] # creating an empty arrayfor img_name in data.Image_ID: img = plt.imread('' + img_name) X.append(img) # storing each image in array XX = np.array(X) # converting list to arrayy = data.Class #set the labelsdummy_y = np_utils.to_categorical(y) # one hot encoding Classes, we need this to match output shape of modelimage = []for i in range(0,X.shape[0]): a = resize(
X[i], preserve_range=True,output_shape(224,224)
).astype(int) image.append(a)X = np.array(image)
Because the data was hand-labeled, it’s a relatively small dataset in the deep learning realm. To synthetically increase the size of the dataset we use data augmentation, a technique in deep learning that creates slightly modified copies of the images in order to increase your data set size. Using ImageDataGenerator from the keras library, we can rescale, add to the brightness range, shear some of the photo, zoom in, and horizontally flip the photos.
from tensorflow.keras.preprocessing.image import ImageDataGenerator
import numpy as npdatagen = ImageDataGenerator(
rescale=1./255,
brightness_range=[0.2,1.0],
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True
)datagen.fit(X)X_aug = []
y_aug = []i = 0for batch in datagen.flow(X,dummy_y, batch_size = 1):X_aug.append(batch[0][0])
y_aug.append(batch[1][0])
i+=1
if i>2524:
breakX = np.append(X,np.array(X_aug),axis=0)
dummy_y = np.append(dummy_y,np.array(y_aug),axis=0)
We will first run the images through the VGG16 base model. VGG16 is a pre-trained model that takes in (224,224) RGB images and converts them into features. It comes out-of-the-box from the keras library and has been trained on millions of images from ImageNet. To read more about VGG16 model check out this site: https://neurohive.io/en/popular-networks/vgg16/
We only run the X data through the VGG16 model to convert it to features because the y data is simply the labels, 0 or 1.
from keras.applications.vgg16 import preprocess_input
from keras.models import Sequential
from keras.applications.vgg16 import VGG16
from keras.layers import Dense, InputLayer, Dropout, Flatten
from sklearn.model_selection import train_test_splitX = preprocess_input(X)X_train, X_valid, y_train, y_valid = train_test_split(X, dummy_y, test_size=0.3, random_state=42) # preparing the validation set
base_model = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3)) X_train = base_model.predict(X_train)X_valid = base_model.predict(X_valid)X_train.shape, X_valid.shape
In this section we will build a custom deep learning model — this will be the model we use with the TikTok api to predict whether or not a trump appears in a video. In this part, we actually have a lot of freedom to change the structure, add layers, subtract neurons, etc. The important thing is that the input layer of your model matches the dimensions of your X data (which you can see by printing X.shape), that you add a flatten layer before your output layer (so that your multidimensional images can match the dimensions of your label), and that your final layer has the same number of neurons as the number of possible labels in your dataset (in our case it is 0 or 1 so the number of neurons is 2).
#Building the modelmodel = Sequential()
model.add(InputLayer((7,7,512,))) model.add(Flatten())model.add(Dense(units=1024, activation='sigmoid')) model.add(Dropout(0.5)) # adding dropout prevents overfittingmodel.add(Dense(units=512, activation='sigmoid')) # hidden layermodel.add(Dropout(0.5)) # adding dropoutmodel.add(Dense(units=512, activation='sigmoid')) # hidden layermodel.add(Dropout(0.5)) # adding dropoutmodel.add(Dense(units=256, activation='sigmoid')) # hidden layermodel.add(Dropout(0.5)) # adding dropoutmodel.add(Dense(2, activation='softmax'))
Now we train the model using our training images from before:
model.fit(X_train, y_train, steps_per_epoch=len(X_train) / 32, epochs = 100, validation_data = (X_valid, y_valid))
Finally save the model so that we can download to a desktop and connect it with the TikTok api:
model.save(‘trump_model_final.h5’)
Now you have a model that can recognize when Trump is in tiktok videos! The next step is to connect it to a tiktok api.
Now for this part you will need to clone the github repository:
After cloning this repo into your local, move your model (trump_model_final.h5) that you downloaded from the previous step into this repo. Then, you can test it out by calling it with the name of a video after. For example, if you want to test with the biden.mp4 that is placed in the repo as an example, simply call:
python3 run_model.py Biden.mp4
You should see this output:
Prediction: [0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
Each number stands for a different frame taken of the video. The 0’s mean the model did not detect any images of Trump, and the 1’s mean that there was a Trump detection in that frame. Notice that in this video, there is only images of Biden, however the model still picked up some false positives. This is a good reminder that this is a very rudimentary model, and to improve accuracy you need to expand the training dataset. To account for this, I only consider when a model has detected Trump in a third of all the frames to be a “positive trump detection”. If you want to try the model with the example Trump video in the repo, run:
python3 run_model.py Trump.mp4
You should now see the output to be:
Prediction: [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
====================Trump is in this video==================
Now that you have the model which recognizes Trump, the python script which feeds an mp4 file video into the model and outputs whether it detects Trump, the only thing missing is the TikTok api. For this part I used drawrowfly’s open source tiktok scraper.
Thank you to drawrowfly for their open source work!
I’ve written a really rudimentary bash shell script that just calls the TikTok api scraper to get the most recent trending video, and then feeds that filepath to the python script, which will evaluate if more than 1/3 of the frames has Trump in it. To convert this into a 24/7 running file, you can just wrap it all in a while loop or write a cron job to regularly schedule it to run.
The dataset that is included in the git repo under Data_deeplearning is a combination of tiktok videos I took captures of that included Trump, tiktok videos I took captures of that included other notable politicians such as Biden, tiktok videos that were trending, and google images of Trump. I found that using only one of the listed sources made the model less effective.
To learn more about deep learning, you can read more here: