What is Light GBM?

Light GBM is a gradient boosting framework that uses a tree-based learning algorithm.

How it differs from other tree based algorithm?

**Light GBM grows tree vertically **while another algorithm grows trees horizontally meaning that Light GBM grows tree **leaf-wise **while another algorithm grows level-wise. It will choose the leaf with max delta loss to grow. When growing the same leaf, a Leaf-wise algorithm can reduce more loss than a level-wise algorithm.

Below diagrams explain the implementation of LightGBM and other boosting algorithms

Why Light GBM is gaining extreme popularity?

The size of data is increasing day by day and it is becoming difficult for traditional data science algorithms to give faster results. Light GBM is prefixed as ‘Light’ because of its **high speed. **Light GBM can **handle the large size** of data and **takes lower memory to run**. Another reason why Light GBM is popular is that it **focuses on the accuracy of results**. LGBM also **supports GPU learning** and thus data scientists are widely using LGBM for data science application development.

**No,** it is not advisable to use LGBM on small datasets. Light GBM is **sensitive to overfitting** and can easily overfit small data. There is no threshold on the number of rows but my experience suggests me to use it only for data with 10,000+ rows.

*We briefly discussed the concept of Light GBM, now what about its implementation?*

Implementation of Light GBM is easy, the only complicated thing is parameter tuning. Light GBM covers more than 100 parameters but doesn’t worry, you don’t need to learn all.

It is very important for an implementer to know at least some basic parameters of Light GBM. If you carefully go through the following parameters of LGBM, I bet you will find this powerful algorithm a piece of cake.

Let’s start discussing parameters.

Control Parameters

**max_depth:** It describes the maximum depth of tree. This parameter is used to handle model overfitting. Any time you feel that your model is overfitted, my first advice will be to lower max_depth.

**min_data_in_leaf:** It is the minimum number of records a leaf may have. The default value is 20, the optimum value. It is also used to deal over fitting

**feature_fraction:** Used when your boosting(discussed later) is random forest. 0.8 feature fraction means LightGBM will select 80% of parameters randomly in each iteration for building trees.

**bagging_fraction:** specifies the fraction of data to be used for each iteration and is generally used to speed up the training and avoid overfitting.

**early_stopping_round:** This parameter can help you speed up your analysis. The model will stop training if one metric of one validation data doesn’t improve in the last early_stopping_round rounds. This will reduce excessive iterations.

**lambda: **lambda specifies regularization. The typical value ranges from 0 to 1.

**min_gain_to_split:** This parameter will describe the minimum gain to make a split. It can be used to control a number of useful splits in the tree.

**max_cat_group: **When the number of categories is large, finding the split point on it is easily over-fitting. So LightGBM merges them into ‘max_cat_group’ groups and finds the split points on the group boundaries, default:64

Core Parameters

**Task: **It specifies the task you want to perform on data. It may be either train or predict.

**application: **This is the most important parameter and specifies the application of your model, whether it is a regression problem or a classification problem. LightGBM will by default consider the model as a regression model.

- regression: for regression
- binary: for binary classification
- multiclass: for multiclass classification problem

**boosting:** defines the type of algorithm you want to run, default=gdbt

- gbdt: traditional Gradient Boosting Decision Tree
- rf: random forest
- dart: Dropouts meet Multiple Additive Regression Trees
- goss: Gradient-based One-Side Sampling

**num_boost_round:** Number of boosting iterations, typically 100+

**learning_rate: **This determines the impact of each tree on the final outcome. GBM works by starting with an initial estimate which is updated using the output of each tree. The learning parameter controls the magnitude of this change in the estimates. Typical values: 0.1, 0.001, 0.003…

**num_leaves:** number of leaves in full tree, default: 31

**device: **default: CPU, can also pass GPU

Metric parameter

**metric:** again one of the important parameter as it specifies loss for model building. Below are a few general losses for regression and classification.

- mae: mean absolute error
- mse: mean squared error
- binary_logloss: loss for binary classification
- multi_logloss: loss for multi-classification

IO parameter

**max_bin: **it denotes the maximum number of bin that feature value will bucket in.

**categorical_feature:** It denotes the index of categorical features. If categorical_features=0,1,2 then column 0, column 1 and column 2 are categorical variables.

**ignore_column:** same as categorical_features just instead of considering specific columns as categorical, it will completely ignore them.

**save_binary:** If you are really dealing with the memory size of your data file then specify this parameter as ‘True’. Specifying parameter true will save the dataset to binary file, this binary file will speed your data reading time for the next time.

Knowing and using above parameters will definitely help you implement the model. Remember I said that implementation of LightGBM is easy but parameter tuning is difficult. So let’s first start with implementation and then I will give idea about the parameter tuning.

*Installating LGBM:*

Installing LightGBM is a crucial task. I found **this** as the best resource which will guide you in LightGBM installation.

I am using Anaconda and installing LightGBM on anaconda is a clinch. Just run the following command on your Anaconda command prompt and whoosh, LightGBM is on your PC.

`conda install -c conda-forge lightgbm`

*Dataset:*

This data is very small just 400 rows and 5 columns (specially used for learning purpose). This is a classification problem where we have to predict whether a customer will buy the product from advertise given on the website. I am not explaining dataset as dataset is self-explanatory. You can download dataset from my drive.

Note: The dataset is clean and has no missing value. The main aim behind choosing this much smaller data is to keep the things simpler and understandable.

I am assuming that you all know basics of python. Go through data preprocessing steps, they are fairly easy but if you have any doubt then ask me in the comment, I will get back to you asap.

*Data preprocessing:*

`import numpy as np`

import matplotlib.pyplot as plt

import pandas as pd# Importing the dataset

dataset = pd.read_csv('...input\Social_Network_Ads.csv')

X = dataset.iloc[:, [2, 3]].values

y = dataset.iloc[:, 4].values# Splitting the dataset into the Training set and Test set

from sklearn.cross_validation import train_test_split

x_train, x_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 0)# Feature Scaling

from sklearn.preprocessing import StandardScaler

sc = StandardScaler()

x_train = sc.fit_transform(x_train)

x_test = sc.transform(x_test)

*Model building and training:*

We need to convert our training data into LightGBM dataset format(this is mandatory for LightGBM training).

After creating a converting dataset, I created a python dictionary with parameters and their values. The accuracy of your model totally depends on the values you provide to the parameters.

In the end block of code, I simply trained the model with 100 iterations.

`import lightgbm as lgbd_train = lgb.Dataset(x_train, label=y_train)params = {}`

params['learning_rate'] = 0.003

params['boosting_type'] = 'gbdt'

params['objective'] = 'binary'

params['metric'] = 'binary_logloss'

params['sub_feature'] = 0.5

params['num_leaves'] = 10

params['min_data'] = 50

params['max_depth'] = 10clf = lgb.train(params, d_train, 100)

Few things to notice in parameters:

- Used ‘binary’ as objective(remember this is classification problem)
- Used ‘binary_logloss’ as metric(same reason, binary classification problem)
- ‘num_leaves’=10 (as it is small data)
- ‘boosting type’ is gbdt, we are implementing gradient boosting(you can try random forest)

We can check results either using confusion matrix or directly calculating accuracy

*Code:*

`#Confusion matrixfrom sklearn.metrics import confusion_matrix`

cm = confusion_matrix(y_test, y_pred)#Accuracyfrom sklearn.metrics import accuracy_score

accuracy=accuracy_score(y_pred,y_test)

*Screenshots of the result:*