How to do it...

Let's see how to find optimal hyperparameters:

1. The full code is given in the perform_grid_search.py file that's already provided to you. We start importing the libraries:

from sklearn import svm
from sklearn import model_selection
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report
import pandas as pd
import utilities 

1. Then, we load the data:

input_file = 'data_multivar.txt'
X, y = utilities.load_data(input_file)

1. We split the data into a train and test dataset:

X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.25, random_state=5)

1. Now, we will use cross-validation here, which we covered in the previous recipes. Once you load the data and split it into training and testing datasets, add the following to the file:

# Set the parameters by cross-validation
parameter_grid = {"C": [1, 10, 50, 600],
                  'kernel':['linear','poly','rbf'],
                  "gamma": [0.01, 0.001],
                  'degree': [2, 3]}

1. Let's define the metrics that we want to use:

metrics = ['precision'] 

1. Let's start the search for optimal hyperparameters for each of the metrics:

for metric in metrics:

    print("#### Grid Searching optimal hyperparameters for", metric)
          
    classifier = GridSearchCV(svm.SVC(C=1), 
            parameter_grid, cv=5,scoring=metric,return_train_score=True) 

    classifier.fit(X_train, y_train)

1. Let's look at the scores:

    print("Scores across the parameter grid:")
    GridSCVResults = pd.DataFrame(classifier.cv_results_)
    for i in range(0,len(GridSCVResults)):
        print(GridSCVResults.params[i], '-->', round(GridSCVResults.mean_test_score[i],3))    

1. Let's print the best parameter set:

    print("Highest scoring parameter set:", classifier.best_params_)

1. If you run this code, you will see the following on your Terminal:

#### Grid Searching optimal hyperparameters for precision
Scores across the parameter grid:
{'C': 1, 'degree': 2, 'gamma': 0.01, 'kernel': 'linear'} --> 0.676
{'C': 1, 'degree': 2, 'gamma': 0.01, 'kernel': 'poly'} --> 0.527
{'C': 1, 'degree': 2, 'gamma': 0.01, 'kernel': 'rbf'} --> 0.98
{'C': 1, 'degree': 2, 'gamma': 0.001, 'kernel': 'linear'} --> 0.676
{'C': 1, 'degree': 2, 'gamma': 0.001, 'kernel': 'poly'} --> 0.533
...
...
{'C': 600, 'degree': 2, 'gamma': 0.001, 'kernel': 'linear'} --> 0.676
{'C': 600, 'degree': 2, 'gamma': 0.001, 'kernel': 'poly'} --> 0.9
{'C': 600, 'degree': 2, 'gamma': 0.001, 'kernel': 'rbf'} --> 0.983
{'C': 600, 'degree': 3, 'gamma': 0.01, 'kernel': 'linear'} --> 0.676
{'C': 600, 'degree': 3, 'gamma': 0.01, 'kernel': 'poly'} --> 0.884
{'C': 600, 'degree': 3, 'gamma': 0.01, 'kernel': 'rbf'} --> 0.967
{'C': 600, 'degree': 3, 'gamma': 0.001, 'kernel': 'linear'} --> 0.676
{'C': 600, 'degree': 3, 'gamma': 0.001, 'kernel': 'poly'} --> 0.533
{'C': 600, 'degree': 3, 'gamma': 0.001, 'kernel': 'rbf'} --> 0.983
Highest scoring parameter set: {'C': 10, 'degree': 2, 'gamma': 0.01, 'kernel': 'rbf'}

1. As we can see in the preceding output, it searches for all the optimal hyperparameters. In this case, the hyperparameters are the type of kernel, the C value, and gamma. It will try out various combinations of these parameters to find the best parameters. Let's test it out on the testing dataset:

    y_true, y_pred = y_test, classifier.predict(X_test)
    print("Full performance report:\n")
    print(classification_report(y_true, y_pred))

1. If you run this code, you will see the following on your Terminal:

1. We have previously said that there are different techniques for optimizing hyperparameters. We'll apply the RandomizedSearchCV method. To do this, just use the same data and change the classifier. To the code just seen, we add a further section:

# Perform a randomized search on hyper parameters
from sklearn.model_selection import RandomizedSearchCV
parameter_rand = {'C': [1, 10, 50, 600],
                  'kernel':['linear','poly','rbf'],
                  'gamma': [0.01, 0.001],
                  'degree': [2, 3]}
metrics = ['precision']
for metric in metrics:
    print("#### Randomized Searching optimal hyperparameters for", metric)
    classifier = RandomizedSearchCV(svm.SVC(C=1), 
             param_distributions=parameter_rand,n_iter=30,           
             cv=5,return_train_score=True)
    classifier.fit(X_train, y_train)
    print("Scores across the parameter grid:")
    RandSCVResults = pd.DataFrame(classifier.cv_results_)
    for i in range(0,len(RandSCVResults)):
         print(RandSCVResults.params[i], '-->', 
                 round(RandSCVResults.mean_test_score[i]

1. If you run this code, you will see the following on your Terminal:

#### Randomized Searching optimal hyperparameters for precision
Scores across the parameter grid:
{'kernel': 'rbf', 'gamma': 0.001, 'degree': 2, 'C': 50} --> 0.671
{'kernel': 'rbf', 'gamma': 0.01, 'degree': 3, 'C': 600} --> 0.951
{'kernel': 'linear', 'gamma': 0.01, 'degree': 3, 'C': 50} --> 0.591
{'kernel': 'poly', 'gamma': 0.01, 'degree': 2, 'C': 10} --> 0.804
...
...
{'kernel': 'rbf', 'gamma': 0.01, 'degree': 3, 'C': 10} --> 0.92
{'kernel': 'poly', 'gamma': 0.001, 'degree': 3, 'C': 600} --> 0.533
{'kernel': 'linear', 'gamma': 0.001, 'degree': 2, 'C': 10} --> 0.591
{'kernel': 'poly', 'gamma': 0.01, 'degree': 3, 'C': 50} --> 0.853
{'kernel': 'linear', 'gamma': 0.001, 'degree': 2, 'C': 600} --> 0.591
{'kernel': 'poly', 'gamma': 0.01, 'degree': 3, 'C': 10} --> 0.844
Highest scoring parameter set: {'kernel': 'rbf', 'gamma': 0.01, 'degree': 3, 'C': 600}

1. Let's test it out on the testing dataset:

 print("Highest scoring parameter set:", classifier.best_params_)
 y_true, y_pred = y_test, classifier.predict(X_test)
 print("Full performance report:\n")
 print(classification_report(y_true, y_pred))
 

1. The following results are returned: