Applications of scikit-learn for geospatial and non-geospatial data

This post is the record of an in-class tutorial from April 2019, introducing scikit-learn for both geospatial and non-geospatial data. It is preserved as it was taught — the Anaconda setup, pinned package versions, and scikit-learn API calls below all reflect the ecosystem as it stood at the time, and the code will not run as-is in a modern Python environment. Treat it as a snapshot of the workshop rather than a maintained tutorial.

Installation

To prepare for the tutorial, please run the following line in your Anaconda prompt:

conda create --name skltutorial python=3.6 spyder scikit-learn matplotlib pandas

Alternatively, you can choose to install scikit-learn into a Python 3.6 environment from the other tutorials, but be aware that this may cause problems with software already installed in that environment. It is not recommended to install scikit-learn into your arcpy environment.

If you choose to attempt this method, simply activate it in the Anaconda prompt, and install the following packages, if they are not already installed:

conda activate python36-geospatial-2019

conda install scikit-learn matplotlib pandas

Luckily for you, the data we will be working with in this tutorial is built into scikit-learn, so there is no need to download any additional files!

What is scikit-learn, and what can it do?

Scikit-learn is a free Python based machine learning library which works in tandem with Scipy and Numpy. Using either built-in or user-defined datasets, scikit-learn can perform functions such as outlier detection, stock price visualization, clustering, classifications, and even facial recognition.

Types of Classification

An unsupervised classification is used to make inferences on an unlabelled input. From this input data, the program is looking to find patterns or grouping in the data. This is particularly useful in EDA (exploratory data analysis). Before the data is input, the program knows nothing about the input OR output. No training data is used.

A supervised classification involves the use of labelled training data. Due to the data being labelled, the program has an idea of the desired output. With the training data being used as a reference tool, inferences can be made about additional data inputs.

Supervised Learning

Handwriting Recognition

This portion of the tutorial is simply a demonstration of handwriting recognition, so there is no need to follow along! However, if you wish to do so, the code we are using will still be available below. The class will follow along with the geospatial portion of the tutorial, utilizing the California Housing dataset.

Utilizing the built-in digits dataset, we can show an example of OCR/handwriting recognition.

print(__doc__)
import matplotlib.pyplot as plt
from sklearn import datasets, svm, metrics

# The digits dataset
digits = datasets.load_digits()

The digits dataset is composed of 8x8 images. By checking the ‘images’ attribute of the dataset, we can see how they are formatted. Digits.target is the portion of the dataset which stores what the digits are ‘supposed’ to look like. Here, we will train it to recognize 4 digits.

images_and_labels = list(zip(digits.images, digits.target))
for index, (image, label) in enumerate(images_and_labels[:4]):
    plt.subplot(2, 4, index + 1)
    plt.axis('off')
    plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
    plt.title('Training: %i' % label)
    
#This data required a classifier, which means the data needs to be put into a matrix
#for this, we need to 'flatten' the data    
n_samples = len(digits.images)
data = digits.images.reshape((n_samples, -1))

# creating a vector classifier
classifier = svm.SVC(gamma=0.001)

# The digits on the first half of the digits
#predict the value of the digit on the second half:
classifier.fit(data[:n_samples // 2], digits.target[:n_samples // 2])

expected = digits.target[n_samples // 2:]
predicted = classifier.predict(data[n_samples // 2:])

#prints the confusion matrix
print("Confusion matrix:\n%s" % metrics.confusion_matrix(expected, predicted))

#prints the images and predictions using enumeration
images_and_predictions = list(zip(digits.images[n_samples // 2:], predicted))
for index, (image, prediction) in enumerate(images_and_predictions[:4]):
    plt.subplot(2, 4, index + 5)
    plt.axis('off')
    plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
    plt.title('Prediction: %i' % prediction)

plt.show()

Using K-nearest Neighbors to classify Wines

from __future__ import print_function
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_wine
from sklearn.neighbors import KNeighborsClassifier

Load_wine is a built in sklearn dataset that is stored as a utils.bunch Five sections in dataset: data, target, target_names, metadata, feature_names By inputting return_X_y = True, we are only extracting the target and it’s associated data

wine = load_wine()
print(wine.target_names)
print(wine.feature_names)
X, y = load_wine(return_X_y=True)

We are splitting our existing data into training and testing datasets. 70% will be used to train the data. 30% will be used for testing. Purpose is to train our classifier and test different data with it.

X_train, X_test, y_train, y_test = train_test_split(
        X, # data records
        y, # targets
        test_size = 0.3) # size of testing pool

KNN allows us to approximate our test data in accordance to our training data. Depending on how many ‘neighbors’ we choose, a test data point can be predicted based on similar data

knn = KNeighborsClassifier(
        # Decides how many data points to relate to
        n_neighbors=2)
        
# Fitting our KNN classifier to our training data set
knn.fit(X_train, y_train)
# creates an array of y classification labels based on provide data. 
# created new sample based on mean data..arbitrary
sample = np.array([13, 2.34, 2.36, 19.5, 99.7, 2.29, 
                   2.03, 0.36, 1.59, 5.1, 0.96, 2.61, 746])
# original data is a list of a list...so we need to reshape our arbitrary sample to match the shape
sample = sample.reshape(1,-1)
new_prediction = knn.predict(sample)
print("Prediction: {}".format(new_prediction))
score = knn.score(X_test, y_test)
print('score = {}'.format(score))
# Switch neighbors from 1, 2, 3
del knn

# Setup arrays to store train and test accuracies
neighbors = np.arange(1, 9)
train_accuracy = np.empty(len(neighbors))
test_accuracy = np.empty(len(neighbors))

# Loop over different values of k
for i, k in enumerate(neighbors):
    # Setup a k-NN Classifier with k neighbors: knn
    knn = KNeighborsClassifier(n_neighbors=k)
    # Fit the classifier to the training data
    knn.fit(X_train, y_train)
    #Compute accuracy on the training set
    train_accuracy[i] = knn.score(X_train, y_train)
    #Compute accuracy on the testing set
    test_accuracy[i] = knn.score(X_test, y_test)

# Generate plot
plt.title('k-NN: Varying Number of Neighbors')
plt.plot(neighbors, test_accuracy, label = 'Testing Accuracy')
plt.plot(neighbors, train_accuracy, label = 'Training Accuracy')
plt.legend()
plt.xlabel('Number of Neighbors')
plt.ylabel('Accuracy')
plt.show()
del knn

Working with the California Housing Dataset

Picking a Variable for regression

Lasso regression can be useful to decide which variable most strongly predicts the average number of rooms in a house.

import matplotlib.pyplot as plt
import os
import pandas as pd
from sklearn.datasets.california_housing import fetch_california_housing
from sklearn.linear_model import Lasso

downloads = os.path.expanduser("~\Downloads")

#create a function to fit the model and plot our results
def fit_and_plot():
    '''runs lasso regression and creates a plot to test
    how strongly different variables predict Average Rooms'''
    lasso = Lasso(alpha = 0.1)
    lasso_coef = lasso.fit(X, y).coef_
    
    plt.plot(range(len(names)), lasso_coef)
    plt.xticks(range(len(names)), names, rotation =     60)
    plt.ylabel('Coefficients')
    plt.savefig(downloads +'\lasso.png', dpi=300, bbox_inches='tight')
    plt.show()

#download the california housing data
cali = fetch_california_housing()
#populate a data frame with the downloaded data
df = pd.DataFrame(data = cali.data, columns = cali.feature_names)

#looking for what variable most strongly predicts Average Rooms
y = df['AveRooms'].values
#transform y from 1 dimensional to two dimensional numpy array
#scikit-learn expects data in this form
y = y.reshape(-1, 1)
#populate X without Y, Latitude or Longitude
#lat and long are too complicated to predict on
X = df.drop(['AveRooms', 'Latitude', 'Longitude'], axis=1).values
#populate a list of column names
names = df.drop(['AveRooms', 'Latitude', 'Longitude'], axis=1).columns

#run lasso regression and plot coefficients
fit_and_plot()

Now we look at the graph and evaluate where we are.

Average number of bedrooms is a strong predictor for average Rooms, but its kind of the same figure, so we will drop this value, and rerun the regression

X = df.drop(['AveRooms', 'AveBedrms', 'Latitude', 'Longitude'], axis=1).values
names = df.drop(['AveRooms', 'AveBedrms', 'Latitude', 'Longitude'], axis=1).columns

#rerun regression and plot with Average Bedrooms removed
fit_and_plot()

Linear Regression

Using linear regression to predict the average number of rooms in a house for each block

import matplotlib.pyplot as plt
import os
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.datasets.california_housing import fetch_california_housing
from sklearn import linear_model

downloads = os.path.expanduser("~\Downloads")

#download the california housing data
cali = fetch_california_housing()
#populate a data frame with the downloaded data
df = pd.DataFrame(data = cali.data, columns = cali.feature_names)

#looking for how strongly median income predicts Average Rooms
y = df['AveRooms'].values
#transform y from 1 dimensional to two dimensional numpy array
#scikit-learn expects data in this form
y = y.reshape(-1, 1)

#pull Median Income out into its own X variable
X_medinc = df['MedInc'].values
#and reshape it as above
X_medinc = X_medinc.reshape(-1, 1)

#split our data into testing and training sets
#Pull 30% of our data aside for testing to validate our model
X_train, X_test, y_train, y_test = train_test_split(X_medinc, y, test_size = 0.3, random_state = 42)

#instantiate our model
lm = linear_model.LinearRegression()
#fit our model to X training set (median income) and y training set (average rooms)
lm.fit(X_train,y_train)

#get an R-squared score on the accuracy of our model
print("Our R-squared is {}".format(lm.score(X_test, y_test)))

#Plot of our dataset and fit line
prediction_space = np.linspace(min(X_medinc), max(X_medinc))
plt.scatter(X_medinc, y)
plt.xlabel('Median Income')   
plt.ylabel('Number of Rooms')
plt.plot(prediction_space, lm.predict(prediction_space), color = 'black', linewidth = 3)
plt.savefig(downloads +'\scatter.png', dpi=300, bbox_inches='tight')
plt.show()

Here we have a plot of our full data set. It appears that we may have an interesting pattern, but it’s hard to tell because there are so many outliers in the income data.

Lets limit the income to $15,000 to see what that large cluster at the bottom looks like. Add the following line to the previous plot right before the plt.show()

plt.ylim([0, 15])

create a Geospatial Visualization of our data

plt.scatter(df.Latitude, df.Longitude, alpha = 0.2, \
            c = df.MedInc, s = df.Population/100)
plt.xlabel('Latitude')
plt.ylabel('Longitude')
plt.savefig(downloads +'\scatter2.png', dpi=300, bbox_inches='tight')
plt.show()

References

1. scikit-learn documentation
2. Datacamp: Supervised Learning
3. Datacamp: Unsupervised Learning