It has its origins in antiquity, and the idea behind it continues to drive revolutionary inventions to this day. What for a long time was found only in myths and works of fiction has, for several decades now, held a firm place at the forefront of science. In dystopian visions of the world, it seizes complete control over the planet, and yet it is capable of optimizing our lives in nearly every area: artificial intelligence.
In our latest blog post, our Xperts explore a specific aspect of this subfield within computer science: machine learning. This subdiscipline focuses on the ability of machines to learn—the artificial generation of knowledge. Using a concise example in Python, our Xperts demonstrate how a model can recognize numbers with the help of machine learning algorithms. First, however, we’d like to clarify a few terms and concepts to ensure a better understanding.
Artificial Intelligence
Artificial intelligence refers to all methods that use a computer algorithm to train a model. The word “artificial” first and foremost indicates that this is something that does not occur in nature. Intelligence is defined here as the ability to direct actions toward achieving goals—tasks in which machines are expected to operate in a manner similar to humans.
The trained model can be used for environmental perception and decision-making. For this, input data for the model must be available. Artificial intelligence therefore relies on a realistic representation of, for example, sensor data or other input variables. The method used to train these models is called machine learning.
Machine Learning
With the help of machine learning algorithms, machines are able to learn from data and recognize patterns. Based on this, they can perform human-like actions, such as making predictions or decisions.
In machine learning, predefined features are used to train a model, and the model takes these features into account. Humans have a significant influence on the selection of these features through various preprocessing steps during the model creation process. This scope of action facilitates feature extraction from the finished model.
Feature extraction refers to a method in machine learning that can be used to identify the most meaningful input variables. For example, in image classification tasks, it can determine which pixels are most likely to be associated with which object.
If you take machine learning one step further—or deeper—you enter the realm of deep learning. Unlike machine learning, this subfield focuses on the hidden layers of a model. To train the model, the features are no longer defined by humans. Since this eliminates some steps prior to training, there is less control over the result than with traditional machine learning methods. The first training steps—the warm-up—involve randomly shuffling the features. With the help of the hidden layers, the model then determines during the training process which features are the most meaningful.
Since this specific form of artificial intelligence requires enormous amounts of training data and computing power—and thus entails high costs—deep learning is not suitable for widespread use. While it was still possible to verify everything manually in machine learning, this is no longer possible with deep learning. The complexity of the calculations is no longer comprehensible to humans on a case-by-case basis.
Regardless of whether we're talking about machine learning or deep learning, both are based on what are known as artificial neural networks.
Artificial Neural Networks
The structure and function of artificial neural networks are modeled after the human brain. They are ideally suited for automating tasks to identify patterns or relationships in large amounts of data. The foundation of every artificial neural network is formed by many interconnected processing units—the artificial neurons.
An artificial neuron consists of several inputs, each of which receives a signal, and an output that transmits the signal. Each input value is assigned a so-called weight, which regulates the contribution of that input to the neuron’s total output. The weighted input values are then summed and passed to the activation function for output.
In an artificial neural network, each of these artificial neurons is connected to the others, and in deep learning, these connections even extend across multiple layers.
Key Performance Indicators
Performance metrics are an important tool for assessing the performance of machine learning models. These are evaluation metrics, and depending on the use case, some are more suitable than others. Below, we’d like to provide a brief overview of the most common of these metrics.
| Value | Symbol or formula | Definition |
|---|---|---|
| True Positive | tp | Data classified as "true positive" |
| True Negative | tn | Data classified as "negative" |
| False Positive | fp | Negative Data Classified as Positive |
| False Negative | fn | Data classified as negative |
Accuracy: Accuracy measures how often the model makes correct predictions. It is calculated by dividing the number of correct predictions by the total number of predictions.
Precision: Precision measures how often the model makes correct positive predictions. It is calculated by dividing the number of correct positive predictions by the total number of positive predictions.
Recall (Sensitivity): Recall measures how often the model identified all true positive cases. It is calculated by dividing the number of correctly identified true positive cases by the total number of true positive cases.
F1 Score: The F1 Score is the harmonic mean of precision and recall. As a result, the F1 Score is less susceptible to misleading accuracy metrics and has become the standard for evaluating accuracy.
These metrics make it fairly easy to see whether a system is skewed too much in one direction. To achieve the desired result, the parameters can be adjusted as needed—for example, by adding a new bias at the end.
Hardware Requirements
In general, almost any standard PC or laptop is sufficient for machine learning. Even for complex applications, the computing power of these devices is adequate. For example, single-board computers such as the Raspberry Pi can also be used to get started with machine learning. When it comes to deep learning, however, the hardware requirements become significantly more demanding. Due to the multi-layer artificial neural networks and the complex calculations involved, entire data centers may be required to run them. The larger and more complex the model, the more rapidly the hardware requirements increase.
Application
Now that we have clarified the basic terms and concepts, we would like to move on to our practical example using Python. For this, we will work on pixel-based digit recognition using logistic regression. This is a mathematical function used to map the model being trained.
Jupyter Notebooks (.ipynb), for example, are ideal for learning, since the code can be broken down into individually executable blocks. Jupyter is available as a standalone application as well as a plug-in for popular code editors.
First, you need to import the required libraries into Python.
The MNIST dataset from the National Institute of Standards and Technology is freely available and contains a large amount of training and test data. It can be downloaded and imported into Python using the following code:
The „mnist“ variable we have just declared still needs to be split into training and test data. For our example, we will use 60,000 training data points and 10,000 test data points.
Data Exploration and Preprocessing
The data in the MNIST dataset consists of images with a resolution of 28×28 pixels.
Using the command plt.imshow(X_test[0].reshape(28, 28)) The first image in the test dataset can be displayed.
To start training the machine learning model, you must first from sklearn.linear_model import LogisticRegression imported and as model = LogisticRegression() be defined.
The number of times the model should iterate through the training dataset is specified by model.max_iter=4 defined. The choice of the number of iterations is of great importance in machine learning. If a model is undertrained, it may not be able to generalize the „features“ it is supposed to recognize. An overtrained model, on the other hand, may be very accurate when tested against the training dataset, but would perform significantly worse on new test datasets.
Finally, using model.fit(X=X_train, y=y_train) The training can begin.
Due to the small number of iterations, training data, and pixels to be processed per image, this model is fully trained within a few seconds. This is also attributable to the computing power of today's hardware.
To determine the model's accuracy, you need the accuracy metrics. These can be calculated using sklearn with the following code:
The resulting confusion matrix describes which numbers the model correctly identifies.
In this graph, the digit 5 is not very distinct, and there are significant discrepancies in the digits 3 and 8. This means that the model has particular difficulty recognizing the 5 as such and often misidentifies it as a 3 or an 8.
The trained model
Using the command plt.imshow(model.coef_[8].reshape(28, 28)) the coefficients of a model (in this case, the number 8) can be visualized. The light yellow areas represent the pixels that the model clearly associates with the number 8. However, if a pixel is located in a dark blue area, this suggests that it should not be classified as an 8.
For each number to be recognized, a probability for the possible numbers 0 through 9 is output at the output neuron at the end of the processing chain. The number with the highest probability is ultimately used as the recognized number.
Side Note: Data Preprocessing
In machine learning, preprocessing the data is essential for achieving better results. The MNIST dataset has already been cleaned of many sources of error.
Possible preprocessing steps for character recognition include reducing brightness information, enhancing edges, or realigning the characters to the center of the image. It may well be worth trying out these methods and then comparing the accuracy values. To do this, you simply need to use the values from X_train and X_test be replaced.
Information Reduction
In the case of the MNIST dataset, brightness values range from 0 to 255 (8 bits). The brightness values can be obtained using X_train.min() and X_train.max() be determined. This information can, for example, be reduced to 1 bit by setting a threshold value below which all values are defined as 0 and above which all values are defined as 1.
Using this method, the input data for the machine learning model was reduced by a factor of 128. However, this preprocessing reduces the accuracy by 1 % in the F1 score. This example clearly demonstrates that in machine learning, models can often be downscaled for less powerful hardware or limited data transmission channels without incurring a significant loss of accuracy.
Edge Detection
Another option for data preprocessing is to generate images using edge detection. Image manipulation can be performed using the Python Image Library (PIL). After importing the library, the pixels of the training data must be read in as images. Edge detection can then be applied to these images. To be able to use the images as training data later, they must be written back to an array as individual pixels. This process is repeated with the test dataset. The resulting array has three dimensions, but our model can be trained with a maximum of two dimensions. The dimensions can be set using np.concatenate reduce.
Unfortunately, this data preprocessing—which results in a loss of accuracy of 2 %—does not improve the F1 score either. The images in the MNIST dataset are already well preprocessed for further processing. However, our small experiment shows that there are numerous options for feeding data into a machine learning model.
How We Work
The potential applications of machine learning are diverse. Machine learning makes a valuable contribution to our projects, particularly in the implementation of predictive maintenance applications. For example, using an artificial neural network, we have developed an application that automatically identifies equipment requiring maintenance.
In the future, the industries in which machine learning will play a key role will continue to expand, not least due to the steady increase in hardware performance. Because of its versatility, machine learning will continue to play a major role alongside deep learning in driving technological change for a long time to come.
