WHAT IS TRANSFER LEARNING? 4 PRE-TRAINED MODEL IN TRANSFER LEARNING

With the easy access to neural network models available today, attempting to build your own model with limited resources seems almost pointless. Instead, you can reuse advanced models that have been proven to offer high accuracy and performance. To achieve this, you need to understand what Transfer Learning is and how it works.

Transfer Learning is a machine learning technique that enables pre-existing models to be developed and applied to a second task. This transfer learning approach is applied in many research fields, especially in Deep Learning. In this article, let’s dive deeper into what Transfer Learning is and how it works.

What is Transfer Learning?

Transfer Learning is a subfield in machine learning and artificial intelligence aimed at applying knowledge gained from one source task to a similar task. Simply put, a model developed for one task is reused and serves as the starting point for a model in a second task.

What is Transfer Learning? Transfer Learning is a subfield of machine learning and artificial intelligence.

For example, text classification on Wikipedia can be applied to classifying legal documents. Or, knowledge gained from classifying cars can be used to recognize and classify bird species in the sky.

Transfer Learning is understood as a method of improving the learning of a new task by transferring knowledge from a related task that has been learned previously. Nowadays, with the use of Transfer Learning, people can build AI applications in a short amount of time.

4.0 TECHNOLOGY IN EDUCATION

The History of Transfer Learning’s Emergence

Transfer Learning first appeared in 1993 in the paper “Discriminability-Based Transfer between Neural Networks.” In this paper, author Lorien Pratt introduced the potential of transfer learning. By July 1997, the term was once again mentioned in the Machine Learning journal with a dedicated issue on the topic.

After some time, this field continued to develop and was merged with related topics such as multi-task learning. The Transfer Learning method was also introduced for the first time in the book Learning to Learn, which presented theories and in-depth research on this approach. To date, Transfer Learning has become a powerful resource that helps tech companies build new AI solutions, pushing the boundaries of machine learning. The power of Transfer Learning has also been acknowledged by Google Brain co-founder Andrew Ng, who stated: “Transfer Learning is the next driving force leading to the commercial success of machine learning after supervised learning.”

What is OCR? Why is OCR technology truly essential?

What is the mechanism of Transfer Learning?

The operation of Transfer Learning is based on three mechanisms, as follows:

Choosing the Source Model

The source model is a model built and trained by others to solve a similar problem. The source model is often developed by tech giants or well-known scientific teams.

Typically, the source model uses a very large dataset as the base data, such as ImageNet or the Wikipedia Corpus. Based on the source model, a large neural network is then developed to solve a specific problem. The chosen source model must be publicly available and permitted for reuse.

Fine-tuning the Model

After accessing the source model, you can leverage the knowledge that the model has learned, such as knowledge about classes, features, weights, and free parameters. Alternatively, you can load the source model into your environment, turning it into a file/folder containing the relevant information and then customize the model to suit your needs. However, it is recommended to prioritize customizing deep learning repositories that store multiple models, such as TensorFlow Hub, Keras Applications, PyTorch Hub, etc.

You can use one of these sources to load the source model into your model. The source model typically comes with all the layers and weights, and you can freely modify the model as needed. The purpose of fine-tuning the model is to improve its accuracy while ensuring the output is in the correct format.

Applying to a New Task (Inference)

In a neural network, the lower and middle layers represent common features, while the upper layers represent task-specific features within the model. Therefore, the new task of the new model will differ from the original task of the source model.

As a result, you need to add new tasks and remove the top layers to improve the accuracy of your task. After that, you can configure the model using a specialized optimizer.

FEDERATED LEARNING AND ITS PRACTICAL APPLICATIONS

4 Pre-Trained Models in Transfer Learning

In addition to understanding the history behind the concept of Transfer Learning, we can explore deeper into the 4 models used in Transfer Learning. Below are the 4 most popular pre-trained models that you can use for tasks in transfer learning:

There are 4 Most Popular Pre-Trained Models in Transfer Learning.

VGG19

VGG19 Model is a convolutional neural network with a depth of 19 layers, built and trained in 2014. This model uses over 1 million images from the original ImageNet model, including color images of 224×224 pixels. As a result, this pre-trained model can classify up to 1000 different objects. The model also allows you to import weights trained by ImageNet. The size and performance of this model are as follows:

• Size: 549 MB
• Top-1 Accuracy: 71.3%
• Top-5 Accuracy: 90.0%
• Number of Parameters: 143,667,240
• Depth: 26

Inceptionv3 (GoogLeNet)

Inceptionv3 Model is a convolutional neural network with a depth of 50 layers that allows you to access all the information in the paper. This model uses the ImageNet dataset and can classify up to 1000 different objects. The input image size for the model is 299×299 pixels. The size and performance of this model are as follows:

• Size: 92 MB
• Top-1 Accuracy: 77.9%
• Top-5 Accuracy: 93.7%
• Number of Parameters: 23,851,784
• Depth: 159

ResNet50

ResNet50 Model is a convolutional neural network with a depth of 50 layers, built and developed by Microsoft in 2015. This model allows access to 1 million images from the original ImageNet dataset, capable of classifying 1000 objects, including color images with a size of 224×224 pixels. This model provides significantly improved performance compared to VGG19, even with a lower level of complexity. The size and performance of the model are as follows:

• Size: 98 MB
• Top-1 Accuracy: 74.9%
• Top-5 Accuracy: 92.1%
• Number of Parameters: 25,636,712

EfficientNet

EfficientNet Model is the most advanced convolutional neural network developed by Google in 2019. This model comes with 8 alternative versions, featuring 5.3 million parameters. Its accuracy reaches Top-1 at 77.1%. The size and performance of this model are as follows:

• Size: 29 MB
• Top-1 Accuracy: 77.1%
• Top-5 Accuracy: 93.3%
• Number of Parameters: ~5,300,000
• Depth: 159

PROFESSIONAL DOCUMENT ARCHIVING PROCESS (2023)

Conclusion

Understanding the concept of Transfer Learning helps technology companies leverage existing models to quickly develop AI solutions. This allows businesses to optimize their human resources, reduce costs, and create AI algorithms in the most efficient and accurate way.

However, when applying the Transfer Learning method, tech companies should pay attention to selecting the right source model, fine-tuning the model, and adding new tasks to ensure the accuracy of the algorithm. We hope that the information provided about “What is Transfer Learning” will help you clarify any doubts about this method and how to apply it effectively in Machine Learning.