In 2022, here are four points to consider regarding artificial intelligence deep learning

By claire reid

Another year of remarkable AI (deep learning) advances is upon us – one filled with remarkable progress, disagreements, and, of course, conflicts. Here are a few of the most significant overarching trends that marked this year in deep learning.

1.SCALE CONTINUES TO BE A MAJOR FACTOR.

Deep learning’s desire to expand neural networks has remained constant for the past few years, as well as the availability of computer resources, high-quality AI hardware, and the development of scale-friendly architectures, such as the transformer model.

DeepMind announced the Gopher, a 280 billion-parameter large language model (LLM), as well as the Generalist Language Model (GLaM), which has up to 1.2 trillion parameters, and Microsoft and Nvidia developed the Megatron-Turing LLM, a 530 billion-parameter LLM.

Emergent abilities are where larger models succeed at accomplishing tasks that would be impossible with smaller ones. This phenomenon has been particularly intriguing in LLMs, where models demonstrate promising results on a wider range of tasks and benchmarks as they grow in size.

Nonetheless, it is important to be aware that some of deep learning’s main problems remain unsolved, even in the most powerful models (more on this in a minute).

2.UNSUPERVISED LEARNING CONTINUES TO BE BENEFICIAL.

Many successful deep learning applications require humans to label training examples, also known as supervised learning. But most internet data is inadequate for this purpose. Data annotation is expensive and slow, creating bottlenecks. This is why researchers have long sought advances in unsupervised learning, where deep learning models are trained without the need for human-annotated data.

In recent years, significant progress has been made in this field, especially in LLMs, who are mostly trained on massive data sets from the internet. In 2022, we also observed other trends in unsupervised learning techniques.

This year, text-to-image models made enormous strides. OpenAI’s DALL-E 2, Google’s Diffusion, and Stability AI’s Diffusion have demonstrated the potential for unsupervised learning. Unlike previous text-to-image models, which required well-annotated pairs of images and descriptions, these models use large datasets of loosely captioned images that already exist on the internet.

3.MULTIMODALITY IS MAKING SIGNIFICANT STRIDES.

Another fascinating characteristic of text-to-image generators is that they can combine many data types in a single model. Deep learning models are able to tackle far more complex tasks because they are able to process many different information types.

When you see a tree and hear the wind rustling in its branches, your mind can quickly associate them together, or you can quickly recall previous experiences.

Multimodality has played an important role in making deep learning systems more flexible. Gato, a deep learning model trained on a variety of data types, excelled in many tasks, including image captioning, interactive dialogues, controlling a robotic arm, and playing games.

Several researchers have gone so far as to say that a system like Gato is all we need to achieve artificial general intelligence (AGI). While many scientists disagree with this view, one thing is certain: multimodality has resulted in significant advances in deep learning.

4.THERE ARE STILL SIGNIFICANT DEEP LEARNING DIFFICULTIES.

Despite the incredible capabilities of deep learning, certain areas remain unresolved. Among them are causality, compositionality, reasoning, planning, intuitive physics, and abstraction and analogy-making.

Scientists in different fields are still working on these ordeals. Pure scale- and data-based deep learning approaches have helped make incremental improvements on some of these difficulties while failing to provide a definitive solution.

Large LLMs may be able to maintain coherence and consistency over longer periods of text. However, they fail on tasks that require meticulous step-by-step reasoning and planning.

Similar to text-to-image generators, they create stunning graphics, but make minor errors when asked to draw images that require composition or require complex descriptions.

Yann LeCun, the Turing Award-winning inventor of convolutional neural networks (CNN), is well-known among scientists who have studied and addressed these issues. The technique is capable of addressing some of the current problems that the field currently faces.

Deep learning has come a long way. However, the more progress we make, the more we become aware of the challenges of constructing truly intelligent systems. Next year will be equally as thrilling.