Abstract

Although deep learning has historical roots going back decades, neither the term 'deep learning' nor the approach was popular just over five years ago, when the field was reignited by papers such as Krizhevsky, Sutskever and Hinton's now classic 2012 deep net model of Imagenet. What has the field discovered in the five subsequent years? Against a background of considerable progress in areas such as speech recognition, image recognition, and game playing, and considerable enthusiasm in the popular press, I present ten concerns for deep learning, and suggest that deep learning must be supplemented by other techniques if we are to reach artificial general intelligence.

Introduction: Is Deep Learning Approaching a Wall?

Deep learning, despite its recent explosion in popularity, has historical roots stretching back decades. The field attracted relatively little notice until just over five years ago when everything changed in 2012 with the publication of highly influential papers such as Krizhevsky, Sutskever and Hinton's 'ImageNet Classification with Deep Convolutional Neural Networks,' which achieved state-of-the-art results on the object recognition challenge known as ImageNet.

Before the year was out, deep learning made the front page of The New York Times, and it rapidly became the best known technique in artificial intelligence, by a wide margin. While the general idea of training neural networks with multiple layers was not new, increases in computational power and data availability made deep learning truly practical for the first time.

Deep learning has since yielded numerous state-of-the-art results in domains such as speech recognition, image recognition, and language translation, and plays a role in a wide swath of current AI applications. Corporations have invested billions of dollars fighting for deep learning talent, with prominent advocates like Andrew Ng suggesting that 'If a typical person can do a mental task with less than one second of thought, we can probably automate it using AI either now or in the near future.'

However, evidence suggests deep learning may be approaching a wall. As François Chollet, author of the Keras neural network library, noted in December 2017: 'For most problems where deep learning has enabled transformationally better solutions (vision, speech), we've entered diminishing returns territory in 2016-2017.' Even Geoff Hinton, the grandfather of deep learning, acknowledges that 'The future depends on some graduate student who is deeply suspicious of everything I have said.'

This paper aims to temper irrational exuberance while considering what the field needs to move forward, addressing both technical researchers and AI consumers with less technical background who wish to understand where the field is headed.

What Deep Learning Is and What It Does Well

Deep learning refers to a class of machine learning techniques that use multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input, allowing the system to learn representations of data with multiple levels of abstraction.

The fundamental strength of deep learning lies in its ability to discover intricate structure in large data sets through backpropagation algorithm, which uses gradient descent to adjust the weights of connections between neurons. This approach has proven exceptionally powerful for problems where:

• The input-output mapping is complex but can be learned from abundant data
• The features needed for the task are not easily specified by human engineers
• The problem involves perceptual tasks similar to those humans and animals solve naturally

Deep learning has achieved remarkable success in several key domains:

Computer Vision

Convolutional Neural Networks (CNNs) have revolutionized computer vision, achieving human-level performance on tasks like image classification, object detection, and facial recognition. The 2012 ImageNet breakthrough demonstrated that deep networks could significantly outperform previous approaches, reducing top-5 error rates from 26% to 15%.

Speech Recognition

Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks have dramatically improved speech recognition systems, with error rates dropping to near-human levels in controlled environments. Companies like Google, Apple, and Amazon have integrated these technologies into their virtual assistants.

Natural Language Processing

Sequence-to-sequence models and attention mechanisms have enabled significant advances in machine translation, text summarization, and question answering systems. Google's Neural Machine Translation system represents one prominent example of this progress.

Game Playing

Deep reinforcement learning has produced systems that surpass human experts in complex games like Go, Chess, and video games. DeepMind's AlphaGo and OpenAI's Dota 2 bots demonstrate the potential of combining deep learning with reinforcement learning.

The common thread in these successes is deep learning's ability to automatically learn relevant features from raw data, eliminating the need for manual feature engineering that dominated earlier approaches to machine learning.

Deep Learning's Weaknesses and Limitations

Despite its impressive successes, deep learning faces fundamental limitations that may prevent it from achieving artificial general intelligence without significant supplementation from other approaches. This section outlines ten key concerns:

1. Deep Learning Is Data-Hungry

Deep learning typically requires enormous amounts of labeled training data to achieve high performance. While humans can learn new concepts from just one or a few examples, deep learning systems often need thousands or millions of examples. This data hunger makes deep learning impractical for many real-world applications where labeled data is scarce or expensive to obtain.

2. Deep Learning Is Shallow and Has Limited Capacity for Transfer

Deep learning systems struggle with transfer learning—applying knowledge learned in one context to different but related contexts. While humans readily transfer skills and knowledge across domains, deep learning systems typically require retraining or fine-tuning for each new task, even when the tasks are conceptually similar.

3. Deep Learning Has No Natural Way to Deal with Hierarchical Structure

Human reasoning often involves understanding hierarchical relationships and structured representations. Deep learning systems, particularly current architectures, have difficulty capturing the recursive, compositional nature of many cognitive tasks, especially in language understanding and reasoning.

4. Deep Learning Has Struggled with Open-Ended Inference

Deep learning systems excel at pattern recognition but struggle with tasks requiring reasoning, inference, and understanding causality. They lack robust mechanisms for drawing conclusions from incomplete information or making logical deductions beyond their training data.

5. Deep Learning Is Not sufficiently Transparent

The 'black box' nature of deep neural networks makes it difficult to understand how they arrive at their decisions. This opacity raises concerns about reliability, safety, and ethics, particularly in high-stakes applications like healthcare, criminal justice, and autonomous vehicles.

6. Deep Learning Has Not Been Well Integrated with Prior Knowledge

Human learning builds upon extensive prior knowledge and common sense. Deep learning systems typically learn from scratch without leveraging the rich structured knowledge that humans accumulate over lifetimes. This limitation makes them inefficient learners and vulnerable to making errors that humans would easily avoid.

7. Deep Learning Cannot Distinguish Causation from Correlation

Deep learning systems identify correlations in data but cannot inherently distinguish causal relationships from spurious correlations. This limitation can lead to models that perform well on training data but fail in real-world deployment when confounding variables change.

8. Deep Learning Presumes a Mostly Stable World

Current deep learning approaches assume that the world is relatively stable and that training and test data come from similar distributions. They struggle with non-stationary environments where the rules or relationships change over time, requiring constant retraining.

9. Deep Learning Works Well as an Approximation but Is Often Not a Complete Solution

While deep learning provides excellent function approximation in many domains, it often works best as a component within larger systems rather than as a complete solution. Real-world applications frequently require integration with symbolic reasoning, databases, and other AI techniques.

10. Deep Learning Is Difficult to Engineer With

The development and debugging of deep learning systems remains more art than science. Training can be unpredictable, with performance sensitive to seemingly minor changes in architecture, hyperparameters, or initialization. This engineering challenge increases development costs and reduces reliability.

These limitations suggest that deep learning alone may be insufficient for achieving artificial general intelligence and that hybrid approaches combining deep learning with other AI techniques may be necessary.

Common Misconceptions About Deep Learning

Several misunderstandings about deep learning's capabilities have contributed to unrealistic expectations:

The Myth of Inevitable Progress

Many assume deep learning will continue its rapid progress indefinitely. However, as noted by leading figures in the field, we may be entering a period of diminishing returns for many applications. The low-hanging fruit may have been harvested, with remaining challenges requiring fundamentally different approaches.

The Confusion of Performance with Understanding

High performance on specific benchmarks does not equate to human-like understanding. Deep learning systems can achieve superhuman performance on narrow tasks while lacking any genuine comprehension of the domain.

The Assumption of Generalization Beyond Training Distribution

There is often an implicit assumption that deep learning systems will generalize robustly to novel situations. In practice, they frequently fail on inputs that differ significantly from their training data, sometimes in ways that seem trivial to humans.

The Belief That Scale Solves Everything

Some proponents suggest that simply scaling up data, computation, and model size will overcome current limitations. While scale has driven much progress, there is evidence that some limitations are fundamental to the approach rather than matters of scale.

Addressing these misconceptions is crucial for setting appropriate expectations and directing research toward the most promising avenues.

Future Directions and Conclusions

Five years into deep learning's resurgence provides an opportune moment for critical reflection. While deep learning has produced remarkable achievements, its limitations suggest the need for new approaches and hybrid systems.

Potential Research Directions

Several promising research directions could address deep learning's limitations:

• Hybrid neuro-symbolic systems that combine deep learning's pattern recognition capabilities with symbolic reasoning
• Few-shot and meta-learning approaches that enable learning from limited data
• Causal modeling techniques that go beyond correlation to understand underlying mechanisms
• More structured neural architectures that better capture hierarchical and compositional structure
• Integration with knowledge bases and common-sense reasoning systems
• More transparent and interpretable models that provide insight into their decision processes

Broader Implications

The limitations of deep learning have implications beyond technical research:

• Education should emphasize a broad AI curriculum rather than focusing exclusively on deep learning
• Industry applications should recognize where deep learning is appropriate and where alternative approaches might be more suitable
• Policy and regulation should account for the limitations and potential failures of deep learning systems
• Research funding should support diverse approaches to AI rather than concentrating exclusively on deep learning

Conclusion

Deep learning represents a powerful tool in the AI toolkit but is unlikely to be sufficient for achieving artificial general intelligence. Its strengths in pattern recognition and function approximation must be complemented with other approaches that address its weaknesses in reasoning, transfer learning, and integration with prior knowledge.

As the field moves forward, researchers should maintain a balanced perspective—recognizing deep learning's substantial contributions while actively exploring its limitations and potential supplements. The most promising path likely involves hybrid systems that combine the best of deep learning with other AI paradigms.

The critical appraisal presented in this paper aims to foster this balanced approach, encouraging both appreciation for what deep learning has achieved and clear-eyed recognition of where it falls short. Such perspective is essential for responsible progress toward more capable and general artificial intelligence systems.

Key Insights