Enhancing Multilingual Models with Active Forgetting
Unveiling the potential of active forgetting in pretraining for superior language model adaptability, particularly for linguistically distant languages in low-data scenarios.
Improving language plasticity through pretraining with active forgetting presents a compelling approach to enhancing the flexibility and efficiency of PLMs across languages. The demonstrated benefits in terms of adaptation speed and performance in low-data settings, especially for distant languages, highlight its potential in a paper “Improving Language Plasticity via Pretraining with Active Forgetting” (Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Ifeoluwa Adelani, Pontus Stenetorp, Sebastian Riedel, Mikel Artetxe).
In this groundbreaking study, scientists have unlocked a new method to vastly improve the adaptability of artificial intelligence in understanding and processing multiple languages. Traditional Pretrained Language Models (PLMs), known for their prowess in a myriad of Natural Language Processing (NLP) tasks, often stumble when introduced to new linguistic territories due to their heavy reliance on vast datasets and significant computational power. Addressing this critical bottleneck, the research introduces an “active forgetting” technique, focusing on the periodic reset of the token embedding layer within these models.
The essence of this approach lies in its simplicity and efficiency. By selectively resetting token embeddings at intervals while maintaining other parameters intact, the model engages in a continuous cycle of re-learning. This process, akin to a meta-learning effect, is believed to bolster the model’s abstract reasoning and generalization skills across diverse languages. It challenges the model to avoid leaning on memorized shortcuts, thereby enhancing its linguistic plasticity.
Empirical evidence from the study paints a compelling picture: PLMs equipped with this active forgetting mechanism surpass their conventional counterparts in cross-lingual transfer tests. These models not only demonstrate superior performance but also achieve it with remarkable speed during the adaptation phase. Their proficiency is especially pronounced in handling languages with significant lexical and grammatical deviations from English, such as Arabic, Hindi, Thai, and Turkish.
This innovative method stands as a testament to the evolving landscape of machine learning, where the ability to quickly adapt and learn from minimal data is increasingly paramount. As the digital world becomes more interconnected, the demand for multilingual AI tools that can seamlessly navigate the complexities of global languages will continue to soar. This research marks a significant step forward, promising a future where language barriers are effortlessly surmounted by intelligent machines, heralding a new era of inclusivity and accessibility in technology.
The concept of improving language plasticity through pretraining with active forgetting offers a novel approach to enhancing the adaptability of pretrained language models (PLMs) to new languages. This method seeks to address the challenges faced when applying PLMs to languages for which they were not originally trained, a significant barrier to universal accessibility of PLM capabilities. Traditional methods, such as learning a new embedding layer for the new language, though effective, are criticized for their data and compute inefficiency.
The proposed solution introduces an active forgetting mechanism during pretraining, characterized by periodically resetting the embedding layer, thereby simulating a form of meta-learning. This encourages the PLM to learn new embeddings more efficiently within a limited number of updates.
The experimental findings, particularly with RoBERTa, validate the effectiveness of this method. Models pretrained with active forgetting demonstrate not only faster convergence during the language adaptation phase but also superior performance in low-data regimes, especially for languages that are linguistically distant from English. These results underscore the potential of active forgetting as a strategy to increase the linguistic adaptability of PLMs, making them more accessible and efficient across a broader range of languages.
However, the approach is not without its limitations. The simplicity of directly resetting embeddings to random initialization may not always be optimal. Future work could explore more sophisticated methods of introducing variability or controlled forgetting, which might yield further improvements in the model’s adaptability and efficiency. Additionally, while the experiments focus on RoBERTa, applying this technique to other architectures or in multi-lingual pretraining contexts could provide more insights into its generalizability and effectiveness across different settings.
Applying language models to a new language can be difficult. This is a barrier to making their capabilities universally accessible.
— cohere (@cohere) January 22, 2024
Here’s Kelly Marchisio @cheeesio explaining “Improving Language Plasticity via Pretraining with Active Forgetting” at #NeurIPS2023. The method… pic.twitter.com/9tfS3lLpPb
More about NeurIPS2023.
Unveiling the potential of active forgetting in pretraining for superior language model adaptability, particularly for linguistically distant languages in low-data scenarios.
Professor Michael Wooldridge’s insightful presentation highlighted human intelligence’s unique aspects, contrasting it with the emerging intelligence of Large Language Models (LLMs). This discussion opens up a vital conversation about the biases we project onto AI and the potential for GPTs to develop a distinct form of intelligence, diverging significantly from human cognition.
IBM’s new synthetic data generation method and phased-training protocol allows enterprises to update their LLMs with task-specific knowledge and skills, taking some of the guesswork out of training generative AI models.
In the latest advancements, artificial intelligence has reached new heights with ChatGPT-4 passing the Turing Test, illustrating AI’s ability to mimic human-like behaviors and decision-making. Concurrently, OpenAI’s Sora has emerged, transforming textual prompts into photorealistic videos, pushing the boundaries of AI’s creative potential. These developments underscore the critical need for ethical frameworks in AI, addressing concerns such as misuse, intellectual property, and the impact on creativity. The rapid evolution of AI technologies like ChatGPT-4 and Sora highlights both the transformative possibilities and the ethical challenges that accompany the blurring of lines between human and machine intelligence.
Explore the innovative implementation of SparseGPT on AWS for pruning massive GPT models efficiently. Discover how this technique retains high accuracy while significantly reducing computational demands.
This excerpt introduces meta-prompting, a novel scaffolding technique to enhance language models by enabling them to function as both orchestrators and specialists. It leverages high-level directives for decomposing complex tasks into simpler subtasks, tackled by expert instances of the same model under specific instructions. This method transforms a single language model into a multi-functional entity, capable of conducting integrated, expert-level analyses and generating refined outcomes. Meta-prompting’s task-agnostic framework simplifies user interactions and incorporates external tools like Python interpreters, significantly improving task performance. Research with GPT-4 demonstrates its effectiveness, showing a marked performance improvement over traditional prompting methods.
In the pharmaceutical and heatlh industry, research and development (R&D) is a pivotal area where innovation drives progress. One of the challenges in R&D is the efficient analysis and interpretation of vast amounts of unstructured data, such as research papers, patents, and lab reports. Topic modeling, a machine learning technique, can be leveraged to unearth hidden themes in such textual data, providing valuable insights for chemical compound research.
In the ever-evolving world of social media, hashtags have become a cornerstone in shaping digital conversations. They are not just mere labels but are pivotal in categorizing and identifying the pulse of social narratives. However, with this utility comes a challenge: the dynamic and polysemous nature of hashtags. This complexity is where the innovative approach of “Hashtag Sense Clustering Based on Temporal Similarity” comes into play. The challenges of hashtags in Twitter (X) Traditionally, hashtags have been used as simple markers to categorize posts or as symbols of community affiliation. But their usage varies greatly, often leading to ambiguity. The same hashtag can represent different topics at different times, and conversely, various hashtags can denote the same subject. This polymorphic nature, coupled…
The article discusses the impact of ChatGPT and other AI technologies on society and the workforce, with a focus on how it will affect different professions. The article also explores the advent of real-time application development and how AI tools like ChatGPT are shifting the paradigm towards personalized applications that are developed on demand, in real-time. The article concludes by providing tips on how to adapt to the disruption brought about by AI, including taking basic AI or machine learning courses and reading top AI books.
In the realm of industrial innovation, the convergence of AI and ML technologies is revolutionizing manufacturing operations. Discover how sophisticated AI-driven predictive maintenance systems leverage natural language programming techniques to enhance operational efficiency and mitigate downtime risks. Explore the integration of advanced language models like GPT-3.5 and LLAMA2 within LangChain, alongside LSTM networks and self-attention mechanisms, to create a robust framework for proactive maintenance strategies. Witness the transformative impact of AI technologies in reshaping traditional industrial paradigms and optimizing production processes for sustained competitiveness and growth.
In a pioneering effort to streamline laboratory knowledge management, a sophisticated system leveraging Natural Language Processing (NLP) and machine learning models, including BERT and GPT, was developed to efficiently manage a massive repository of scanned documents. By applying advanced techniques such as topic modeling, document clustering, and semantic similarity analysis, this system significantly improved document accessibility, categorization, and retrieval. The creation of a detailed ontology, integrated with public data sources, further enhanced data interoperability and research collaboration, showcasing the transformative potential of NLP in handling complex data landscapes.
In a pioneering move to enhance global access to agricultural and environmental data, a consortium of research organizations has launched the Info Finder, an online search tool designed to revolutionise the dissemination of specialized information in these fields. This collaborative effort, featuring contributions from the World Agricultural Information Center of the FAO, Future Harvest Centers worldwide, and the CGIAR, underscores a significant leap forward in digital transformation efforts within agriculture. With the platform harnessing FAO’s cutting-edge technologies and adhering to common standards such as the Agrovoc agricultural thesaurus, Info Finder emerges as a beacon of innovation. It paves the way for rapid access to a vast reservoir of knowledge, promising to play a crucial role in supporting sustainable agricultural practices and ensuring food security across the globe. The involvement of Massimo Buonaiuto, a leading figure in data science and digital transformation, highlights the critical intersection of technology and agricultural research, driving forward the agenda for a more informed and sustainable future.
This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Read More