Blog

BLIP-2: Scalable Pre-training of Multimodal Foundation Models for the World's First Open-source Multimodal Chatbot

Salesforce Data Cloud, the first real-time CRM, is turning your data into real-time customer magic. You might have witnessed how our AI research is powering intelligent experiences with Identity Resolution for Fuzzy Matching featured at Dreamforce. The goal of identity resolution is to identify the same individuals across datasets and

TLDR; We introduce the Salesforce CausalAI Library, an open source library for causal analysis of time series and tabular data. The Salesforce CausalAI Library aims to provide a one-stop solution to the various needs in causal analysis including handling different data types, data generation, multi-processing for speed-up, utilizing domain knowledge

TL;DR: Text-to-image diffusion models are very adept at generating novel images from a text prompt, but current adaptations of these methods to image editing suffer from a lack of consistency and faithfulness to the original image. Many of these discrepancies can be traced to the difficulty of inverting the

TL;DR: Slack is a powerful productivity tool that professionals and teams around the globe depend on to communicate and keep up with vital information at work, making async work efficient. However, with an increase in the number (and variety) of conversations on Slack that a typical user keeps track

TL;DR: We present BotSIM, a data-efficient end-to-end Bot SIMulation toolkit for evaluation, diagnosis, and improvement of commercial task-oriented dialogue (TOD) systems. BotSIM's “generation-simulation-remediation'' paradigm can accelerate the end-to-end bot evaluation and iteration process by: (1) reducing the effort needed to create test cases; (2) enabling a better understanding of

We introduce an automated method for evaluating the quality of AI-generated text by repurposing prior human evaluation data. The method is called Near-Negative Distinction.

Conference Overview Next week, the Thirty-sixth annual Conference on Neural Information Processing Systems (NeurIPS) will be held in New Orleans, Louisiana from Monday, November 28th, through Friday, December 9th. NeurIPS will include invited talks, demonstrations, oral and poster presentations of accepted papers. Along with the conference is a professional exposition

Background We live in a digital world where many of us depend on customer service to be at its best. Whether it's calling about an order issue or needing help to activate an account, we are always reaching out to customer service to save the day. We are no longer

TL;DR: Applications are now open for the next cohort of Salesforce’s AI Residency Program. We invite researchers and engineers to join our 12-month training program, designed to help launch their careers while serving as AI Residents at Salesforce AI Research. Candidates based in the U.S. can apply

We summarize insights from our FAccT 2022 CRAFT workshop that focused on Concepts of Fairness & Transparency, Applied RAI Practices, Organizational Approaches to RAI & Cultural Change, and Public Policy & Regulation.

TL;DR: Deep reinforcement learning (RL), a powerful learning framework to train AI agents, can be slow as it requires repeated interaction with a simulation of the environment. Our original WarpDrive accelerates multi-agent deep RL on NVIDIA GPUs, enabling 10-100x speedups compared to alternative CPU+GPU implementations of multi-agent simulations.

AUTHORS: Chenghao Liu, Quang Pham, Doyen Sahoo, Donald Rose TL;DR: Nonstationary data, which changes its statistical properties over time, can make time series forecasting difficult. Despite the recent success of deep learning techniques for time series forecasting tasks, these methods are not scalable for applications where data arrives sequentially

TL;DR: The performance of existing time-series forecasting methods can degrade due to non-stationarity, where the statistical distribution of time-series data changes over time. Our new DeepTime method overcomes non-stationarity issues by leveraging a “forecasting as meta-learning” framework on deep time-index models. DeepTime achieves competitive accuracy on the long-sequence time-series

AUTHORS: Zeyuan Chen, Ran Xu, Luyu Yang, Donald Rose TL;DR: Many methods designed to preserve online privacy propose complex security measures to protect sensitive data. We believe that not storing any sensitive data is the optimal way to preserve privacy, so we propose a “burn after reading” online framework:

Imagine you find yourself in the cockpit of a next-generation spacecraft—the kind that can get you from low Earth orbit to the Kuiper belt without breaking a sweat. How do you envision controlling it? Science fiction has conditioned us to equate futuristic technology with dazzling complexity, so you might

As we say a fond farewell to summer (bummer!), let's look back and review some of the stellar work reported on by Salesforce AI researchers during the past few months. (For more details, we encourage you to click the link for each project to read the full blog post.) --------------------------------------------------------------------------------

TL;DR: LAVIS (short for LAnguage-VISion) is an open-source deep learning library for language-vision research and applications, offering comprehensive support for a wide range of tasks, datasets, and state-of-the-art models. Featuring a unified interface and modular design, it’s easy to use off-the-shelf and to extend with new capabilities. With

TL;DR: We developed a new time-series forecasting model called ETSformer that leverages the power of two frameworks. By combining the classical intuition of seasonal-trend decomposition and exponential smoothing with modern transformers – as well as introducing novel exponential smoothing and frequency attention mechanisms – ETSformer achieves state-of-the-art performance. Background Before diving

AUTHORS: Chen Xing, Mingfei Gao, Donald Rose TL;DR: Most AI object detection methods work only on limited object categories, due to the human effort required for bounding-box annotations of training data. We developed a new method that automatically generates pseudo bounding-box annotations of diverse objects from large-scale image-caption pairs,

We brought together 24 practitioners from 17 organizations to talk about a success in their practice they believed others could emulate or a challenge they were facing & wanted to brainstorm with others. We summarize the insights from 2 breakout discussions: Governance/Accountability & Incentives.

TL;DR:  Salesforce Research and Mila announce AI for Global Climate Cooperation, a working group collaboration and competition to design negotiation protocols and climate agreements. We plan to coauthor a peer-reviewed scientific paper with top-performing teams; insights will be distilled into a policy brief shared with leading policymakers, informing future

TL;DR: CodeRL is a new framework for program synthesis through holistic integration of pretrained language models and deep reinforcement learning. By utilizing unit test feedback as part of model training and inference, and integrating with an improved CodeT5 model, CodeRL achieves state-of-the-art results on competition-level programming tasks. The following

Conference Overview This weekend will kick off the thirty-ninth International Conference on Machine Learning (ICML). This conference specifically aims to bring together professionals who are dedicated to the advancement of Machine Learning (ML) in Artificial Intelligence. Participants at ICML come from many different backgrounds, including academic and industrial researchers, entrepreneurs

Conference Overview This weekend marks the start of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). NAACL provides a regional focus for members of the Association for Computational Linguistics (ACL) in North America. NAACL organizes annual conferences, promotes cooperation and information exchange among

Conference Overview The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR) is the annual conference on Computer Vision. CVPR is composed of both the main conference, as well as workshops and other courses, to provide a unique learning experience and networking opportunities in the field of Computer Vision. CVPR

AUTHORS: Sharvin Shah, Jin Qu, Donald Rose TL;DR: TaiChi is an open source library for few-shot NLP, designed for data scientists and software engineers who want to get some quick results or build proof-of-concept products but don’t have much experience with few-shot learning (FSL). The library abstracts complex

TL;DR:OmniXAI (short for Omni eXplainable AI) is designed to address many of the pain points in explaining decisions made by AI models. This open-source library aims to provide data scientists, machine learning engineers, and researchers with a one-stop Explainable AI (XAI) solution to analyze, debug, and interpret their

AUTHORS: Huan Wang, Aadyot Bhatnagar, Doyen Sahoo, Wenzhuo Yang, Steven Hoi, Caiming Xiong, Donald Rose TL;DR: Time series data is a critical source of insights for many applications, including IT Operations, Quality Management, Financial Analytics, and Inventory & Sales Management. While a variety of dedicated packages and software exist, engineers

TL;DR: We propose ALPRO, a new video-and-language representation learning framework which achieves state-of-the-art performance on video-text retrieval and video question answering by learning fine-grained alignment between video regions and textual entities via entity prompts. For more background (a review of key concepts used in this post), please see the

Lead Author: Xi Ye TL;DR: We propose RnG-KBQA, a Rank-and-Generate Approach for Question Answering over Knowledge Bases, which enables answering natural language questions over large-scale knowledge bases. Our approach is capable of answering questions about topics never seen in the training data, which makes it generalizable to a broad

TL;DR: WarpDrive is a flexible, lightweight, easy-to-use end-to-end reinforcement learning (RL) framework; enables orders-of-magnitude faster training on a single GPU. PyTorch Lightning enables you to modularize experimental code, and build production-ready workloads fast. Together, they can help significantly accelerate multi-agent RL R&D. Reinforcement Learning: Agents Learn by Maximizing

Conference Overview This year marks the 60th annual meeting of the Association for Computational Linguistics Conference (ACL [https://www.2022.aclweb.org/]). ACL is the premier international scientific and professional society for people working on computational problems involving human language, a field often referred to as either computational linguistics or

TL;DR: The AI Economist, a reinforcement learning (RL) system, learns dynamic tax policies that optimize equality along with productivity in simulated economies, outperforming alternative tax systems. We have now expanded this research, which is being published in the interdisciplinary scientific journal Science Advances. Humans or AI: Which Can Design

Conference Overview This year marks the Tenth International Conference on Learning Representations ( ICLR [https://iclr.cc/Conferences/2022]), one of the premier academic conferences dedicated to advancing research in representation learning - a type of machine learning also referred to as feature learning or deep learning. ICLR features the latest

AUTHORS: Tian Xie, Xinyi Yang, Angela Lin, Donald Rose Introduction and Background Creating a system capable of conducting a meaningful conversation with a human and helping them accomplish tasks is one of the ultimate goals of Artificial Intelligence (AI), and has been since AI’s beginnings. Meanwhile, as real conversational

Links: Research Paper [https://arxiv.org/abs/2203.13474], Github [https://github.com/salesforce/CodeGen] -------------------------------------------------------------------------------- Can you imagine a machine writing an app for you, just by telling it what you want? As futuristic as this scenario sounds, it’s actually here today. Salesforce AI Research outlines conversational AI

TL;DR: BLIP is a new pre-training framework for unified vision-language understanding and generation, which achieves state-of-the-art results on a wide range of vision-language tasks. Background For a review of some terms and definitions used in this blog, see our Appendix. Vision and language, two of the most fundamental methods

Recommendation systems are common in the consumer world. For example, Netflix, YouTube, and other companies use these systems to recommend items you would probably like, based on data about you - such as what items you've consumed (e.g., watched) before. But recommendation systems are not just for consumers. Enterprises

TL;DR: Today’s NLP models, for all their recent successes, have certain limitations. Case in point: they exhibit poor performance when processing multilingual code-mixed sentences (each containing multiple languages). Our new approach addresses this problem by constructing code-mixed inputs designed to degrade (or “attack”) the model, exposing the limitations

Salesforce is excited to announce its new AI Residency Program. We are inviting researchers and engineers to join our 12-month training program, designed to help launch their careers while serving as AI Residents at Salesforce AI Research.

Discover CTRLsum, a novel framework for controllable text summarization that enables users to control multiple aspects of generated summaries by interacting with the summarization system through textual input in the form of a set of keywords or descriptive prompts.

December 21, 2021 The leading AI research conference, NeurIPS 2021, has recently wrapped up, spanning seven very full days, 2,344 accepted papers, eight invited talks, ten tutorials, and nearly 70 workshops. Though there was diverse and innovative thought leadership on display, I found myself drawn to the particular topics

TL; DR: Introducing CodeT5 --- the first code-aware, encoder-decoder-based pre-trained programming language model, which enables a wide range of code intelligence applications including code understanding and generation tasks. CodeT5 achieves state-of-the-art performance on 14 sub-tasks in the CodeXGLUE code intelligence benchmark. -------------------------------------------------------------------------------- CodeT5 for code-related understanding and generation tasksGiven the

tldr: WarpDrive is an open-source framework to do multi-agent RL end-to-end on a GPU. It achieves orders of magnitude faster multi-agent RL training with 2000 environments and 1000 agents in a simple Tag environment. WarpDrive provides lightweight tools and workflow objects to build your own fast RL workflows. Check out

This blog accompanies the interactive demo [https://einstein.ai/the-ai-economist/ai-policy-foundation-and-covid-case-study] and paper [https://arxiv.org/abs/2108.02904]! Dive deeper into the underlying simulation code [https://www.github.com/salesforce/ai-economist ] and simulation card [https://github.com/salesforce/ai-economist/blob/master/COVID-19_Simulation-Card.pdf] . In this blog post, we

August 5, 2021 In June 2021, four of us from the Salesforce AI Ethics and Conversational Design teams collaborated with the Montreal AI Ethics Institute [https://montrealethics.ai/] (MAIEI) to facilitate a workshop on the responsible creation and implementation of chatbots and conversational assistants. Connor Wright [https://www.linkedin.com/

AI can learn the language of biology to create artificial proteins across multiple protein types that are functional and unseen in nature.

> TL; DR: We propose a new vision-language representation learning framework which achieves state-of-the-art performance by first aligning the unimodal representations before fusing them. Vision and language are two of the most fundamental channels for humans to perceive the world. It has been a long-standing goal in AI to build intelligent

An AI model can generate sequences across natural language and proteins with attributes that go beyond the training distribution