Databricks Shakes Up the Chatbot Market with Open Source ChatGPT Alternative
The immense popularity of OpenAI‘s ChatGPT has taken the tech world by storm, sparking an arms race among companies to develop their own advanced conversational AI models. Databricks, a leading provider of cloud-based data warehousing and analytics solutions, has recently thrown its hat into the ring with the release of Dolly – an open source chatbot model that promises to give ChatGPT a run for its money.
Democratizing Conversational AI
Databricks‘ primary motivation behind Dolly is to make large language models more accessible and affordable for a wider range of companies. While OpenAI has begun offering paid plugins for businesses to customize ChatGPT for their needs, the costs can be prohibitive for smaller organizations with tighter budgets.
By open sourcing Dolly, Databricks aims to level the playing field and "democratize the magic of ChatGPT," as they put it in a recent blog post announcing the release. Now, potentially millions of companies can leverage the power of Dolly to train their own custom chatbot models at a fraction of the cost of licensing ChatGPT or building a model from scratch.
This has major implications for the burgeoning conversational AI market. Up until now, OpenAI has had a near-monopoly with the viral success of ChatGPT. But with viable open source alternatives like Dolly emerging, we could see a proliferation of advanced chatbots across industries as the barriers to entry come down. More competition should drive innovation and make the technology more powerful and cost-effective over time.
According to a 2023 report by Grand View Research, the global conversational AI market size is expected to reach $32.62 billion by 2030, registering a CAGR of 23.6% from 2022 to 2030. Within this massive and fast-growing market, open source solutions like Dolly could carve out a significant niche by appealing to companies priced out of enterprise offerings.
"Language models have become increasingly powerful in recent years, but access has been limited to fairly small and privileged groups," says Thomas Wolf, co-founder and Chief Science Officer at Hugging Face, an open source platform for machine learning. "The release of Dolly is a major step towards making this technology available to a much wider audience. It has the potential to unlock a lot of value and innovation across industries."
How Dolly Fits Into Databricks‘ Strategy
To understand the significance of Dolly, it‘s important to situate it within the context of Databricks‘ broader business. Founded in 2013 by the original creators of Apache Spark, Databricks has become a powerhouse in the $100 billion data and analytics industry, with a valuation of $38 billion as of its most recent funding round in August 2021.
The company‘s core product is the Databricks Lakehouse Platform, which combines the best elements of data warehouses and data lakes to create a unified solution for storing, processing, and analyzing structured and unstructured data. Built on top of open source technologies like Spark, Delta Lake, and MLflow, the platform enables enterprises to do everything from ETL to machine learning on massive datasets.
While Databricks has seen strong adoption of its Lakehouse offering, with over 7,000 customers including 40% of the Fortune 500, the company is always looking for new ways to add value and expand its footprint. This is where Dolly comes in.
At first glance, a chatbot model may seem like an odd fit for a data management company. But viewed through the lens of Databricks‘ mission to make big data accessible and actionable for everyone, it starts to make more sense. By open sourcing Dolly, Databricks can showcase its AI capabilities and entice developers and data scientists who may not be using its platform currently.
Once companies start building on top of Dolly and seeing its potential, the thinking goes, they‘ll be more likely to turn to Databricks for their broader data needs as well. In this way, Dolly serves as a gateway drug of sorts to the Databricks ecosystem.
There are also synergies in the other direction. Companies already using Databricks for data storage and analytics can seamlessly integrate Dolly into their pipelines to create intelligent chatbot experiences powered by their proprietary datasets. By making it easy to deploy custom chatbots at scale, Databricks can help its customers unlock new use cases and drive business value from their data investments.
Under the Hood of Dolly
From a technical perspective, what‘s most impressive about Dolly is the speed and efficiency with which it was developed compared to ChatGPT. Whereas OpenAI spent untold sums training its model on a huge corpus of online data, Databricks took a much more streamlined approach.
Rather than start from scratch, Databricks built Dolly using GPT-J, an open source language model created by independent AI research group Eleuther AI. GPT-J shares a similar architecture to OpenAI‘s GPT-3 model that powers ChatGPT, with a few key differences:
- GPT-J has 6 billion parameters compared to GPT-3‘s 175 billion, making it more computationally efficient
- GPT-J was trained on a smaller dataset filtered for higher quality
- GPT-J uses less than one-fifth the carbon footprint of GPT-3 thanks to more sustainable training practices
By building on top of GPT-J, Databricks was able to develop Dolly with a fraction of the resources and environmental impact of creating a model like GPT-3 from scratch. This allowed the company to bring Dolly to market faster and at a lower cost than if it had gone the proprietary route.
So how does Dolly actually stack up to ChatGPT in terms of natural language understanding and generation? While no fully transparent benchmarks have been released yet, early anecdotal reports are promising.
In a Twitter thread, Databricks engineer Shriphani Palakodety shared examples of Dolly engaging in freeform conversation, answering followup questions, admitting mistakes, and even writing code. The model exhibited an impressive command of language and ability to perform complex reasoning.
Of course, more rigorous testing is needed to truly assess Dolly‘s capabilities. But if it can come close to matching ChatGPT‘s performance with a slimmed down architecture, that bodes well for its potential to be fine-tuned for a variety of downstream tasks.
Applications and Use Cases
Once released into the wild, an open source model like Dolly can be adapted and applied to all sorts of contexts. Some potential use cases include:
Customer service automation: With Dolly, companies could build chatbots to handle routine inquiries and support requests, freeing up human agents to focus on more complex issues. The model‘s strong language understanding could enable more natural and efficient issue resolution.
Educational content creation: Dolly could be fine-tuned to generate explanatory content, answer student questions, and even provide interactive tutoring across different subject areas. This could help make quality education more accessible and personalized.
Creative writing assistance: By ingesting large corpora of literature, Dolly could be used to aid in story outlining, character development, and even drafting entire novels or scripts. It could serve as an AI-powered muse for writers looking to spark new ideas.
Research and analysis: Trained on scientific and academic literature, Dolly could help researchers parse complex ideas, generate literature reviews, and even propose new hypotheses or experiments. It could accelerate the pace of discovery in fields from medicine to social science.
Personal assistants: Dolly could power the next generation of smartphone AI assistants, with more contextual awareness and task completion abilities. Imagine having a pocket-sized chatbot that can help manage your calendar, answer questions, and even handle open-ended requests.
The beauty of an open source base model is that it can be customized and optimized for countless niche applications. As more developers experiment with Dolly and share their results, we‘re likely to see an explosion of creative use cases across domains.
Responsible AI Development
As with any powerful technology, the release of Dolly also raises important questions about ethics and responsible development. While open sourcing AI models can democratize access and accelerate innovation, it also creates risks around misuse and unintended consequences.
To its credit, Databricks has taken proactive steps to mitigate these risks with Dolly. The model was trained using oversight from the company‘s internal AI ethics committee to avoid perpetuating social biases and toxic outputs. The training data was carefully filtered and audited for sensitive content.
Databricks has also published detailed documentation on Dolly‘s intended use, capabilities, and limitations. This includes guidance on handling outputs and interacting with the model responsibly to prevent harmful behaviors.
That said, once Dolly is open sourced, Databricks will have limited control over how it is ultimately used and modified by third parties. It falls on the broader AI ethics community to establish norms and best practices to ensure open source models are developed and deployed safely.
Some key considerations include:
- Implementing proper safeguards against misuse, such as content filtering and usage monitoring
- Requiring transparency around training data and model architectures to enable auditing
- Investing in AI literacy education for users and setting appropriate expectations
- Engaging diverse stakeholders in the development process to minimize blind spots
Initiatives like the Partnership on AI and the Open AI Integrity Project provide forums for different actors to collaborate on these challenges. By proactively addressing ethics concerns, the AI community can work to maximize the benefits and minimize the risks of open source models like Dolly.
Competitive Landscape and Future Outlook
Databricks is hardly the only company looking to challenge OpenAI‘s dominance in the chatbot space. In the wake of ChatGPT‘s massive success, incumbent tech giants and startups alike have been racing to develop their own conversational AI offerings.
Google, for one, is rumored to be readying multiple ChatGPT competitors for release in 2023, including a search chatbot called Apprentice Bard and a conversational version of its PaLM language model. The company has reportedly instituted a "code red" to rapidly refocus efforts on AI in response to the threat posed by OpenAI.
Meanwhile, Meta has touted the conversational abilities of its BlenderBot model and begun testing a new AI called LLaMA adapted from GPT-3. Anthropic, an AI safety startup co-founded by OpenAI‘s former VP of Research, has developed an advanced chatbot called Claude using "constitutional AI" techniques to imbue it with values like honesty and kindness.
And this is just the tip of the iceberg. Countless other companies big and small are working on their own takes on the technology behind the scenes. With so many well-funded players vying for a piece of the conversational AI pie, we‘re likely to see rapid advancements in the field in the coming years.
So where does this leave open source upstarts like Dolly? While they may lack the resources and brand recognition of the tech titans, they have a few key advantages. For one, their barrier to adoption is much lower since companies can freely access and build on top of the models without costly licensing.
This could enable a grassroots groundswell of development and innovation around open source chatbots that outpaces what any single company can achieve. Just as community-driven projects like Linux and Tensorflow have become ubiquitous in their domains, we could see open source language models emerge as the default for many applications.
Open source chatbots may also have an edge when it comes to trust and transparency. In an era of growing concerns around AI ethics and bias, being able to examine a model‘s code and training data is becoming increasingly important for organizations. Open source provides that auditability in a way that closed-source systems from Google or OpenAI cannot match.
"As AI systems become more prevalent and high-stakes, the ability to independently verify their behavior and hold them accountable is critical," says Rachel Thomas, director of the Center for Applied Data Ethics at the University of San Francisco. "Open source chatbot models like Dolly are a step in the right direction towards making AI development more transparent and trustworthy."
Ultimately, the success of open source chatbots will depend on the strength of the communities that coalesce around them. If enough developers buy into the vision and contribute their efforts, we could see models like Dolly quickly close the capability gap with their proprietary counterparts. Robust governance structures and funding models will also be key to sustaining development over the long run.
Databricks, for its part, seems committed to fostering a vibrant ecosystem around Dolly. In addition to releasing the base model, the company has pledged to provide ongoing support and tooling to help developers build and deploy custom chatbots. It‘s also hinted at plans for a more purpose-built AI model for conversational use cases down the line.
As the technological building blocks for chatbots become increasingly commoditized, the real differentiator will be in the data, fine-tuning, and domain expertise that companies bring to bear. Expect to see a proliferation of specialized chatbots optimized for different industries and use cases, with a mix of proprietary and open source underpinnings.
In the end, the genie is out of the bottle when it comes to conversational AI. ChatGPT has shown the world what‘s possible and whet the public‘s appetite for more. The race is now on to democratize this transformative technology and make it accessible to all. Open source models like Dolly have a crucial role to play in shaping that future – one conversation at a time.
