Writing an AI Chatbot in Rust and Solid.js

Here is the full source code if you want to use to use as a starting point for any of your projects.

The motivation

I want to explore the viability of using Rust to write AI chatbots while measuring the performance difference, speed of development, and productivity.

I opted to use Axum because I didn’t need any complicated features and figured since it is a very thin layer over hyper, it would be fast to implement basic features. Although, if you need more advanced features out of the box in the sense of batteries included type of frameworks, I would suggest giving Leptos a look.

For the frontend I wanted to give solid.js a try so I went on with that using only the features included in the core library without pulling any community packages only adding tailwind to give the UI a better look.

The Experience

I’ve always seen AI projects being written in Python and sometimes in Node.js. I didn’t have my hopes up for using Rust, but I was pleasantly surprised. I used the rustformers-llm crate, which was surprisingly good and very straightforward to use. I also followed the performance optimization recommendations in the repository like this one in cargo.toml :

[profile.dev.package.ggml-sys]
opt-level = 3

Which made a very big difference in compilation times.

I am using an commodity machine to run the llama, so waiting for 30 to 40 seconds was really good outcome for my expectations as running similar models through python really gave me a hard time.

Certainly! Writing Rust code can be both rewarding and challenging. One of the quirks I encountered while working with Rust is the lack of automatic Clone implementation for certain types, such as the llm struct. This means that if you want to create a copy of an instance of llm, you’ll need to implement the Clone trait manually on wrapper type.

However, this aspect of Rust serves a purpose. By requiring explicit Clone implementations, Rust encourages developers to think critically about the cost of copying data and encourages the use of more efficient alternatives, such as borrowing or passing references.

I ended up with the quirk below for the router to avoid making the code longer :

let app = Router::new().route("/api/chat", post(handler)).layer(CorsLayer::permissive()).with_state(Arc::new(mdl));

and in the handler I went for : let res = llm_model::gen_conversation(&*mdl, chat);

SolidJS is very similar to React but with a much simpler syntax. I don’t have much to add regarding the frontend experience other than it was pretty straight forward.

Conclusion

When it comes to developing AI chatbots, using Rust can prove to be an immensely productive choice. Rust offers a seamless development experience, allowing developers to create intricate and sophisticated chatbot systems efficiently. One of the most notable advantages of Rust is its robust memory safety guarantees, which significantly reduces the risk of runtime errors and memory leaks.

Unlike some other programming languages, Rust’s borrow checker ensures that developers adhere to strict ownership and borrowing rules, helping to prevent common pitfalls and bugs. Although it may require some initial effort to understand and navigate the borrow checker, once you become accustomed to its rules and norms, working with Rust becomes a smooth sail.

In addition to its safety features, Rust boasts exceptional performance, making it an ideal choice for AI chatbot development. The language’s low-level control and efficient memory management allow chatbots to handle complex computational tasks quickly and with minimal resource consumption.

Moreover, Rust’s extensive ecosystem provides a multitude of libraries and frameworks that can significantly streamline the chatbot development process. The availability of libraries such as rust-bert for natural language processing and conversational for building conversational agents enables developers to leverage existing tools and focus on building the core functionality of their chatbots.

By opting to use Rust, you not only harness the power of a modern and expressive programming language but also ensure that your AI chatbot is equipped with the utmost efficiency and reliability. So why not dive into Rust and unlock the potential of building AI chatbots that deliver exceptional user experiences?