Back to Coding: Update 3

I am happy to report that Chaai assistant will soon be recruiting alpha testers, just in time for back-to-school. Details on how to sign up are coming soon! See who is eligible.

In the last 3 months I have donned many hats while building Chaai. When I spent a whole week focused on just one aspect of the product, I realized that I cannot afford to do this for everything on the todo list. A reminder of the ever-present tradeoff between making a useful product for the user, and getting something to market quickly to learn and adjust. I had to learn to be ok with sharing when the output is “good enough”.

Still I have spent a high percentage of my time testing, and fixing big and small bugs. I don’t want bugs to become a distraction to the real feedback I want from the alpha users. This testing/bug-fixing experience is a reminder of how engineers spend eighty percent of their development time taking the product from eighty to ninety percent. No product is ever hundred percent. Living products are constantly changing and users often use them in unexpected ways, unlocking new bugs.

Another significant development in the last 3 months was the announcements at WWDC, Apple’s annual developer conference. Below are some thoughts on Apple Intelligence’s impact on Chaai:

• Chaai’s first use case (calendar-management) is a transactional use case. I believe transactional use cases are at the highest risk of being solved by the platform. But they are also the simplest to solve (take some unstructured data e.g. your email messages, parse into some structure and forward to an existing tool i.e. calendar). A context aware Siri will be able to do this easily. But will Apple support this workflow for google mail? Or Outlook? This is why the basic transactional use cases still make sense for Chaai.
• I am a fan of the context aware/screen aware update of Siri. While building Chaai, we spent a lot of time figuring out and perfecting the app extension to make it easier to upload screenshots to Chaai. Context aware Siri will make it much easier for apps like Chaai to get the relevant information and help the user.
• I am also excited about an on-device LLM model. If this was available today, Chaai would save a round trip to GPT which in turn will save money and improve the performance of the app.

Finally, I wanted to share one of my aha moments. As I work on this problem, I am amazed by the capability of the LLM technology to chain our tools e.g. see my example where I ask for a running schedule, make any changes I need, and then have it added to my calendar as if I am working with an actual human assistant.

If you are curious, here are things I thought through on my way to the alpha build:

1. Features and Quality: What features should be in the alpha? How much testing is enough esp since LLMs are non deterministic in their behavior? Which supporting features (like onboarding, notifications, settings, help) should I build now?
2. Audience: Who is my alpha’s target audience? Where do I find my target audience? How do I recruit members of my target audience? What marketing materials do I need? How do I make the product available to my alpha users?
3. Business: What are the mechanics of setting up a business? What is my budget for the alpha? Should I be raising money? The biggest tradeoff for me in this bucket is that I cannot build the product and chase investors at the same time. Both of these activities are intensive, especially for a newbie.

As with each update, here are some technical learnings:

• As we were building features like daily summaries, upcoming weekend summary, our token usage skyrocketed. In June I spent 5 times more money on ChatGPT compared to May. I thought I was testing a lot (I was!), falling into the confirmation bias trap. But then I noticed that the app on my phone was becoming slower and slower. GPT was often confused. The problem was that Chaai was maintaining a single thread with GPT i.e every time GPT was responding to a user query it was going through the whole conversation history which could have been two months long. It seems obvious now! We had to think through the various strategies to reset the thread. In general, it is a very good idea to have a truncation strategy when building against GPT’s assistant API. You can simply set the truncation-length on the run.
• As the app now supports coach capability (and very soon meal planning), we added a layer for intent detection so that the user request can be sent to the right agent. Right now the call for intent detection is going to GPT. This has implications on the performance of the app and the user’s perceived delay. But we were willing to live with this until iOS supports intent detection on the device through on-device LLM capability. I foresee a major re-architecture when we build multi step planning as the app gets more complex.
• We added an app extension so that you can send screenshots to Chaai, similar to sending screenshots to the Messages app. The app had to be rearchitected so that the main app was embedded within a higher structure that contained both the app and the app extension (see cocoapods development pods). We ran into bugs using SwiftData between the app and the app extension. Data changes in one are ‘half’ available in the other until a cold start forces a complete sync. A lot of time was spent bandaging this issue. We considered enabling cloudkit but the significant changes to the data model added additional risk for the alpha. Google integration gave a bit of trouble working between extension and app but it simply due to misconfiguration of keychain.
• Some prompts are evolving to use one-shot/templates. This has been working well! Here is an example I used in the prompt that generates the weekend notification: E.g. Hi <name>! Looks like you have a <describe the weekend based on the type of events and the number of events the user has scheduled eg. packed/open/relaxing/hectic> weekend coming up starting with <use the event data to create this section eg. dinner this evening/your saturday run/your sunday swim>. See more details, and get organized for a <fantastic/beautiful/boss/some-interesting-adjective> weekend!

Thanks for reading. If you are convinced that the needs of parents and families are important in shaping the future of AI technology, I would love to hear for you.