This was a project I started essentially out of curiosity. I wanted to see how far I could go with building a react native mobile app using Cursor AI IDE and expo. Now it turned to something I genuinely want to build and complete.
Think of it like an email app that acts like your personal assistant helps you find emails, tasks and meetings. Each meeting would have an AI agent that records and transcribes your meeting and it shows within the app.
All emails are indexed and sent to your personal vector database - right now I'm using pinecone along with Claude Haiku via Openrouter for the RAG setup. This essentially makes it easy for the agents to search your emails and is also privacy first so your private emails are safe. However I'm looking to update the vector db setup to use a local index called Objectbox so it runs locally.
You can also send voicenotes to the model and it responds. It takes in your voice input the transcribes it using the Whisper API from Open AI then sends the transcribed text as a tool request to claude which then checks your vectordb and returns the data you requested for. Of course this process takes about 15-30 seconds for now, I'm hoping to shorten it by integrating Objectbox locally for the vector search.
Authentication is handled by Clerk and it works seamlessly. I haven't launched the app yet but I'm running the server locally by tunnelling my localhost using Cloudflare because i needed https traffic and of course Ngrok wants me to pay for it.
I also burned a lot of tokens trying to do email summaries, which I had to take off until I figure out the best way to do it. For now the RAG setup via Openrouter is enough.
I also initially had used Elevenlabs to respond to voice input via the chat but I decided to yank it off and try to use telephony, where you can call the agent. Tried Hume and it didn't really work out as i expected. I need to try Livekit and Deepgram next to see the next option to use. It's been an exciting experience so far - Follow my thread on X for more updates on this.