Sprained Ankle x Hackathon

Fresh out of a hackathon opening ceremony, brimming with ideas and eagerly eyeing my barbeque pizza, I got a tad careless.

The stairs were tall and hard, and as I tripped and fell, I tried my best to catch my pizza 😭. Though I succeeded, my efforts would probably have been better served at protecting my ankle, which almost immediately began to balloon.

I was with my friend Anshi (shoutout Anshi ❤️), who slowly helped me walk back home, and I ignored the swelling, deciding to sleep it off. As it turns out, this probably wasn't the greatest decision.

The Next Morning

When I woke up the next morning, with a foot faintly resembling a clown shoe, I realized sleeping it off probably hadn't been the best strategy. I slowly hopped to the health center on campus, and got my first ever boot and crutches.

• PS: Walking on crutches is an INSANE workout. I felt like I did an hour of pilates after 10 minutes of walking 😂. I should probably start going to the gym...

Since it was the weekend, and the x-ray technician had to drive in, I was sitting in the waiting room for a pretty long time. The hackathon's only prompt was that our project had to use AI, but I narrowed it down to a learning based platform, and ... Memora was born!

The Idea

In ancient times, people would hold massive books worth of information in their minds. They did this using "memory palaces". You might've heard about it in shows like Sherlock or Hannibal.

The Story

Simonides of Ceos was a Greek poet at a very important party. It was a room full of aristocrats, military leaders, and philosophers, and when Simonides went outside during dinner to receive a message, he never could've anticipated that the roof would collapse. Though everyone was crushed, he identified every single body.

How? A memory palace. By using the physical location to arrange mental imagery, he remembered an incredible amount of information in order. To this day, people use this technique to memorize digits of Pi, the periodic table, and so much more.

What I Built

My guiding thought was, If people could build memory palaces back in the day, how much easier could it be with the help of AI and Augmented Reality?

Regardless of what you have to memorize or what test you have to study for, AI can break it into building blocks that can be placed around a familiar location using Augmented Reality. You could reap all the benefits of making a memory palace without much of the planning.

I finally had an idea - time to get to the code!!!

Starting to Code

• Main Tech Stack: Google's Gemini API, Swift + ARKit
• Things I didn't know: iOS development, ARKit, GeminiAPI
• Things I knew: How to learn :)

Since I didn't have any experience with native iOS development, the very first thing I did was download the latest version of XCode (which took 3 hours !!). While it was downloading, I started playing around with the Gemini API using Python, getting it to generate objects for a memory palace.

Finally, when Xcode was set up, I opened up Xcode's Reality Composer Pro and started playing around with usdz files. I've never worked with 3D object software before, so this part was awesome 💖

By this time, I ended up getting called in for an x-ray, which came back inconclusive (they didn't know if it was a fracture or a sprain), and I went home and crashed 😴. The next morning, though, I woke up at 3 AM and got back to work!

Starting out with some basic Swift tutorials, I built out a simple starting frontend, and made my first object in ARKit, a solid cube.

It took me a while to get this displaying on the user's side, but after that, I was on a roll, setting up the Gemini API to make calls from the text the user passed in, and using that data to create and display AR objects. It hit 10 AM on Sunday, and since my parents had come over to visit on Dad's Weekend, and the hackathon deadline was 12 PM, it was time to submit. Here's what I had at that point!

Some Difficulties + Lessons Learned

This was a huge learning process, with tech that was all new for me. Here are some specific bumps I spent a lot of time working through:

1. iOS Compatibility: Since I was testing an AR app, I had to use a physical phone instead of a simulator. There was just one problem—my phone is on an extremely old iOS version 😭. It took a lot of playing around to understand the dependencies between different iOS versions and ARKit, but I learned so much about iOS dev from it!
2. Music Feature: I wanted to add Gemini-generated audio as a user enters the AR memory palace, but it turned out to be a huge struggle to embed that audio dynamically and have it play only when the user opened the memory palace. It's still a work in progress 😅

Next Steps: I plan to continue expanding the app so that the text and AR objects produced are more cohesively grouped, ensuring the visualization looks good regardless of room size.

"The beautiful thing about learning is nobody can take it away from you." – B.B. King

Conclusion

This was the most chaotic weekend I've had in a while, but now I have a newfound understanding of Xcode and RealityKit, and I own a boot!