AI. 2023 Recap & 2024 Outlook.
In 2023, we spent a good portion of the year experimenting, playing, and exploring with AI — all in preparation for what 2024 will unleash. Last year, I wrote thoughts and predictions for the upcoming 2023. MidJourney V4 has just come out and we had no idea how AI motion/animation would come screaming at us at full force — and it changed everything. In less than 6 months, AI has ushered in a new era and a new type of filmmaking.
So before we dive into 2024, let’s look at how we used AI in 2023 in commercial and personal work. I started a new company called Secret Level during the summer of 2023. Secret Level is a next-generation entertainment studio. We are a collective of some of the smartest storytellers and makers in Hollywood, gaming, technology, and advertising — and we are all in on AI.
Throughout the year, we used AI in all stages of ideation, proof of concept, and throughout the entire production process of all our commercial projects. Dave Clark, one of our closest collaborators at Secret Level, and I discussed creating a spec commercial with AI and a half day later, this Adidas commercial was created. It went viral and was a culture-defining moment in AI.
Justin Hackney used NERF technology and photogrammetry in a music video and interactive website for the artist Baby Queen. I think we will see this technology integrated more into the web in 2024.
Years ago, to capture a person or object in 3D, you had to go to a volumetric capture studio and spend a great deal of time and money to get decent results. When we were shooting a project with Snapchat in Mexico City, we used AI to volumetrically capture actors and sets using just an iPhone.
We used AI throughout the project, but one of the most interesting aspects was creating the actual Phantom House using a combination of AI tools — from Midjourney to Photoshop AI. The final house was created as a 3D CG model that was used for VFX, an AR lens, and a 3D printed house that is on the shelf in the antagonist’s study.
In October, Justin Hackney also created a mind-blowing reimagining of Alice in Wonderland, through the lens of a Matrix-like world built by her father.
The ability to conceive a trailer for a project that can generate hype for pitches to movie studios wasn’t easily accessible in December 2022. In just one year, AI artists can easily add motion, celebrity VO, AI-assisted VFX, — opening up a brand new world of AI-driven storytelling. Rewind to March 2023, when I first combined an AI image with an AI-generated voice and lip-synced animation. At the time this was ground-breaking. It’s amazing how far we’ve come in only 9 months.
The spring and summer of 2023 were the competing betas of Runway’s Gen1 and Pika Labs text to video. The early results were very crude but also horrifying (which I love!). I created two text-to-video horror short films, but this Pika film, The Motel Room, was the closest to the nightmare fuel of a classic music video from the 90s.
Dave Clark ignited the internet, yet again, with the trailer for his upcoming feature film, Another.
Using a combination of traditional filmmaking with the latest AI visual effects, Dave was able to create a new type of post-production and pre-viz pipeline that is the future of filmmaking. We are going to see AI augmenting live-action and VFX throughout 2024.
And I can’t close out a 2023 recap without talking about audio. Suno.AI completely blew me away with its ability to write and produce a sometimes catchy song in any genre. I asked it to write a song about the sad coffee maker that was being replaced with a new one for Christmas — in the style of a vintage crooner. Create any song, in any style, instantly. It’s like magic.
There are dozens and dozens of new tools that are coming out of beta that are going to increase the quality of images, video, and audio. The most recent addition to everyday AI tools is the Magnific AI, an upscaling tool. Below you can see the results of Midjourney V6 on the right and the detail it could add via its AI engine — the extra detail (skin textures, hair, etc) that just made everything look more “real”.
2024: The Year Everything Became Real
If 2023 was the beta year, 2024 is going to explode with people embracing AI in commercial and feature film work. To be honest, I haven’t been this excited about the future of tech and storytelling since 1995 — at the dawn of the internet. We are on the verge of massive leaps that will seamlessly combine AI, storytelling, and technology in ways we can only dream of today, but within MONTHS will be fully realized in reality.
I put together some insights, informed predictions, and thoughts on some big moments for AI, video, and experiences in 2024. The categories are Ultra-Real AI Video, Spatial and Mixed Reality, Real-Life Virtual Characters, Gaming, Mixed Reality Fan Experiences, 3D Gaussians, and Branded Gamfication.
Ultra-Real AI Video
With very advanced modeling in Stable Diffusion to many AI motion-based libraries, AI will evolve to understand how everything and anything moves. The quality of Runway’s Gen2 and Pixalabs has grown insanely fast in only 6 months — just wait until we add a few more competitors to the mix. Stills or text to video isn't going away, it will just evolve exponentially.
3D will be another game changer — from changing camera positions in a Midjourney photo to developing 3D assets from AI-generated images. 3D allows filmmakers and storytellers to clearly define the lensing, camera movements, and elements within a scene. By the end of 2024, I believe we will have fully integrated programs that allow us never to step footage on a set and imagine it all, virtually, using AI. While this will NEVER replace traditional filmmaking, it will simply augment storytellers with more tools to visualize what is bouncing around in their brains.
Spatial & Mixed Reality
Apple’s headset is going to launch in the first few months of 2024 and with it will come the dawn of spatial computing and spatial video. The iPhone 15 is already rigged to capture spatial video and the only way to truly experience it will be on the Vision Pro.
This will usher in a whole new era of photography and video that will be centered around experiencing moments.
I am very excited about the continual blurring of the lines of reality and virtual. This example of drawing in the real world, that is affecting the fabric of a virtual character is just the tip of the iceberg of what is coming.
Real-Life Virtual Characters
We’ve had countless conversations in 2023 about creating virtual characters — from full virtual bands to bringing back deceased celebrities who can be puppeteered by living actors. 2024 will be the year to bring to life many virtual characters that will live and move and talk, just like any other being. Virtual consciousness will continue to evolve in 2024, and we should see a hit song by a virtual band, maybe a virtual movie star, and the rebirth of many deceased actors that you will be able to interact with. With so many discussions happening in 2023, I anticipate 2024 to be the year we make it real.
Gaming
From NPC that’s you can have endless conversations with, to video games that created entirely from AI to games that will simply never end — 2024 will see a massive infusion of AI throughout the video game industry. These games will be called AI-first and will dominate the video game industry from now on.
Earlier in 2023, Replica Studios introduced a take on the Matrix Awakens demo in which you can have endless conversations with NPC using your voice using a GPT-like backend. This is certainly the future of video games.
But creating characters that can have endless conversations with the audience isn’t just for video games. Earlier this year, we developed a full Character LLM for a personality we were creating for a commercial — a character that understood what it meant to be human because of ______ product. For this, I used Inworld, which was the perfect tool to develop the rules and backstory for a character — allowing me to have full audio conversations with it. There are so many applications outside of video games that make this exciting.
Haz Dulull, one of our collaborators, and his team created Moontopia — a groundbreaking Fortnite island. Go to where your audience is and give them an incredible experience powered by your brand. We are super hot on this and are in the early process of developing several branded Fortnite islands.
Mixed Reality Fan Experiences
Experiential storytelling is exploding right now. Netflix is the first major traditional entertainment studio to completely go for it with the opening of Netflix Houses.
Fan experiences will dominate 2024 and beyond. Super Mario Land opened at Universal Studios, blending a plethora of mixed reality moments — using virtual to enhance the real world and vice versa. Every fan-based experiential project should seamlessly blend the real world with an augmented reality piece that enhances your experience.
Last year, we had to pleasure of creating the Hulu Motel — which connected dozens of Hulu-related IPs into one seamless location. The experience had over 30 immersive theater actors who checked you into the hotel, played characters from shows, and overall immersed people into the story — from an Only Murders in the Building escape room to bikers from the Mayans.
3D Gaussians
NERFs, Gaussian Spalatting, and Radiance Field Rendering will continue blurring the lines of reality vs digital worlds. With the ability to scan the real world and instantly create 3D environments that can be easily uploaded to the web and other digital environments, the merging of our two realities will quickly.
I am most excited about using this in actual production — being able to change angles after you’ve filmed a scene, and exploring a scene that someone has shot from multiple cinematic angles.
Instead of AI creating images from scratch, we will use this technology to enhance filmed experiences, and using WebGPU, we will be able to do all of this in a browser and on mobile.
Branded Gamification
2024 will be a year of gamifiying real life. There are over 3 billion gamers in the world and as our real world and virtual worlds begin to collide, we will see true gamification in all forms of advertising and brand loyalty. Collaboration, leaderboards. challenges, personalization, and badges — all will create deeper experiences for the audience and strong connections to the brand.
Experiences in both the real world and virtual worlds that let the audience truly interact with your content are the most rewarding. Letting the audience play a role in the story is crucial to any great gamified experience — and luckily, I’ve been able to do it for the past 20+ years with programs like Elf Yourself, Remote Control Tourist, Take This Lollipop, Twenty One Pilot’s Never Ending Music Video.
Finishing Up
There is SO MUCH going on right now and at insanely accelerated speeds — it’s almost impossible for any one person or company to keep up. This was one of the main reasons I decided to start Secret Level. The world of technology and storytelling is moving at a breakneck pace and we need experts from all disciplines working together to share, experiment, and work together.
I’ve said this every year (for years), but I mean it now more than ever:
This is, hands down, the most exciting time to be a storyteller who embraces technology.
Later this month we will be launching a project that combines a TVC, social media content, and an interesting app that all heavily features AI. It’s been exciting to build a company that not only stands at the forefront of storytelling + technology but is positioned to always be one step ahead. Follow me for more updates on what will be an incredibly exciting year of innovation.
— Jason
