Sora, OpenAI’s New Video Generation Model Creates Dreadfully Mesmerizing Videos

From a street scene in Tokyo, through the California goldrush and Italy’s Amalfi coast, to amazing nature footage – Sora has it all. It’s amazing and also somewhat alarming to see how realistic videos it can generate from simple prompts.

A new video generation model has been unveiled by OpenAI, a pioneer in the development of artificial intelligence (AI). Sora converts text instructions into a video of up to one minute, according to the company, which is also developing the ChatGPT chat application and the DALL-E image generator. Sora can generate video not only from text, but also from still images, and can extend or add missing details to existing video. This is a remarkable achievement that could have many applications and implications for the future of media and communication.

Sora is based on a deep neural network that learns from a large dataset of videos and captions. It can handle a variety of domains, such as animals, sports, landscapes, and human faces. Sora can also generate videos with sound and motion, adding to the realism and immersion.

To use Sora, one simply needs to provide a text description of the desired video, such as “a cat playing with a ball of yarn” or “a person skiing down a snowy mountain”. Sora will then synthesize a video that matches the description as closely as possible, using its learned knowledge and creativity.

In their announcement, you can also watch 38 example videos, all of which they claim were created by Sora one by one, based on the short text instructions alone, and without any post-production modifications – which sounds pretty incredible given the quality of the videos and how photorealistic some of them are.

Here are some examples of Sora’s videos, alongside the textual prompt from which the model generated them.

Prompt: A stylish woman walks down a Tokyo street filled with warm glowing neon and animated city signage. She wears a black leather jacket, a long red dress, and black boots, and carries a black purse. She wears sunglasses and red lipstick. She walks confidently and casually. The street is damp and reflective, creating a mirror effect of the colorful lights. Many pedestrians walk about.

Prompt: Drone view of waves crashing against the rugged cliffs along Big Sur’s garay point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore. A small island with a lighthouse sits in the distance, and green shrubbery covers the cliff’s edge. The steep drop from the road down to the beach is a dramatic feat, with the cliff’s edges jutting out over the sea. This is a view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highway.

Prompt: Extreme close up of a 24 year old woman’s eye blinking, standing in Marrakech during magic hour, cinematic film shot in 70mm, depth of field, vivid colors, cinematic

Prompt: Historical footage of California during the gold rush.

Prompt: A large orange octopus is seen resting on the bottom of the ocean floor, blending in with the sandy and rocky terrain. Its tentacles are spread out around its body, and its eyes are closed. The octopus is unaware of a king crab that is crawling towards it from behind a rock, its claws raised and ready to attack. The crab is brown and spiny, with long legs and antennae. The scene is captured from a wide angle, showing the vastness and depth of the ocean. The water is clear and blue, with rays of sunlight filtering through. The shot is sharp and crisp, with a high dynamic range. The octopus and the crab are in focus, while the background is slightly blurred, creating a depth of field effect.

Sora, as demonstrated in the announcement and example videos, excels at producing intricate videos featuring various characters and movements, with detailed backgrounds. It seamlessly integrates multiple settings within a single video, automatically transitioning between scenes while maintaining consistency across characters and elements. Such capabilities underscore Sora’s deep understanding of written language, enabling it to comprehend both the intent behind instructions and their real-world implications.

Sora is not perfect, however, which OpenAI recognizes. For instance, it may struggle with accurately simulating physical laws and causality in complex scenes. An illustration of this is when a character bites into a cookie but the cookie lacks a bite mark. Spatial orientation remains imperfect, occasionally mixing up right and left sides. Additionally, the model may not consistently follow descriptions of events unfolding over time, such as expected camera movements.

The company opens its statement introducing its new model by emphasizing its commitment to aiding people in their work rather than replacing them, a concern voiced by critics of AI’s rapid advancement. In line with this, the company highlights its decision to offer Sora to artists, designers, and filmmakers ahead of its official release, seeking feedback to enhance its functionality and better serve their needs.

In addition to displacing artists and creative professionals, another prevalent concern is that simplifying and enhancing image and video generation could facilitate increasingly persuasive scams. Recently, a Hong Kong employee was deceived into transferring $25 million believing he was video chatting with his bosses, despite being the only real participant in the call. YouTube has also been struggling with deepfake impersonations (of MrBeast, for example) flooding its platform, which it hopes to tackle by introducing new AI-related rules. Furthermore, there’s the recent revelation that China is leveraging AI-generated images to impact the US presidential election campaign.

To allay these concerns, OpenAI is engaging experts to assess potential misuse risks before Sora’s release. They’re investigating issues like pseudo-news, hate speech, and biases inherited from training data, such as gender or racial bias.

Moreover, they aim to deter or complicate misuse of their technology. They’re developing tools to detect videos created with Sora and will integrate an open standard called C2PA, providing metadata indicating AI generation. However, this metadata’s utility is limited; the company acknowledges it can be removed, even unintentionally, such as when uploaded to social platforms and subsequently deleted.

“Despite extensive research and testing, we cannot predict all of the beneficial ways people will use our technology, nor all the ways people will abuse it,” the company closes its statement.

Sora is still in development and not publicly available yet and no specific release date has been announced.