The agent framework for rendering videos on any theme or topic from text prompts or image lists, outputs up to three minutes long is now open-source. It also comes with high-level endpoints for generative image creation and editing, model-composable assistants, search, and recommendations.. Everything you need to build an enterprise media library or knowledge base that is automatically catalogued and instantly searchable.
Run the entire infrastructure on a VM or your local machine. Samsar runs in Docker and is fully customizable, with support for multiple generative API providers.
History
Almost three years ago, this journey began, as everything does.. at a hackathon. The first build was a generative image editor, which, in my opinion, remains better than any online meme creator available today.
From that humble proof of concept emerged the final framework: the world’s first model-composable text-to-video (T2V) and image-list-to-video agent, capable of creating state-of-the-art generative videos of up to three minutes from a single prompt.
Timeline
First Iteration
An image generator and meme creator, the first iteration let you import or generate images, segment them to replace specific parts, add funny text, shapes and .
Connected to OpenAI’s newly available API, it could generate DALL·E 2 images directly within the image editor studio. You could add text, segment elements, apply simple HTML animations, and export the results as SVGs.
Many memes were created and NFTs minted using this first iteration.
Second Iteration
A video is a sequence of images—typically 24 or 30 frames per second, though sometimes 16.
We already had an ImageSession data structure, so we began exploring a higher-level VideoSession structure. It would hold the starting image and a JSON definition of HTML animations. A separate pub-sub queue would execute the animation logic to generate each frame, while the final stage would render those frames into a video via FFmpeg. This became the foundation of the pipeline.
The second iteration rendered simple HTML canvas animations.
Version 2.1 did not yet have native LLM inference. You could provide your own narrative, and it would generate a theme and text-to-image assets before stitching the scenes together. However, you could already use an LLM to create a cohesive theme across multiple video renders, allowing you to produce connected “episodes.”
10 episode playlist on Poe's raven
The second sub-iteration, version 2.2, launched once LLMs became capable of generating both narratives and themes from simple text prompts or instructions. Here is an example from version 2.2. Although the current version is far more advanced, those early renders had a rustic charm of their own.
Initial Version 2 could not do text subtitle animations, and only basic HTML animations
Next iterations, just following the tech as it became available, added support for both portrait and landscape renders, more image models, including the newly available FLUX 1.1 family of models via Fal and animated text and subtitles.
Later iterations could do more advanced HTML animations, Subtitle / Text animations and more image model options
Third Iteration
Although video APIs had been available for a while, they weren't available on API until around late 2024, Fal launched the first open-source video APIs, and the Runway API became generally available. As additional APIs launched, new possibilities emerged- including combining generative video with HTML animation to create richer experiences.
Next iterations introduced additional capabilities as provider model adapters provided richer interfaces including first/last frame generation. Using RunwayML via Samsar Vidgenie, users could optionally select the starting and ending frames for each session, making it possible to create truly one-of-a-kind videos in styles never seen before—and perhaps never seen again.
1-shot T2V with start frame and end frame settings, all scenes are continuous.
Fourth iteration
By this point, we had a stable image-to-video pipeline capable of producing videos up to two minutes long. It could reliably lip-sync non-human characters and generate one-shot text-to-video outputs in multiple formats.
Although the realism was not yet as strong as in version 5, this iteration supported a wider range of models and could produce renders in many interesting styles.
Fifth Iteration: Final form
Bug Fixes and improvements from 3 years of insights
January this year, Model accuracy improved, video models could finally understand context from the photo and implement physics at a realistic level.
VEO3.1, Cosmos 3 Super, Happy Horse 1.1 all have incredible physics understanding and realism and models like NanoBanana Pro and GPT Image 2 can render complex technical charts and diagrams with high degree of accuracy. Furthermore improvements in inference models for both inference and vision
Here I showcase some demoes from the latest version, even within the latest version some renders are a bit dated, you will see noticeable difference in quality in renders over days .
Documentary/ Natural Geography style T2V
Go back in time to create historical documentaries or peek forward to see what the future might look like. Each inference model has it's own view of the world. Each image model renders it differently and each video model imagines it in its brush.
Or create generative documentaries from topics of interest. Here we asked the AI to create a generative documentary on king snakes in the wild.
Generative summary style text-to-video
1-shot generative summary of Gone with the wind movie. You can ask for specific themes like critical summary, montage, etc.
Ask for movie summary on any topic, prompt and get render video.
Generative video-game render
Create video game strategies or re-enactments of epic games from text prompts, see the exact prompt used for each T2V render in the description section of the video.
or Live-action based on video game. Here we imagine imperial age warfare with Japanese tech-tree in Age of Empires II brought to life via a live-action video game.
One-shot live-action based on video game theme
Alternate history style videos, 1-shot. What-if curiosities.
Think what if romans had steam-engine, or what if Victorians had discovered rocket science, or what if lake mega-chad had existed today?
Well now you don't have to wonder, just prompt any curiosity, choose your favorite model settings - and render !
Alternate history: What is romans had Steam engine
Or imagine what pre-historic landscapes, flora and fauna would look like had they existed in modern times .. Just prompt to render, there is nothing like it and there will be nothing like it.
What if lake Mega Chad had existed today
Architecture
The whole thing can run on a 32 GB Macbook pro without any issues.
Also when running locally without using any public media hosting, the system will automatically create short-lived public urls via cloudflare tunneling for image to video and video to video tasks. You can keep working in your local, without hosting it on a server.
For enterprise deployments, you can enable SMTP, Nginx reverse-proxy, built-in logger, AWS + Cloudfront configuration.
Two SoTA inference providers, 6 SoTA T2V models and 6 SoTA I2V models embeddings are available out-of-the-box, you can also extend to include your own models via creating compatible adapters or call your local models via Fal API compatible payload.
It is fully composable, so for example you can use your local GPU for singular tasks like Inference or Image to video while offloading the remainder of the pipeline to SoTA public model adapters.
Samsar is free to use, under MIT license. there is charge to use the product, you can configure 3rd party model APIs and hook it up to the product and get billed via their billing system. The product also supports deployed SamsarOne as a model API provider (Universal fallback which can be used in addition to custom adapters)
Generic Features
Generic features out-of-the-box whch require no generation credits or API keys. Just plug-and-play.
Best-in-class video editor
The Samsar studio video editor is free to use without adding any API keys. upload your own video trim video, speed up parts, add audio
Image editor
For single image edit tasks , there is a standalone image editor shipped with the bundle as well, although it works best with at-least one or more generative edit APIs enabled, you can do simple image edit tasks likwe crop resize image , add text overlay or html animation to image without spending any credits at all !!
Generative Features
Plug and play SoTA generative models via adapters and endpoints of your choice. Multiple adapter options are available for each model option, you only need a minimal subset of adapters to unlock all features of studio and 1-shot T2V and Image-List to video render.
Enable search and recommendations for existing product by integrating the APIs into your system, or build a search / recommendations library for your generated video artifacts.
Enterprise Features
A production ready app and POC with a polished UI doesn't look that different at first glance.
Built-in routine garbage collector
Generative media pipelines are a lot of fun, until you realize it choked the hard-drive.
Every day a CRON job runs within your docker container that cleans up intermediate frame generations for your agent sessions ensuring only the final deliverables are persisted and intermediate cache files don't fill up the hard-drive space.
Nginx reverse proxy with domain and SSL cert
Configure public IP or domain for your Samsar instance and serve a public API layer for your internal use-cases.
You can optionally serve
Invite team-members to your project to collaborate on your sessions and assign usage quota to team members for generative tasks.
SMTP/SES configuration
Configure SMTP for forgot password and render completion emails. Also after completing SMTP and reverse proxy setup unlocks a hidden feature, an easter-egg if you will . Collaborative generative sessions.
Admin user can if they setup SMTP,
Built-in Grafana logger for live debugging
For live logging and tracing, developers and those who wish to modify the system can view live logs for their system , also .
Also useful if you with to trace the pipeline to understand what the system is doing.
Built in collaborative-features, invite team-members to your project
Download, get publish metadata , titles and captions for your video right from within the Studio, publish into your gallery setup search and recommendations for your user. It is an all-in-one package to deliver full-stack media power-house.
Whether you wanted to just watch an anime , or start your own cooking channel or perhaps create marketing videos for your products.
Fully customizable to the machine settings
If you have a bigger machine, up the default number of threads for lightning fast renders.
Have an enterprise API key? Tune up the concurrency of your generative pipelines.
Every step of the way the product has been better than anything available on the market, our first iteration is probably SoTA still for simple slide-show videos, if using the latest models will give a much better experience, than the demoes from late 2024.
Git checkout the code
samsarone
Want to see what can be built with this?
We built a generative media library from demo publications, entirely built upon samsar-js stack.
Checkout the Gallery
