Show HN: Serve 100 Large AI models on a single GPU with low impact to TTFT
Show HN (score: 5)Description
With this project you can hot-swap entire large models (32B) on demand.
Its great for:
Serverless AI Inference
Robotics
On Prem deployments
Local Agents
And Its open source.
Let me know if anyone wants to contribute :)
More from Show
Show HN: On the edge of Apple Silicon memory speeds
Show HN: On the edge of Apple Silicon memory speeds I have developed open source CLI-tool for Apple Silicon macOS. It measures memory speeds in different ways and also latency. It can achieve up to 96-97% efficiency on read speed on M4 base what is advertised as 120GB/s. All memory operations are in assembly.<p>I would really appreciate for results on different CPU's how benchmark works on those. I have been able to test this on M1 and M4.<p>command : 'memory_benchmark -non-cacheable -count 5 -output results.JSON' (close all applications before running)<p>This will generate JSON file where you find sections copy_gb_s, read_gb_s and write_gb_s statics.<p>Example M4 with 10 loops: "copy_gb_s": { "statistics": { "average": 106.65421233311835, "max": 106.70240696071005, "median": 106.65069297260811, "min": 106.6336774994254, "p90": 106.66606919223108, "p95": 106.68423807647056, "p99": 106.69877318386216, "stddev": 0.01930653530818627 }, "values": [ 106.70240696071005, 106.66203166240008, 106.64410802226159, 106.65831409449595, 106.64148106986977, 106.6482935780762, 106.63974821679058, 106.65896986001393, 106.6336774994254, 106.65309236714002 ] }, "read_gb_s": { "statistics": { "average": 115.83111228356601, "max": 116.11098114619033, "median": 115.84480882265643, "min": 115.56959026587722, "p90": 115.99667266786554, "p95": 116.05382690702793, "p99": 116.09955029835784, "stddev": 0.1768243167963439 }, "values": [ 115.79154681380165, 115.56959026587722, 115.60574235736468, 115.72112860271632, 115.72147129262802, 115.89807083151123, 115.95527337086908, 115.95334642887214, 115.98397172582945, 116.11098114619033 ] }, "write_gb_s": { "statistics": { "average": 65.55966046805113, "max": 65.59040040480241, "median": 65.55933583741347, "min": 65.50911885624045, "p90": 65.5840272860955, "p95": 65.58721384544896, "p99": 65.58976309293172, "stddev": 0.02388146120866979 },<p>Patterns benchmark also shows bit more of memory speeds. command: 'memory_benchmark -patterns -non-cacheable -count 5 -output patterns.JSON'<p>Example M4 from 100 loops: "sequential_forward": { "bandwidth": { "read_gb_s": { "statistics": { "average": 116.38363691482549, "max": 116.61212708384109, "median": 116.41264548721367, "min": 115.449510036971, "p90": 116.54143114134801, "p95": 116.57314206456576, "p99": 116.60095068065866, "stddev": 0.17026641589059727 } } } }<p>"strided_4096": { "bandwidth": { "read_gb_s": { "statistics": { "average": 26.460392735220456, "max": 27.7722419653915, "median": 26.457051473208285, "min": 25.519925729459107, "p90": 27.105171215736604, "p95": 27.190715938337473, "p99": 27.360449534513144, "stddev": 0.4730857335572576 } } } }<p>"random": { "bandwidth": { "read_gb_s": { "statistics": { "average": 26.71367836895143, "max": 26.966820487564327, "median": 26.69907406197067, "min": 26.49374804466308, "p90": 26.845236287807374, "p95": 26.882004355057887, "p99": 26.95742242818151, "stddev": 0.09600564296001704 } } } }<p>Thank you for reading :)
Show HN: Cachekit – High performance caching policies library in Rust
Show HN: Cachekit – High performance caching policies library in Rust
Show HN: AI video generator that outputs React instead of video files
Show HN: AI video generator that outputs React instead of video files Hey HN! This is Mayank from Outscal with a new update. Our website is now live. Quick context: we built a tool that generates animated videos from text scripts. The twist: instead of rendering pixels, it outputs React/TSX components that render as the video.<p>Try it: <a href="https://ai.outscal.com/" rel="nofollow">https://ai.outscal.com/</a> Sample video: <a href="https://outscal.com/v2/video/ai-constraints-m7p3_v1/12-01-26-18-47-41" rel="nofollow">https://outscal.com/v2/video/ai-constraints-m7p3_v1/12-01-26...</a><p>You pick a style (pencil sketch or neon), enter a script (up to 2000 chars), and it runs: scene direction → ElevenLabs audio → SVG assets → Scene Design → React components → deployed video.<p>What we learned building this:<p>We built the first version on Claude Code. Even with a human triggering commands, agents kept going off-script — they had file tools and would wander off reading random files, exploring tangents, producing inconsistent output.<p>The fix was counterintuitive: fewer tools, not more guardrails. We stripped each agent to only what it needed and pre-fed context instead of letting agents fetch it themselves.<p>Quality improved immediately.<p>We wouldn't launch the web version until this was solid. Moved to Claude Agent SDK, kept the same constraints, now fully automated.<p>Happy to discuss the agent architecture, why React-as-video, or anything else.
Show HN: SubTrack – A SaaS tracker for devs that finds unused tools
Show HN: SubTrack – A SaaS tracker for devs that finds unused tools Hi HN,<p>I built SubTrack to help teams find unused SaaS tools and cloud resources before they silently eat into budgets.<p>The motivation came from seeing how hard it is to answer simple questions: – Which SaaS tools are actually used? – Which cloud resources are idle? – What will our end-of-month spend look like?<p>SubTrack connects to tools like AWS, GitHub, Vercel, and others to surface unused resources and cost signals from one place. Recently I added multi-account support, currency localization, and optional AI-based insights to help interpret usage patterns.<p>This is an early-stage project and I’m actively iterating. I’d really appreciate feedback—especially from people managing cloud or SaaS sprawl.
No other tools from this source yet.