Abdelrahman (Ogail) Elogeel

Microsoft Just Made 100B Parameter LLMs Run on Your Laptop. Who Governs Them? BitNet breaks the GPU barrier. 1.58 bit quantization. 100 billion parameters on a single CPU. No cloud. No API costs. Local inference at human reading speed. This is a genuine breakthrough for accessibility. But accessibility without governance is liability. Running locally solves the infrastructure problem. It does not solve the governance problem. A 100B parameter model on your laptop can still hallucinate with confidence. It can still produce biased outputs. It can still make decisions with no audit trail. It can still violate constitutional bounds you never defined. Privacy from cloud providers is not the same as privacy from the model itself. Local execution means your data stays on your device. It does not mean the model operates within governance constraints. The question is not where the model runs. The question is whether the model can prove it operated within bounds. YIN-GATEWAY solves this. YIN-GATEWAY is a cryptographic control plane that sits between any LLM and its users. Cloud model, local model, quantized model, full precision model. The deployment changes. The governance requirement does not. Layer 2 Admissibility Gates reject non-compliant queries before execution. The model never sees requests that violate policy. Layer 3 Constitutional Bounds verify every output before it reaches users. If the output would violate governance constraints, it does not proceed. Layer 14 SENTINEL Verification runs independent binary checkpoint validation. Pass or fail. No partial compliance. Layer 15 Human Override requires cryptographic signatures before high-risk outputs proceed. Not a checkbox. A mathematical proof that a human authorized this specific action. BitNet gives you a 100B parameter model on your CPU. YIN-GATEWAY gives you cryptographic proof that the model operated within constitutional bounds, regardless of where it runs. The hardware revolution is here. The governance layer was already built. https://lnkd.in/eb2EdEwR 32 USPTO applications. 3,057 claims. Any model. One governance layer. Enforced behavior. #BitNet #YINMazariArchitecture #YINGATEWAY #AIGovernance #CryptographicCompliance #Microsoft #LocalAI #LLM #PrivacyPreserving #ResponsibleAI #EdgeAI #CPUInference #OpenSource

4

1 Comment

You are in a system design interview at Meta for an IC4 role. The interviewer leans in and asks: "If a photo is already on my phone, how can WhatsApp still delete it when the sender taps Delete for Everyone?" Here is how you should break it down 👇 A lot of people think “Delete for Everyone” means WhatsApp somehow reaches into your gallery and erases a file from your device. That is not really what is happening. Btw, if you’re preparing for system design/coding interviews, check out my free mock interview tool on Layrs. You can use it for free here: https://lnkd.in/gpCn7t2T The real answer depends on where the photo is stored, how the chat app references it, and what exactly the app is allowed to delete. 1. First clarify the product behavior Before jumping into architecture, say this clearly in the interview: There are two different cases here. a) The photo exists only inside the chat app’s managed storage b) The photo has already been saved to the user’s gallery / camera roll / filesystem This distinction is everything. If the photo is only inside WhatsApp’s controlled storage, the app can remove the local file and remove the message reference from the chat UI. If the photo was exported or auto-saved to the gallery, WhatsApp usually cannot reliably delete that copy, because that file is now outside the app’s normal ownership boundary. So “Delete for Everyone” is mostly about: - deleting the message record - deleting the media attachment reference - deleting any app-managed cached copy - syncing that deletion event to all participants It is not magic remote file deletion across the whole phone.  2. High-level idea The clean way to think about this is: A chat message is metadata plus optional media. For a photo message, the system usually has: - message_id - chat_id - sender_id - media_id - timestamp - status - optional encryption metadata And the actual photo itself may live in: - encrypted object storage on server for temporary delivery - app sandbox / local database / media cache on device - optionally the user’s gallery if exported When sender taps Delete for Everyone, the system does not chase every byte everywhere. It sends a deletion command tied to the message_id.  3. What happens when the photo is first sent Here is the send flow: 1. User picks a photo 2. App encrypts media and prepares message metadata 3. App uploads encrypted media blob or sends it through media pipeline 4. App sends message metadata referencing that media 5. Receiver downloads the encrypted media 6. Receiver stores it locally in app-managed storage 7. Chat UI shows the message by resolving message_id -> media_id -> local file Read rest of the post: https://lnkd.in/gpv-wGze

158

12 Comments

Indeed. A small number of big tech companies have grossly overinvested in LLMs and overextended themselves -- even laying off tens of thousands of critically needed human employees and enshittifying their primary products and services to reallocate investment into it -- shooting Nvidia's profits and shares to the moon -- all in an insane attempt to "be first" for something that has no forseeable compelling ROI. It's a collective delusion and hype bubble. This is markedly different from the strategic "operate in the red to build marketshare" strategy of the few survivors of the dot-com bust, because at least those survivors were able to articulate (and realize) a logical ROI after reaching a tipping point of critical mass. This is more like the 98% of dot-com startups that burned bright fueled by VC right until they burned themselves out because they never actually had any believable plan for monetizing anything they were building. The lack of ROI will come rudely crashing down on these companies over the next several quarters as they run out of tricks to cook the books and hide the losses. Shareholders will get spooked and back out, share prices will crash, and big tech companies that have bet the farm on LLMs will suddenly find themselves without the necessary employees, capital, quality products/services, or reputational sway needed to save themselves. And when they collapse, Nvidia will collapse with them. And then maybe, finally, we can get back to the business of making things of actual value to humans.

19

I've been discussing this recently with Frances Buontempo in the context of coding assistants. We hear a lot from developers saying that LLM assistants in their fave IDEs are great for generating a lot of the (boring) boilerplate code needed to stand up a non-trivial system. However, most popular IDEs have been able to do exactly this for decades--whether with "wizards", code-block and project templates, or other various forms of meta-programming. The difference is that the "old" way is much less hungry of resources, while simultaneously being much more predictable and reliable at automatically generating the "boring" stuff. Coding assistants are also good at other things, of course, but many of the popular (in my conversations with developers) tasks are those I would much prefer to do myself, such as writing tests.

15

6 Comments

The paper “Challenges and Research Directions for LLM Inference Hardware” by Patterson and Ma frames what the industry is already talking about - inference is becoming constrained less by compute and more by memory bandwidth and capacity. As model sizes push toward 10 trillion parameters, an HBM-only strategy becomes economically and physically untenable. The authors argue for a tiered memory hierarchy and position High Bandwidth Flash (HBF) as a capacity-oriented extension of the memory system rather than as traditional storage. HBF borrows the 3D-stacking model from HBM but applies it to NAND flash. Instead of a discrete SSD controller connected to flash over a narrow interface, HBF integrates logic and NAND dies into a single vertical stack. A logic base die - similar in role to HBM’s base layer - implements the PHY, multi-channel flash controllers, and ECC. Vertical TSVs run through the stack, enabling thousands of parallel data paths and aggregate read bandwidths of ~1.6 TB/s per stack. A single HBF stack can provide ~512 GB of capacity, compared to 48–64 GB per HBM4 stack. The interface uses wide, synchronous, DDR-style signaling to the accelerator - similar to HBM. There are clear trade-offs, though. HBF operates at microsecond-scale latencies. Accesses are page-based (kilobytes), and small random reads quickly degrade effective bandwidth. Write endurance is limited (on the order of 10⁵ cycles), making HBF unsuitable as a general-purpose cache or a write-heavy tier. For this reason, HBM continues to remain critical for KV cache, activations, and other latency-sensitive accesses. Some systems are tiering KV cache - keeping hot KV in HBM while spilling colder data into CPU-attached LPDDR over coherent high-speed interconnects. HBF fits naturally with read-mostly data: model weights and slowly changing retrieval corpora. Weights are large, read-bandwidth-hungry, and not written during inference. Increasing per-node weight capacity by an order of magnitude reduces the number of accelerators required to fit a model, directly impacting system scale, power, and network complexity. HBF has shifted from theory to a formal industry roadmap following the 2025 SanDisk and SK Hynix alliance, with standardization underway and sampling slated for late 2026, targeting a full market debut on inference platforms by early 2027. Kioxia is pursuing an incremental path, moving multi-terabyte flash closer to accelerators over PCIe 6.0 with aggressive prefetching. Different implementations, same direction: flash is being pulled into the memory hierarchy! HBF is a serious attempt to address the capacity bottleneck. The open questions are no longer about whether this makes sense, but how to architect systems around it - software scheduling of page-based reads to hide microsecond latencies, balancing between HBM/HBF for different inference workloads, reducing tail latency, etc. Overall, I am excited to see more innovation for the memory wall problem!

156

6 Comments

🔎 Recent Observations in Embedding Modeling (Gemini & Qwen3) The frontier in embedding modeling is increasingly shaped by large LLMs like Google Gemini (https://lnkd.in/dp8ygAuy) and Alibaba Qwen3 (https://lnkd.in/dREPHW8m). Both teams are converging on a similar recipe for building state-of-the-art embeddings for retrieval, ranking, and search. 🎯 Common Patterns Emerging: • Leverage LLM Backbones: Both approaches start from large, pretrained LLMs (decoder-only, causal architectures) rather than building encoder models from scratch. • Synthetic Data Generation: Synthetic positive and hard negative pairs are created in bulk—using the LLM itself—to augment or even replace scarce labeled data. • Contrastive Fine-Tuning: Embedding quality is improved by fine-tuning with contrastive objectives, with in-batch / hard negatives and multi-stage training. • Task Conditioning: Simple instruction prompts (like "query:" or "passage:") help the model generalize across tasks without re-architecting. • Pooling over Causal Outputs: Instead of modifying the attention mask to be bidirectional, these models pool the outputs of a decoder-only, left-to-right model for sequence-level representations. 💡 As these practices become standard, a tough question remains for downstream adoption: how can we fine-tune LLM-based embedding models for a specific task without catastrophic forgetting, if we lack the diverse multi-task or general-domain data that major labs have access to? • Mixing in original language modeling or general tasks is a proven way to retain model generality, but it requires substantial extra data and resources. • In practice, many organizations have only narrow downstream data. This creates a real risk of overfitting or catastrophic forgetting when fine-tuning for new embedding tasks. Would love to hear thoughts, practical experiences, or solutions from those tackling this problem!

55

5 Comments

When I worked at Google, I was lucky to collaborate with some of the brightest machine-learning (ML) engineers. They worked on feature engineering. By picking the factors to guide the ML model, their advances could generate tens to hundreds of millions of additional revenue. Imagine an Excel spreadsheet with hundreds of columns of data. Add two columns, multiply two, divide by another, and subtract a fourth. Each of these is a feature. ML models used features to predict the best ad to show. It started as a craft, reflecting the vibes of the era. Over time, we’ve mechanized this art into a machine called AutoML that massively accelerates the discovery of the right features. Today, reinforcement learning (RL) is in the same place as feature engineering 15 years ago. What is RL? It’s a technique of teaching AI to accomplish goals. Consider a brave Roomba. It presses into a dirty room. Then it must make a cleaning plan and execute it. Creating the plan is step 1. To complete the plan, like any good worker, it will reward itself, not with a foosball break, but with some points. Its reward function might be: +0.1 for each new square foot cleaned, -5 for bumping into a wall, and +100 for returning to its dock with a full dustbin. The tireless vacuum’s behavior is shaped by this simple arithmetic. (NB : I’m simplifying quite a bit here.) Today, AI can create the plan, but isn’t yet able to develop the reward functions. People do this, much as we developed features 15 years ago. Will we see an AutoRL? Not for a while. The techniques for RL are still up for debate. Andrej Karpathy highlighted the debate in a recent podcast. This current wave of AI improvement could hinge on RL success. Today, it’s very much a craft. The potential to automate it—to a degree or fully—will transform the way we build agentic systems.

38

1 Comment

Why does an E3 level SWE at Meta make only ~190k/year, while an E8 level engineer makes over ~$2M/year, even though both engineers are ICs and spend the same time at work? I have spent the last 5 years at Meta as an IC. I joined with a little over 15 years of experience, and I’ve worked with many solid engineers in this time. Here is how I think about that compensation jump. 1. Same hours, completely different “unit of work” An E3’s unit of work is usually a task or a ticket. An E8’s unit of work is a multi year problem for the company. E3: “Implement this service, fix this bug, write this feature.” E8: “How do we cut infra cost by 20 percent across this product” or “How do we make this platform safe to scale to 10x users.” One person is paid to execute. The other is paid to decide what is worth executing in the first place. 2. Radius of impact E3 usually impacts a file, a service, maybe a small team. E8 shapes whole orgs and product lines. If an E3 ships something great, the impact is great but local. If an E8 ships the right platform, hundreds of engineers become faster and the company saves or earns millions every year. Comp tracks the area of the circle you influence. 3. Risk and downside protection At junior levels, mistakes are usually contained and reversible. At senior staff levels, a bad call can burn tens of millions or damage the brand. E8s are paid for judgment under ambiguity. They decide which bets the company should not make, which migrations can wait, which “shiny idea” is going to kill reliability. You pay more to people whose good judgment protects you from very expensive failures.  4. They scale themselves This can happen in a few ways. 1. Delegation with ownership They define the shape of the problem, then hand large pieces to other senior and mid level engineers while keeping the bar and direction clear. 2. Knowledge that travels They write RFCs, public comments, FAQs, wikis, internal posts. One answer helps hundreds of people who will face the same issue next quarter. 3. Tools over heroics Instead of unblocking people manually all day, they build tools, libraries or guardrails so others can unblock themselves. One well designed tool can save thousands of engineering hours every year. This is what “scaling yourself” actually looks like. The company pays for that multiplier.  5. Ownership of the “uncomfortable problems” Junior engineers usually work inside a well defined box. E8s take ownership between the boxes. They pick up problems that:  Span many teams and no one really “owns”  Require aligning leaders who disagree  Have product, infra, legal and security angles at the same time Most people avoid those because they are messy, political and slow. Very senior ICs lean into them. That is where a lot of value sits. Continued ↓

201

15 Comments

Imagine you just joined Google as an SWE. It’s your first week, and you’re overthinking everything. You’re surrounded by world-class engineers, new systems, and an overwhelming urge not to mess up. The pressure to prove yourself is real, and so is the anxiety…so what should you do? Over the years, I’ve moved across orgs, teams, and sometimes, entire companies. Here’s exactly what I did (and still do) to ramp up faster and enjoy the process without letting anxiety win → Don’t let yourself stay stuck. If you’re blocked for more than 30 minutes, ask for help. Share what you’ve tried, show initiative, and intent. → Read way more code than you’re assigned. Dive into related modules, historic PRs, and old design docs. The context you gain helps you spot patterns and avoid mistakes. → Ask about the story behind the system. Find out why things are built a certain way, what’s failed before, and what’s on the roadmap. Understanding the “why” beats memorizing the “what.” → Volunteer for unglamorous work. Take ownership of tests, docs, or nagging integration bugs. This is where you quietly learn the most and build your internal network. → Offer help before you’re an expert. Chime in when you see questions, review PRs, or share a script. Even small contributions get you noticed for the right reasons. → Connect with skip-levels and cross-team leads. Set up short intros to learn about the broader vision and team priorities. You’ll spot opportunities to add value beyond your immediate tasks. → Document what you learn as you go. Share notes, update READMEs, or post quick how-tos in team channels. Your future self and your teammates will thank you. → Shadow someone during incidents or launches. Sit in on debugging sessions or war rooms, even if you’re just observing. You’ll see how senior folks think, react, and communicate under pressure. → Keep a running list of “how did that break?” moments. Every time you see a system fail or a hotfix go live, dig in. Reverse-engineering past problems is the fastest way to understand the architecture. → Show up with curiosity and intent, every day. Don’t just tick off JIRA tickets. Look for the bigger picture. People notice when you genuinely want to learn and contribute, not just “do your job.” Some orgs assign mentors, some don’t. Either way, these habits put you in the driver’s seat. Ramping up is less about brilliance, more about consistent intent. And the ones who learn this early end up leading teams later.

709

51 Comments

Stock vesting schedules have been morphing, as we all know. And Waymo just slipped a new experiment into the mix. Two recent Waymo offers shared on Levels.fyi tell the story: - An L3 software-engineer package that vests RSUs over two years, 50 percent each year. - An L5 package that keeps the classic four-year, evenly split design. That contrast hints at a deliberate split strategy: shorter horizons for early-career hires, longer runways for senior talent. The logic tracks. New grads often treat their first role as a springboard, so a two-year vest ensures most of their grant actually pays out before they move on. For seasoned engineers, who hold bigger grants and deeper domain knowledge, Waymo still wants the retention pull of a four-year schedule. Waymo isn’t alone. Across tech, equity timelines are turning into a tuning knob: - Lyft issues single-year new-hire grants, then refreshes annually. - Uber front-loads at 35 / 30 / 20 / 15. - NVIDIA moved to 40 / 30 / 20 / 10. - Pinterest compresses to roughly 50 / 33 / 17 over three years. - A handful of high-growth startups grant two-year blocks by default. What’s driving the shift? 1. Pay-for-performance. Shorter cycles let companies re-price grants and reward top performers without being locked into four-year promises. 2. Dilution math. Fewer unvested shares lower equity overhang, especially valuable when stock prices spike. 3. Flexibility. Refreshers become the main retention lever, not year-three and year-four vest cliffs. 4. Talent liquidity. For early-career hires, faster access to equity can feel more tangible than a promise that’s years away. If you’re negotiating, remember that earlier liquidity doesn’t always mean bigger upside. Always compare total grant size, ask how refreshers are determined, and run the tax math on concentrated vesting events. Have you seen a two-year or front-loaded schedule in your offer recently? Comment below with your thoughts and I’ll send over a vesting report we put together here at Levels.fyi. #vestingschedule

137

8 Comments

just talked to a hardware startup in sf as a team of 2, that just raised $M for their seed round. they've hit: hundreds or pre-orders, millions of impressions on social, and 2nd iteration of product in less than 10 months. ⚡ speed is the most important thing for building. and we're excited to help anyone who wants to test their speed this summer in ai + hardware with the 30 worldwide hack we're hosting with Boost VC, @Flux, Studio 45, and more. we treat building like a sport at worldwide studios, and think there needs to be more open access sprints like this. more info below.

47

1 Comment

Today I spoke with two frinds from Google AI and learned that the KPI for engineers at Google and many large tech companies has fundamentally changed. A key metric now is how many tokens you are able to use. The more tokens you use, the greater your potential contribution—one person can now accomplish the work that once required ten or even a hundred people. #HerPowerAI #Google #WorkModel #KPI

1