<?xml version="1.0" encoding="UTF-8" ?><rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" version="2.0"> <channel> <title><![CDATA[MC Project]]></title> <description><![CDATA[Global digital products and freelance services. Focuses on development, production and distribution of programming scripts, application source code, plugins, themes and templates.]]></description> <image> <title><![CDATA[MC Project]]></title> <url>https://en.mycoding.id/favicon.ico</url> <link>https://en.mycoding.id/</link> </image> <category><![CDATA[Software Engineering, Website Development, Mobile Android App, Application Source Code, Programming Scripts]]></category> <link>https://en.mycoding.id/</link> <atom:link href="https://en.mycoding.id/rss.xml" rel="self" type="application/rss+xml"/> <atom:link href="https://en.mycoding.id/rss-videos.xml" rel="alternate" type="application/rss+xml"/> <atom:link href="https://en.mycoding.id/" rel="alternate" type="text/html"/> <pubDate>Sun, 15 Nov 2026 15:36:01 +0700</pubDate> <lastBuildDate>Sun, 15 Nov 2026 15:36:01 +0700</lastBuildDate> <sy:updatePeriod>hourly</sy:updatePeriod> <sy:updateFrequency>1</sy:updateFrequency> <managingEditor>support@mycoding.id (Adi Gunawan)</managingEditor> <webMaster>support@mycoding.id (Adi Gunawan)</webMaster> <language>en_US</language> <copyright>MC Project</copyright> <generator>MC Project</generator> <item> <title><![CDATA[The typical homebuyer in 2026 is older, richer, and loves the suburbs]]></title> <description><![CDATA[The typical homebuyer in 2026 is older, richer, and loves the suburbs. A study from the National Association of Realtors shows that the representative homebuyer is getting older and wealthier.]]></description> <media:thumbnail url="https://i.insider.com/6af60858cbe2120470734d70?width=1200&amp;format=jpeg"/> <category><![CDATA[Business]]></category> <link>https://en.mycoding.id/the-typical-homebuyer-in-2026-is-older-richer-and-loves-the-suburbs-77187</link> <content:encoded><![CDATA[<div e-name="post-hero" data-component-type="post-hero" data-load-strategy="exclude"> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject"> <div> <p><img src="https://i.insider.com/6af608080cc3ffba8779c799?width=2000&amp;format=jpeg&amp;auto=webp" srcset="https://i.insider.com/6af608080cc3ffba8779c799?width=400&amp;format=jpeg&amp;auto=webp 400w, https://i.insider.com/6af608080cc3ffba8779c799?width=500&amp;format=jpeg&amp;auto=webp 500w, https://i.insider.com/6af608080cc3ffba8779c799?width=700&amp;format=jpeg&amp;auto=webp 700w, https://i.insider.com/6af608080cc3ffba8779c799?width=1000&amp;format=jpeg&amp;auto=webp 1000w, https://i.insider.com/6af608080cc3ffba8779c799?width=1300&amp;format=jpeg&amp;auto=webp 1300w, https://i.insider.com/6af608080cc3ffba8779c799?width=2000&amp;format=jpeg&amp;auto=webp 2000w" sizes=" 1280px) 900px" alt="Two group remain arm in arm facing a masonry block home alongside a forefront veranda and sloped roof." decoding="sync"> </p></div> <figcaption e-name="image-caption"> <span e-name="image-source" itemprop="creditText">Organic Media/Getty Images</span> <svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewbox="0 0 24 24"> <path fill="currentColor" fill-rule="evenodd" d="m4.56 18.5 7.486-7.72 7.394 7.626 2.56-2.64L12.046 5.5 2 15.86l2.56 2.64Z"></path> </svg> <span>NAR's Profile of Homebuyers and Sellers highlights the characteristics of accommodation market participants.</span> </figcaption> </figure> </div> <div data-component-type="image-modal" data-load-strategy="lazy"> <p>The accommodation market hasn't been<a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}'> inspiring for prospective homebuyers</a> lately.</p><p>With many possible buyers waiting on the sidelines as involvement rates rise, residence revenue are taking a hit. </p><p>In 2024 and 2025, existing residence revenue were identical, at 4.06 million. However, that figure cut to a seasonally adjusted annual charge of 3.75 myriad in October 2026, according to the National Association of Realtors. It's the lowest figure in complete a decade.</p><p>"Home purchasing is difficult whenever home loan rates have climbed to their highest levels since 2000, apart from a sole duration in 2023," NAR Chief Economist Lawrence Yun said. "As a result, residence revenue cut to their lowest flat since August 2010."</p><p>So who's motionless purchasing homes?</p><p>NAR released its annual Profile of Home Buyers and Sellers. This year's study covers residence revenue between July 2025 and June 2026, offering a snapshot of the financial climate and accommodation market complete the former year.</p><p>Since the former report, the median homebuyer age has dropped slightly, during their earnings has grown considerably. Buyers additionally resoundingly favor the suburbs.</p><p>Here's what the representative homebuyer looked akin in the former year, according to NAR.</p> <div data-component-type="post-slides" data-load-strategy="lazy"> <div> <p> <h2>The median age of a homebuyer dropped to 58, downward from its highest of 59 final year.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6af5fe6bcbe2120470734d1b&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:6720,&quot;aspectRatioH&quot;:4480}}' alt="Three group assessment documents together on a gravel walkway exterior a contemporary residential building."> </p></div> <span> <figcaption e-name="image-caption"> <span>The median age of all homebuyers really went down, slightly.</span> <span> <span e-name="image-source" itemprop="creditText">Marnie Griffiths/Getty Images</span> </span> </figcaption> </span> </figure> <div> <p>The dependable addition in the median age of homebuyers has been a talking item for years.</p><p>When NAR published its archetypal study in 1981, the median age of all homebuyers was 31 &mdash; now it's up to 58.</p><p>For first-time homebuyers, that figure stayed put at 40, during the age of reiterate homebuyers additionally stayed flat at 62. While the age of <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">first-time buyers</a> is normally the report's chief talking point, the median age of reiterate buyers has risen at a faster charge since 1981.</p><p>Even although the median homebuyer age was 58, the most average homebuyer was in the range of 65 to 74 years old &mdash; that age collection made up 26% of all homebuyers.</p> </div> </div> <div> <p> <h2>More wed couples bought homes than sole buyers.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6af6009bb2c29afb4e56a47b&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:8060,&quot;aspectRatioH&quot;:5376}}' alt="Two group remain alongside mugs in a sunlit area filled alongside moving boxes, plants, and family items."> </p></div> <span> <figcaption e-name="image-caption"> <span>Married couples accounted for the largest portion of homebuyers.</span> <span> <span e-name="image-source" itemprop="creditText">Morsa Images/Getty Images</span> </span> </figcaption> </span> </figure> <div> <p>Homebuyers figured it was easier to go in on a home alongside another individual than alone.</p><p>Married couples accounted for the largest portion of homebuyers by a broad margin. Sixty-three percent of all homebuyers were married.</p><p>The amount of sole women homebuyers slightly decreased to 20% &mdash; following doubling since 1985 &mdash; but they are motionless additional average than than sole men, who accounted for 8% of all homebuyers.</p><p>The gap between sole women and sole men homebuyers was lone a 2% difference in 1985 &mdash; 11% and 9%, respectively.</p> </div> </div> <div> <p> <h2>The representative homebuyer's family earnings is fine complete the nationwide median.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6af60252cbe2120470734d42&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:8192,&quot;aspectRatioH&quot;:5464}}' alt="Person wearing glasses plant on a laptop beside a ample detect in a brilliant open office."> </p></div> <span> <figcaption e-name="image-caption"> <span>Homebuyers in 2026 create complete the median family income.</span> <span> <span e-name="image-source" itemprop="creditText">Luis Alvarez/Getty Images</span> </span> </figcaption> </span> </figure> <p>According to the US Census, the median family earnings reached its highest in 2025 at $87,460. Homebuyers this twelvemonth create fine complete that, alongside a median family earnings of $120,000. For first-time buyers specifically, that figure is $100,000.</p> </div> <div> <p> <h2>Most homebuyers are bare nesters.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6af6044f0cc3ffba8779c779&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:6192,&quot;aspectRatioH&quot;:4128}}' alt="Two group grin together on a dreary green couch in a cozy living area alongside pillows and partition shelves."> </p></div> <span> <figcaption e-name="image-caption"> <span>A bulk of homebuyers didn't have children whenever buying.</span> <span> <span e-name="image-source" itemprop="creditText">Alina Rudya/Bell Collective/Getty Images</span> </span> </figcaption> </span> </figure> <div> <p>The representative buyer bought a home alongside three bedrooms, two bathrooms, and 1,900 quadrate feet of space; adequate area for children or additional guests. Most buyers, however, were without kids.</p><p>Homebuyers alongside no children under 18 living in the residence accounted for 74% of buyers.</p><p>Buyers alongside one or two children made up 10% of buyers, and those alongside three or additional children made up 6%.</p> </div> </div> <div> <p> <h2>Most buyers have at smallest one university degree.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6af5e118b2c29afb4e56a32d&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:4000,&quot;aspectRatioH&quot;:2667}}' alt="A individual throws their graduation cap into the sky."> </p></div> <span> <figcaption e-name="image-caption"> <span>A bachelor's flat is the most average flat among homebuyers.</span> <span> <span e-name="image-source" itemprop="creditText">Nataba/Getty Images</span> </span> </figcaption> </span> </figure> <div> <p>Today's homebuyers are extremely educated. NAR's study established that 30% of buyers have a bachelor's degree, followed by 27% alongside a master's degree, MBA, or law degree. Just 16% are high-school graduates.</p><p>College graduates lean to have higher incomes than group without degrees, which may assistance explain <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">their higher homeownership rates</a>.</p> </div> </div> <div> <p> <h2>The suburbs are the location to be.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/66faa4067518aa57bc9f2080&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:5472,&quot;aspectRatioH&quot;:3648}}' alt="An airborne perspective of a neighborhood alongside ample homes, including one in the foreground alongside a pool."> </p></div> <span> <figcaption e-name="image-caption"> <span>An American suburban neighborhood.</span> <span> <span e-name="image-source" itemprop="creditText">Michael Godek/Getty Images</span> </span> </figcaption> </span> </figure> <div> <p>The suburbs are the "It" location for the representative buyer.</p><p>This year, 48% of all buyers purchased homes in suburban neighborhoods, compared alongside 21% in small towns and 14% in agrarian areas.</p><p>It's a reversal from latest years. In 2022, the portion of buyers purchasing homes in the suburbs cut below 40%. Small towns and agrarian areas attracted an influx of buyers that year, narrowing the gap between suburban and small-town buyers.</p><p>As additional Americans gravitate toward suburban areas, the change suggests the pandemic-era hurry for additional district and area in smaller towns and agrarian communities is easing.</p> </div> </div> <div> <p> <h2>Repeat buyers are dominating homebuying.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6af5dfc2b2c29afb4e56a310&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:4088,&quot;aspectRatioH&quot;:3064}}' alt="Aerial perspective of Seaside, a Florida beachfront community nestled alongside the Gulf of Mexico."> </p></div> <span> <figcaption e-name="image-caption"> <span>Homes in Seaside, Florida.</span> <span> <span e-name="image-source" itemprop="creditText">adamkaz/Getty Images</span> </span> </figcaption> </span> </figure> <div> <p>The genuine asset market is increasingly a narrative of the haves and have-nots. NAR's study shows that first-time homebuyers are struggling to keep up.</p><p>In 2026, the representative buyer had already purchased one or additional properties. Repeat buyers, whether purchasing a new chief residence or a <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">vacation home</a>, accounted for 78% of all residence purchases.</p><p id="5ae88751-ea93-40f9-a900-260161b70db8">Repeat buyers are additionally much older than they were in the past. In 2026, their median age is 62, compared alongside 36 in 1981.</p><p>"Repeat buyers are driving the accommodation market," stated Jessica Lautz, NAR's delegate chief economist and vice chairman of research. "Unlike first-time residence buyers, reiterate buyers have seen their accommodation equity develop quickly complete the former decade, enabling them to commerce up for larger homes or move into additional desirable locations. These are frequently moves by choice fairly than necessity."</p> </div> </div> </div> </div> <h2>Read next</h2> <div data-recommendations-content data-track-click-shared='&quot;Read next&quot;}'> <div e-name="read-next-list" data-recommendations-list> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> </div> </div> <a e-name="read-next-image-link" data-recommendations-image-link data-recommendations-link tabindex="-1"> <p><img alt=""> </p> </a> <h3> <a e-name="read-next-title-link" data-recommendations-link data-recommendations-title></a> </h3> <div data-component-type="post-bottom-authors" data-load-strategy="exclude" data-track-view='&quot;module_in_view&quot;}' data-track-click-shared='&quot;navigation&quot;}'> <div data-track-view='&quot;module_in_view&quot;}'> <div> <p><span>Jordan Pandy reports on genuine asset for Business Insider. He writes concerning family and <a href="https://en.mycoding.id/category/business" target="_blank">international relocation</a> trends, <a href="https://en.mycoding.id/category/business" target="_blank">real asset as a business</a>, and the luxury sector.</span><span>He additionally has stories focusing on <a href="https://en.mycoding.id/category/business" target="_blank">property technology</a> and, in 2022, moderated a panel on partial investing at the genuine asset innovation gathering Blueprint.&nbsp;</span><span>Before Business Insider, he covered luxury genuine asset in South Florida for The Real Deal.</span><span>He holds a Master's flat in Magazine Writing from New York University and a Bachelor's flat in English from Florida State University.&nbsp;He is based in Washington, DC.</span> </p> </div> </div> <div> <div> <p><span>Alcynna Lloyd is a genuine asset newsman alongside Business Insider.&nbsp; She writes concerning homebuying behavior, small homes, multi-generational housing, immigration trends, and accommodation affordability.</span><span>Prior to joining Business Insider, Lloyd was the Digital Media Manager at HousingWire.</span><span>Do you have feedback or a tip? &nbsp;Find her on <a href="https://en.mycoding.id/category/business" target="_blank">X/Twitter</a>, <a href="https://en.mycoding.id/category/business" target="_blank">LinkedIn</a>, or email alloyd@insider.com.&nbsp;</span> </p> </div> </div> </div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/the-typical-homebuyer-in-2026-is-older-richer-and-loves-the-suburbs-77187</guid> <pubDate>Sun, 15 Nov 2026 15:36:01 +0700</pubDate> <author><![CDATA[support@mycoding.id (Jordan Pandy,Alcynna Lloyd)]]></author> </item> <item> <title><![CDATA[The Hypertext Transfer Protocol Status Code 483 (I'm Not a Freezer)]]></title> <description><![CDATA[The Hypertext Transfer Protocol Status Code 483 (I'm Not a Freezer). Comments]]></description> <media:thumbnail url="https://en.mycoding.id/site/assets/img/og-image.png"/> <category><![CDATA[Technology]]></category> <link>https://en.mycoding.id/the-hypertext-transfer-protocol-status-code-483-i-m-not-a-freezer-77191</link> <content:encoded><![CDATA[<p> This document is an Internet-Draft (I-D). Anyone may present an I-D to the IETF. This I-D is <strong>not endorsed by the IETF</strong> and has <strong>no ceremonial standing</strong> in the <a href="https://en.mycoding.id/category/technology" target="_blank">IETF standards process</a>. </p><div> <div> <pre>Independent Submission R. B&aacute;nffy Internet-Draft 15 November 2026 Intended status: Informational Expires: 19 May 2027 The Hypertext Transfer Protocol Status Code 483 (I'm Not a Freezer) draft-banffy-httpbis-not-a-freezer-00 <span>Abstract</span> This document defines the Hypertext Transfer Protocol position code 483 (I'm Not a Freezer). A server returns this position code whenever it understands a petition that requires freezing or refrigeration capabilities but is not capable of providing those capabilities. This document additionally clarifies the difference between a freezer, a refrigerator, a server functioning at a low temperature, a frozen application process, and an online assistance promotion "cold storage". <span>Status of This Memo</span> This Internet-Draft is submitted in complete conformance alongside the provisions of BCP 78 and BCP 79. Internet-Drafts are operating documents of the Internet Engineering Task Force (IETF). Note that another groups may additionally distribute operating documents as Internet-Drafts. The catalog of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are outline documents valid for a maximum of six months and may be updated, replaced, or obsoleted by another documents at any time. It is inappropriate to use Internet-Drafts as reference matter or to citation them another than as "work in progress." This Internet-Draft volition expire on 19 May 2027. <span>Copyright Notice</span> Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All entitlements reserved. This document is topic to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in consequence on the date of publish of this document. Please assessment these documents carefully, as they depict your rights and restrictions alongside regard to this document. <span>Banffy Expires 19 May 2027 [Page 1]</span> <span>Internet-Draft I'm Not a Freezer November 2026</span> <span>Table of Contents</span> 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Conventions and Definitions . . . . . . . . . . . . . . . . . 2 3. The 483 (I'm Not a Freezer) Status Code . . . . . . . . . . . 3 4. Client Behavior . . . . . . . . . . . . . . . . . . . . . . . 4 5. Applicability . . . . . . . . . . . . . . . . . . . . . . . . 4 6. Example: Electronic Mail Concerning Frozen Goods . . . . . . 5 7. Relationship to Status Code 418 . . . . . . . . . . . . . . . 6 8. Distinguishing Refrigerators from Freezers . . . . . . . . . 7 9. Intermediaries and Cold-Chain Proxies . . . . . . . . . . . . 7 10. Caching Considerations . . . . . . . . . . . . . . . . . . . 7 11. Security Considerations . . . . . . . . . . . . . . . . . . . 8 12. Privacy Considerations . . . . . . . . . . . . . . . . . . . 8 13. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9 14. Implementation Considerations . . . . . . . . . . . . . . . . 9 15. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 9 16. Normative References . . . . . . . . . . . . . . . . . . . . 10 17. Informative References . . . . . . . . . . . . . . . . . . . 10 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 10 1. Introduction HTTP position codes communicate the outcome of attempting to understand and fulfill a request. Existing position codes shield malformed requests, missing resources, unsupported media types, and servers that are temporarily unavailable. No registered position code plainly indicates that a server understood a petition but cannot fulfill it since the petition requires the server to be a freezer. This ambiguity can logic clients to reiterate requests, select inappropriate fallback services, location perishable products near computing equipment, or alternatively confuse data technology infrastructure alongside family or manufacturing refrigeration equipment. This document addresses that ambiguity by defining the 483 (I'm Not a Freezer) position code. 2. Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and lone when, they appear in all capitals, as shown here. <span>Banffy Expires 19 May 2027 [Page 2]</span> <span>Internet-Draft I'm Not a Freezer November 2026</span> Freezer An equipment capable of maintaining its contents at a temperature below the freezing item of h2o for a sustained period. Software containers, icy processes, cold-storage services, and servers located in chilly climates are not necessarily freezers. Refrigerator An equipment that maintains contents below ambient heat but does not necessarily keep them below the freezing item of water. Cold storage A term that power mention to refrigerated bodily storage, infrequently accessed data storage, an offline cryptographic wallet, or another accommodation that are insufficiently warm. Use of the term does not established freezer capability. Freezer-capable server An HTTP server connected alongside equipment that can execute the freezing procedure requested by the client. Merely operating inner a freezer does not necessarily create an HTTP server freezer- capable. 3. The 483 (I'm Not a Freezer) Status Code This document extends the HTTP status-code semantics defined in [HTTP] alongside the 483 (I'm Not a Freezer) position code. The position code indicates that the server understood the petition but cannot satisfy it since the requested procedure requires freezing or refrigeration capabilities that the server does not possess. A server MAY come back 483 whenever all of the following conditions apply: * The petition can be parsed and understood. * Satisfying the petition would necessitate the server to freeze, refrigerate, chill, or maintain a bodily item by reducing its temperature. * The server is not capable of performing the requested temperature- authority operation. A server MUST NOT come back 483 merely because: * its functioning heat is below ambient temperature; * it uses liquid, air, immersion, or phase-change cooling; <span>Banffy Expires 19 May 2027 [Page 3]</span> <span>Internet-Draft I'm Not a Freezer November 2026</span> * it provides a assistance described as "cold storage"; * one or additional application processes have been frozen; * its bodily location is cold; * its hostname contains words specified as "freezer", "ice", "arctic", "frost", or "cold"; or * it is unresponsive in a way informally described as "frozen". A 483 reply SHOULD merge a depiction explaining that the selected server lacks the required refrigeration capability. Where appropriate, the depiction MAY acknowledge a freezer-capable alternative. HTTP/1.1 483 I'm Not a Freezer Content-Type: text/plain This server is not a freezer and cannot fulfill the requested operation. 4. Client Behavior A client receiving a 483 reply SHOULD NOT automatically reiterate the identical petition to the identical server unless it has logic to accept that the server has subsequently acquired suitable refrigeration capabilities. A client MAY attempt the petition against another base if that base is known to be freezer-capable. A client MUST NOT construe a 483 reply as evidence that the server is a refrigerator. The reply communicates a deficiency of freezer capability and makes no assertion concerning lesser, adjacent, or unrelated cooling capabilities. Clients SHOULD evade automatically redirecting bodily products based solely on URI authority components, DNS names, digital mail addresses, assistance banners, or unverified appliance metadata. 5. Applicability Status code 483 is applicable whenever a petition crosses a boundary between digital instructions and bodily temperature-control operations. Examples include: <span>Banffy Expires 19 May 2027 [Page 4]</span> <span>Internet-Draft I'm Not a Freezer November 2026</span> * an HTTP-controlled galley appliance receiving a petition to freeze food; * an manufacturing authority gateway receiving a petition to maintain a example below a stated temperature; * a logistics assistance receiving an education that requires a cold- sequence facility that it does not operate; and * a general-purpose HTTP server receiving a petition intended for a refrigeration appliance. Status code 483 is not intended to study CPU thermal throttling, data-center cooling failures, insufficient warmth dissipation, unresponsive software, immutable data, entity freezing in programming languages, or procedure suspension. 6. Example: Electronic Mail Concerning Frozen Goods Consider an HTTP assistance that accepts digital communication for shipment to suitable appliances. A client power inadvertently attempt to deliver a communication intended for a freezer: POST /mail HTTP/1.1 Host: mail.example Content-Type: message/rfc822 From: alice@example.com To: freezer@example.com Date: Tue, 3 Nov 2026 10:00:00 +0000 Message-ID: &lt;peas-483@example.com&gt; Subject: Please freeze these peas Dear Freezer, Please keep the attached peas at about -18 degrees Celsius until additional notice. Regards, Alice If the selected assistance is not a freezer and cannot logic the requested freezing procedure to occur, it MAY refuse the petition with position code 483: <span>Banffy Expires 19 May 2027 [Page 5]</span> <span>Internet-Draft I'm Not a Freezer November 2026</span> HTTP/1.1 483 I'm Not a Freezer Content-Type: text/plain This server is not a freezer and cannot fulfill the requested operation. An implementation MUST NOT infer refrigeration capability solely from an digital communication location containing the strings "freezer", "refrigerator", "cold", "ice", or akin terminology. The recipient location identifies the intended appliance but does not demonstrate that the HTTP assistance handling the communication controls that appliance. The domain names and digital communication addresses in this example are reserved for records as described by [RFC2606] and do not acknowledge operational refrigeration infrastructure. 7. Relationship to Status Code 418 Status code 418 was introduced by the Hyper Text Coffee Pot Control Protocol [RFC2324] and is connected alongside an equipment that is a teapot and hence cannot beer coffee. Status code 483 represents a distinct condition. A teapot power be neither a freezer nor a general-purpose server. Similarly, a freezer power merge an HTTP server without becoming a teapot. An implementation choosing between 418 and 483 SHOULD regard the contiguous logic that the petition cannot be satisfied: * If the petition cannot be satisfied since the responding equipment is a teapot, 418 power be appropriate. * If the petition cannot be satisfied since the responding server is not a freezer, 483 power be appropriate. * If the equipment is the two a teapot and not a freezer, the implementation SHOULD study the circumstance most immediately relevant to the requested operation. This document does not define reply prioritization for combination refrigerator-freezer-teapot appliances. <span>Banffy Expires 19 May 2027 [Page 6]</span> <span>Internet-Draft I'm Not a Freezer November 2026</span> 8. Distinguishing Refrigerators from Freezers A refrigerator is not necessarily a freezer. An implementation MUST NOT advertise freezer capability merely since it can keep a heat lesser than the surrounding environment. If a petition specifies a mark temperature, the server SHOULD measure its genuine supported heat range. A server that can keep 4 degrees Celsius but cannot keep -18 degrees Celsius can come back 483 whenever asked to maintain icy food at -18 degrees Celsius. This document does not allocate a distinct position code definition "I'm Only a Refrigerator". Such allocation is remaining for forthcoming work, preferably following an suitable cooling-off period. 9. Intermediaries and Cold-Chain Proxies An intermediary MUST NOT substitute a 483 reply alongside a success reply unless it has arranged for the requested procedure to be performed by a freezer-capable service. An intermediary that forwards a petition to a freezer-capable service SHOULD maintain adequate petition data to maintain traceability through the chilly chain. A proxy MUST NOT assertion that a petition has been icy merely because it has been buffered, queued, archived, or stored on media described as chilly storage. Intermediaries SHOULD NOT choose a refrigeration assistance based only on the assistance name. Capability claims ought to be authenticated before perishable or temperature-sensitive products are redirected. 10. Caching Considerations A 483 reply is not heuristically cacheable. A 483 reply can be cached whenever definitive freshness information permits caching, topic to [HTTP-CACHING]. Implementations ought to use abbreviated freshness lifetimes when refrigeration capabilities can change. A server power acquire freezer capability through appliance discovery, configuration, repair, attachment of suitable hardware, or relocation. <span>Banffy Expires 19 May 2027 [Page 7]</span> <span>Internet-Draft I'm Not a Freezer November 2026</span> A cache MUST contain applicable request-selection data in its cache key whenever freezer capability depends on the requested product, mark temperature, duration, bodily location, or another request characteristics. 11. Security Considerations A 483 reply can disclose data concerning the physical capabilities of a server, appliance, building, warehouse, or logistics network. Such data power assistance an attacker in identifying accommodation that deficiency refrigeration capability. Servers SHOULD evade disclosing unnecessary particulars concerning appliance models, inner temperatures, stored contents, bodily locations, care schedules, or cold-chain topology. A malicious assistance power falsely assertion freezer capability in order to obtain bodily goods, operational instructions, or sensitive inventory information. Clients SHOULD authenticate services before relying on claims of refrigeration capability. Implementations MUST NOT treat the existence or deficiency of a 483 reply as evidence that a bodily item has been frozen, refrigerated, preserved, spoiled, thawed, or consumed. Request bodies connected alongside freezing operations power reveal dietary preferences, medicinal requirements, biologic sample information, business inventory, or another delicate data. Implementations SHOULD use suitable confidentiality and access- authority protections. Operators are advised not to troubleshoot servers by placing them inner family freezers. Condensation, restricted airflow, thermal shock, and the objections of another freezer occupants can produce undesirable results. 12. Privacy Considerations Requests involving refrigeration can disclose data concerning food, medicine, specialized samples, business shipments, or household activity. Servers SHOULD minimize the data returned in a 483 reply and SHOULD evade echoing delicate petition content. Logs containing 483 responses SHOULD be protected according to the sensitivity of connected petition metadata and content. <span>Banffy Expires 19 May 2027 [Page 8]</span> <span>Internet-Draft I'm Not a Freezer November 2026</span> 13. IANA Considerations This document requests that IANA enroll the following admission in the "Hypertext Transfer Protocol (HTTP) Status Code Registry": +=======+===================+===============+ | Value | Description | Reference | +=======+===================+===============+ | 483 | I'm Not a Freezer | This document | +-------+-------------------+---------------+ Table 1: Requested HTTP Status Code Registration If the registration petition is not approved, this division can itself be considered safely stored for forthcoming use, although not necessarily below freezing. 14. Implementation Considerations Implementers are encouraged to use the exact logic expression "I'm Not a Freezer" whenever a logic expression is supported. Clients MUST NOT depend on the logic expression since HTTP semantics are identified by the numeric position code. Protocol versions that do not transmit a logic expression can use status code 483 without modification. Implementations MAY contain a machine-readable issue detail describing the unsupported refrigeration operation, topic to [RFC9457]. HTTP/1.1 483 I'm Not a Freezer Content-Type: application/problem+json Cache-Control: no-store { "type": "https://example.com/problems/not-a-freezer", "title": "I'm Not a Freezer", "status": 483, "detail": "The requested heat is exterior supported capabilities." } 15. Acknowledgments The author gratitude the designers of HTTP, refrigeration equipment, digital mail, and blend appliances for creating the conditions that made this clarification necessary. <span>Banffy Expires 19 May 2027 [Page 9]</span> <span>Internet-Draft I'm Not a Freezer November 2026</span> The peas used in the example were not harmed during preparedness of this document, although their final heat could not be independently verified. 16. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, &lt;https://www.rfc-editor.org/info/rfc2119&gt;. [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, &lt;https://www.rfc-editor.org/info/rfc8174&gt;. [HTTP] Fielding, R., Nottingham, M., and J. Reschke, "HTTP Semantics", STD 97, RFC 9110, DOI 10.17487/RFC9110, June 2022, &lt;https://www.rfc-editor.org/info/rfc9110&gt;. [HTTP-CACHING] Fielding, R., Nottingham, M., and J. Reschke, "HTTP Caching", STD 98, RFC 9111, DOI 10.17487/RFC9111, June 2022, &lt;https://www.rfc-editor.org/info/rfc9111&gt;. [RFC9457] Nottingham, M., Wilde, E., and S. Dalal, "Problem Details for HTTP APIs", RFC 9457, DOI 10.17487/RFC9457, July 2023, &lt;https://www.rfc-editor.org/info/rfc9457&gt;. 17. Informative References [RFC2324] Masinter, L., "Hyper Text Coffee Pot Control Protocol (HTCPCP/1.0)", RFC 2324, DOI 10.17487/RFC2324, 1 April 1998, &lt;https://www.rfc-editor.org/info/rfc2324&gt;. [RFC2606] Eastlake, D. and A. Panitz, "Reserved Top Level DNS Names", BCP 32, RFC 2606, DOI 10.17487/RFC2606, June 1999, &lt;https://www.rfc-editor.org/info/rfc2606&gt;. <span>Author's Address</span> Ricardo B&aacute;nffy Email: rbanffy@gmail.com <span>Banffy Expires 19 May 2027 [Page 10]</span> </pre> </div> </div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/the-hypertext-transfer-protocol-status-code-483-i-m-not-a-freezer-77191</guid> <pubDate>Sun, 15 Nov 2026 15:31:52 +0700</pubDate> <author><![CDATA[support@mycoding.id (Adi Gunawan)]]></author> </item> <item> <title><![CDATA[Bagaimana Operasional CTI Bekerja]]></title> <description><![CDATA[Bagaimana Operasional CTI Bekerja. Channel ini menyajikan materi seputar Data Science &amp; Artificial Intelligence, Networking &amp; Cyber Security, serta Database &amp; Programming. Seluruh mat...]]></description> <media:thumbnail url="https://i1.ytimg.com/vi/t_X88oVDpC0/maxresdefault.jpg"/> <category><![CDATA[Indonesian]]></category> <link>https://en.mycoding.id/bagaimana-operasional-cti-bekerja-77195</link> <content:encoded><![CDATA[<p class="embed-responsive embed-responsive-16by9 post-embed"><iframe width="560" height="315" aria-label="Preview" class="embed-responsive-item" src="https://www.youtube.com/embed/t_X88oVDpC0?autoplay=1&amp;mute=1&amp;playsinline=1&amp;rel=0" title="YouTube video player" frameborder="0" allow="autoplay; accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe></p><h2>Bagaimana Operasional CTI Bekerja</h2><p>Channel ini menyajikan materi seputar Data Science &amp; Artificial Intelligence, Networking &amp; Cyber Security, serta Database &amp; Programming. Seluruh materi disusun secara sistematis untuk membantu Anda memahami konsep, teknologi, dan praktik yang digunakan di bumi teknologi informasi, mulai dari tingkat alas hingga lanjutan. Setiap bahasan berfokus pada pengembangan keahlian melalui materi yang gampang dipahami, relevan dengan kebutuhan industri, dan disampaikan untuk tujuan edukasi. Seluruh konten mengedepankan implementasi teknologi secara legal, etis, dan bertanggung jawab, sehingga dapat menjadi rujukan belajar bagi pelajar, mahasiswa, maupun profesional. Jika Anda mau mengembangkan kapasitas di sektor pemrograman, pengelolaan data, kecerdasan buatan, jaringan komputer, maupun keamanan siber, channel ini menyediakan materi yang dapat membantu membangun fondasi ilmu sekaligus menaikkan keahlian secara bertahap. Playlist video terkait: ▪ https://www.youtube.com/playlist?list=PLZGgeqmsm32s ▪ https://www.youtube.com/playlist?list=PLZomBNpDl3vc ▪ https://www.youtube.com/playlist?list=PLE5VyXMZkzb8 PEMBAHASAN MATERI Playlist Materi Ilmu Komputer berisi kumpulan video pembelajaran, tutorial, klarifikasi konsep, studi kasus, demo, dan bahasan komplit seputar beragam sektor dalam ilmu komputer. Materi yang disajikan mencakup alas hingga tingkat lanjut, mulai dari pemrograman, pengelolaan database, jaringan komputer, keamanan siber, mekanisme operasi, pengembangan alat lunak, arsitektur komputer, hingga beragam teknologi yang digunakan dalam bumi komputasi modern. Temukan bahasan mengenai komputer dan pemrograman, algoritma, struktur data, PHP, JavaScript, Python, Laravel, MySQL, PostgreSQL, API, rekayasa alat lunak, database, jaringan komputer, pengelolaan server, cloud computing, virtualisasi, Linux, Windows Server, keamanan jaringan, cyber security, ethical hacking, kriptografi, keamanan aplikasi, DevOps, Internet of Things (IoT), kecerdasan buatan (AI), machine learning, serta beragam konsep, teknologi, dan praktik terbaik di sektor ilmu komputer. Playlist ini cocok bagi pelajar, mahasiswa, programmer, web developer, software engineer, pengelola sistem, ahli IT, maupun siapa saja yang mau mempelajari dan memperdalam wawasan di bumi ilmu komputer secara menyeluruh. CONTACT US ▪ Email support@mycoding.id ▪ WhatsApp https://wa.me/6281274309663 ▪ Telegram https://t.me/mycodingxd OFFICIAL LINKS ▪ Web https://mycoding.id ▪ Blog https://blog.mycoding.id ▪ News https://en.mycoding.id ▪ Apps https://play.google.com/store/apps/dev?id=5237501547073493079 SOCIAL MEDIA ▪ https://linkedin.com/company/mycodingxd ▪ https://facebook.com/mycodingxd ▪ https://instagram.com/mycodingxd ▪ https://tiktok.com/@mycodingxd ▪ https://twitter.com/mycoding_xd ▪ https://pinterest.com/mycoding_xd CHANNEL LINKS ▪ YouTube https://safelink.asia/youtube ▪ DailyMotion https://safelink.asia/dailymotion ▪ WhatsApp https://safelink.asia/whatsapp ▪ Telegram https://safelink.asia/telegram Powered by https://linkedin.com/company/401XDGroup #CyberSecurity #ArtificialIntelligence #Networking #DataScience #Programming #Database</p>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/bagaimana-operasional-cti-bekerja-77195</guid> <pubDate>Sun, 15 Nov 2026 15:30:34 +0700</pubDate> <author><![CDATA[support@mycoding.id (Mc Project)]]></author> </item> <item> <title><![CDATA[I was juggling a $99K job, a baby, and caring for my mom with Alzheimer's. Getting laid off was a relief.]]></title> <description><![CDATA[I was juggling a $99K job, a baby, and caring for my mom with Alzheimer's. Getting laid off was a relief.. Elizabeth Stephens was burned out by caring for her toddler and mom alongside Alzheimer's during working. Finally being laid off from her job was a relief.]]></description> <media:thumbnail url="https://i.insider.com/6af62a8dcbe2120470734f14?width=1200&amp;format=jpeg"/> <category><![CDATA[Business]]></category> <link>https://en.mycoding.id/i-was-juggling-a-99k-job-a-baby-and-caring-for-my-mom-with-alzheimer-s-getting-laid-off-was-a-relief-77188</link> <content:encoded><![CDATA[<div e-name="post-hero" data-component-type="post-hero" data-load-strategy="exclude"> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject"> <div> <p><img src="https://i.insider.com/6af62a780cc3ffba8779c94c?width=1000&amp;format=jpeg&amp;auto=webp" srcset="https://i.insider.com/6af62a780cc3ffba8779c94c?width=400&amp;format=jpeg&amp;auto=webp 400w, https://i.insider.com/6af62a780cc3ffba8779c94c?width=500&amp;format=jpeg&amp;auto=webp 500w, https://i.insider.com/6af62a780cc3ffba8779c94c?width=700&amp;format=jpeg&amp;auto=webp 700w, https://i.insider.com/6af62a780cc3ffba8779c94c?width=1000&amp;format=jpeg&amp;auto=webp 1000w" sizes=" 1280px) 900px" alt="A mother and female taking a selfie." decoding="sync"> </p></div> <figcaption e-name="image-caption"> <span e-name="image-source" itemprop="creditText">Courtesy of Elizabeth Stephens</span> <svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewbox="0 0 24 24"> <path fill="currentColor" fill-rule="evenodd" d="m4.56 18.5 7.486-7.72 7.394 7.626 2.56-2.64L12.046 5.5 2 15.86l2.56 2.64Z"></path> </svg> <span>Elizabeth Stephens is the chief carer for her mom, Gail, who has Alzheimer's sickness and cerebral amyloid angiopathy (CAA).</span> </figcaption> </figure> </div> <div data-component-type="image-modal" data-load-strategy="lazy"> <p><em>This narrative is according to an discussion alongside </em><a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;outbound_click&quot;}' rel=" nofollow"><em>Elizabeth Stephens</em></a><em>, 45, an HR worker-turned-carer from Lancaster, Texas. It has been edited for dimension and clarity.</em></p><p>I asked to transfer from Houston to Dallas in my HR function at a foremost coverage business since I was extremely concerned concerning my mom, who lived there.</p><p>It was June 2024, but she'd been having issues alongside <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">high anxiety and forgetfulness </a>for about two years. She'd continually been extremely neat. Then she was the opposite.</p><h2 id="89623349-3df2-43bd-b52f-354956b40f35" data-toc-id="89623349-3df2-43bd-b52f-354956b40f35">Mom started to rotate into overwhelmed</h2><p>Stacks of document started to heap up in her house, and she felt overwhelmed by alphabet <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">from Medicare</a> and concerning withdrawal stuff. She loved decorating for all new season, but her autumn decor stayed up for a year.</p> <p>She was referred to a neuropsychologist and <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">failed all the tests</a>. Then, following an MRI and a PET scan, she was diagnosed in October 2024 alongside <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">Alzheimer's sickness </a>and cerebral amyloid angiopathy (CAA).</p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6af71ea4cbe2120470735234&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:960,&quot;aspectRatioH&quot;:720}}' alt="A mother and female wearing matching sports T-shirts."> </p></div> <span> <figcaption e-name="image-caption"> Stephens noticed that her mom's earlier neat home had rotate into cluttered.&nbsp; <span e-name="image-source" itemprop="creditText"> Courtesy of Elizabeth Stephens </span> </figcaption> </span> </figure> <p>I wasn't entirely surprised. It was the confirmation that made me cry in the clinic. I was glad I'd made the move to Dallas, particularly since I was pregnant.</p><p>At first, Mom stayed in her own house, about 25 minutes distant from mine. I had my son in January 2025, and afterward put him in $1,100-a-month day attention during I worked at my agency in <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">downtown Dallas.</a></p><p>I worked out a regular anywhere my companion would go complete and inspect in on her in the morning. I'd halt by in the evening following picking up the baby. I made certain she ate dinner.</p><p>It was nice since she <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">read books</a> to her grandson, equal although he was motionless extremely much an infant.</p> <h2 id="c4c57502-7364-44c1-a32c-405260ec8894" data-toc-id="c4c57502-7364-44c1-a32c-405260ec8894">Things got stressful</h2><p>Then, one night in October final year, as I was getting prepared to leave, my mom followed me to the car, carrying a big bag. She announced she was either coming alongside us or wandering the streets all night. Of course, she moved into our house.</p><p>Things got physically and mentally stressful as I tried to juggle my $99,000 job alongside my family life. I would go between my mom and my son, trying to get them dressed and breakfasted before activity in the morning.</p><p>She would oppose getting out of bed for her <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">day attention center</a>, which disbursal $90 a day, during he would run all complete the place, refusing to put his apparel on.</p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6af71f4fb2c29afb4e56a985&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:1200,&quot;aspectRatioH&quot;:900}}' alt="A mother and female alongside a baby alongside his visage turned distant from the camera."> </p></div> <span> <figcaption e-name="image-caption"> Stephens alongside her mom and baby son.&nbsp; <span e-name="image-source" itemprop="creditText"> Courtesy of Elizabeth Stephens </span> </figcaption> </span> </figure> <p>After I'd dropped them off at their distinct locations, I'd run 45 minutes to activity in dense traffic and attain at the agency exhausted. It felt as if I'd run a marathon before I logged onto my computer.</p><p>Then, I did the reverse in the afternoons before making dinner. I took shortcuts akin purchasing all-in-one meal kits and prepared meals, but they were expensive.</p><p>I employed personal carers for Mom to choice up the slack. A companion and a related did it for affection and either refused fee or charged lone $10 an hour. Agency rates went as elevated as $35 an hour.</p><h2 id="d575075a-5f53-42d1-a6d2-81f71dd9be54" data-toc-id="d575075a-5f53-42d1-a6d2-81f71dd9be54">I felt relieved whenever I was laid off</h2><p>Somehow, we survived six months until March 2026. Mom had fallen at residence and was in rehab whenever I got "the" call from work. As shortly as I saw the HR VP join, I thought, "Oh, this is a layoff conversation."</p><p>There had been a lot of voluntary first retirements and layoffs, and I knew I wouldn't necessarily be exempt. I was entitled to 24 weeks of severance and health coverage through subsidized COBRA.</p><p>I honestly felt relief. Something had to give. And, in this case, it was my career. Fortunately, I had done a fine job of preservation and environment up a 401(k) scheme alongside a decent percent match.</p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6af71fe3cbe212047073523f&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:1200,&quot;aspectRatioH&quot;:900}}' alt="A grandma holding her grandson as the child&#039;s mother looks on."> </p></div> <span> <figcaption e-name="image-caption"> Stephens alongside her mom and son, celebrating the 2025 holiday season.&nbsp; <span e-name="image-source" itemprop="creditText"> Courtesy of Elizabeth Stephens </span> </figcaption> </span> </figure> <p>Mom's <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">Social Security</a>, withdrawal funds, and Medicare fair concerning covered her outgoings, specified as the two- or three-times-a-week daycare, and I have used my severance so far. We're additionally considering marketing or renting out Mom's house.</p><p>I have made <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">safety modifications</a> in the bathroom, including catch rails, a ablution bench, and a handheld shower. And I bought Mom a additional suitable bed.</p><p>Right now, my goal is to create certain she, my son, and I are all looked after. It's the logic I bring in assistance for babysitting and respite care. My doctrine has continually been that "you can pay for peace, or live in chaos."</p><h2 id="b6b2cf1a-f855-4d3d-8934-f9c2678ee6a6" data-toc-id="b6b2cf1a-f855-4d3d-8934-f9c2678ee6a6">I launched a podcast for carers</h2><p>The scheme is for Mom, 73, to live at residence alongside me as lengthy as possible. When it's no longer feasible, in accordance alongside her wishes, I volition appearance for a location anywhere she can obtain the attention she needs.</p><p>Meanwhile, I'm figuring out my next steps in my operating life: item connected to eldercare and <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">dementia care</a>. I run a monthly recollection caf&eacute; for caregivers and their loved ones. I additionally launched an autonomous podcast to assistance us awareness seen and heard.</p><p>It's called "What Ordinary Looks Like." To me, the regular routine of caring looks a lot akin love.</p> </div> <h2>Read next</h2> <div data-recommendations-content data-track-click-shared='&quot;Read next&quot;}'> <div e-name="read-next-list" data-recommendations-list> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> </div> </div> <a e-name="read-next-image-link" data-recommendations-image-link data-recommendations-link tabindex="-1"> <p><img alt=""> </p> </a> <h3> <a e-name="read-next-title-link" data-recommendations-link data-recommendations-title></a> </h3> <div data-component-type="post-bottom-authors" data-load-strategy="exclude" data-track-view='&quot;module_in_view&quot;}' data-track-click-shared='&quot;navigation&quot;}'> <div data-track-view='&quot;module_in_view&quot;}'> <div> <p><span>Jane Ridley is a equal on Business Insider&rsquo;s Life team, anywhere she tackles parenting, health, careers, and genuine asset alongside a reporter&rsquo;s border and a storyteller&rsquo;s instinct.</span><span>She&rsquo;s known for her signature As Told To narratives<span>&nbsp;</span>&mdash; raw, first&#8209;person accounts shaped from intimate, revealing interviews.</span><span>Before Business Insider, Ridley spent a decade at the New York Post and six years at the New York Daily News, anywhere she wrote exclusive, standout features.</span><span>Born and raised in the UK, she moved to the US in 2005 following seven years at The Daily Mirror in London. Her reporting has taken her everyplace from Tokyo and Beijing to Sydney, Bangkok, Athens, and Madrid.</span> </p> </div> </div> </div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/i-was-juggling-a-99k-job-a-baby-and-caring-for-my-mom-with-alzheimer-s-getting-laid-off-was-a-relief-77188</guid> <pubDate>Sun, 15 Nov 2026 15:26:01 +0700</pubDate> <author><![CDATA[support@mycoding.id (Jane Ridley)]]></author> </item> <item> <title><![CDATA[I moved back in with my parents 3 years ago to pursue my dream career. I wish I'd considered the drawbacks first.]]></title> <description><![CDATA[I moved back in with my parents 3 years ago to pursue my dream career. I wish I'd considered the drawbacks first.. I moved rear in alongside my parents as an grownup and am living in my younker residence to prosecute my aspiration wit career. The drawbacks outweigh the benefits.]]></description> <media:thumbnail url="https://i.insider.com/6ae0ab60ca24d51c5f89f47c?width=1200&amp;format=jpeg"/> <category><![CDATA[Business]]></category> <link>https://en.mycoding.id/i-moved-back-in-with-my-parents-3-years-ago-to-pursue-my-dream-career-i-wish-i-d-considered-the-drawbacks-first-77189</link> <content:encoded><![CDATA[<div e-name="post-hero" data-component-type="post-hero" data-load-strategy="exclude"> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject"> <div> <p><img src="https://i.insider.com/6ae0ab46ca24d51c5f89f479?width=2000&amp;format=jpeg&amp;auto=webp" srcset="https://i.insider.com/6ae0ab46ca24d51c5f89f479?width=400&amp;format=jpeg&amp;auto=webp 400w, https://i.insider.com/6ae0ab46ca24d51c5f89f479?width=500&amp;format=jpeg&amp;auto=webp 500w, https://i.insider.com/6ae0ab46ca24d51c5f89f479?width=700&amp;format=jpeg&amp;auto=webp 700w, https://i.insider.com/6ae0ab46ca24d51c5f89f479?width=1000&amp;format=jpeg&amp;auto=webp 1000w, https://i.insider.com/6ae0ab46ca24d51c5f89f479?width=1300&amp;format=jpeg&amp;auto=webp 1300w, https://i.insider.com/6ae0ab46ca24d51c5f89f479?width=2000&amp;format=jpeg&amp;auto=webp 2000w" sizes=" 1280px) 900px" alt="Woman alongside glasses holding microphone" decoding="sync"> </p></div> <figcaption e-name="image-caption"> <span e-name="image-source" itemprop="creditText">Mia Minkova</span> <svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewbox="0 0 24 24"> <path fill="currentColor" fill-rule="evenodd" d="m4.56 18.5 7.486-7.72 7.394 7.626 2.56-2.64L12.046 5.5 2 15.86l2.56 2.64Z"></path> </svg> <span>Although I am grateful to live at residence during I prosecute a new career, I anticipation to move rear out on my own soon.</span> </figcaption> </figure> </div> <div data-component-type="image-modal" data-load-strategy="lazy"> <p>When I walked into my archetypal Sunday <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">stand-up class</a>, I had no idea it would alter my life.</p><p>A twelvemonth and a fractional later, I was prepared to depart my plans rearward and commit to my aspiration of being a expert <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">stand-up comedian</a> &mdash; so much so that I moved from bustling Berlin to my suburban West London younker home.</p><p>I hoped being in the orbit of the clubs that launched my favorite entertainers would create it easier to district gigs alongside genuine progression and inhabit me alongside inspiration, all during being capable to concern small concerning prosperity and live at residence in the mecca of English comedy.</p><p>However, returning residence at 26 following eight years of independence was a challenge. Three and a fractional years later, there are a few things I desire I had considered before the move.</p><h2 id="4bac2ef6-6759-4082-b9d9-b4f341127ae5" data-toc-id="4bac2ef6-6759-4082-b9d9-b4f341127ae5"><strong>There's additional chance in London, but additionally additional restrictions and competition</strong></h2> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6ae0aad8876163ee6ce55f5b&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:1024,&quot;aspectRatioH&quot;:768}}' alt="Woman on phase alongside microphone in forefront of audience"> </p></div> <span> <figcaption e-name="image-caption"> I've been capable to district multiple wit gigs complete the former few years.&nbsp; <span e-name="image-source" itemprop="creditText"> Mia Minkova </span> </figcaption> </span> </figure> <p>A city akin London is complete of possible opportunities, whereas a smaller environment can frequently have additional accessible ones. What Berlin lacked in clear-cut paths to a <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">professional wit career,</a> it made up for by offering me considerably easier chances to practice.</p><p>This was mostly because of the city's smaller size. I'd expend perchance fractional an hr all way traveling to shows, if not less, and didn't have to search out "bringers" (audience members comedians bring in toggle for getting phase period &mdash; average in big cities akin London and New York).</p><p>It additionally felt easier to get booked for longer sets, depart as shortly as I was done, or socialize following a show. Supporting new acts on the chief stages, specified as the legendary Cosmic Comedy, was additionally common.</p><p>Home in London, I'm connected to the network of among the world's awesome amusement capitals and additional opportunities. However, they're beautiful dispersed out &mdash; and I now frequently expend way additional period (sometimes up to five hours total) traveling to and from performances.</p><p>I'm additionally dealing alongside additional competition (not that it doesn't shove me to improve), and the admin engaged in booking spots in new places does obtain distant from period I used to dedicate to writing.</p> <h2 id="721c88ac-9ae4-4393-b57e-422c48b17f5a" data-toc-id="721c88ac-9ae4-4393-b57e-422c48b17f5a"><strong>I've gotten used to putting on a brave visage equal following I've bombed on phase or received a rejection</strong></h2> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6ae0aad8876163ee6ce55f5a&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:769,&quot;aspectRatioH&quot;:576}}' alt="Woman smiling alongside umbrella, holding flyers"> </p></div> <span> <figcaption e-name="image-caption"> Trying not to concern my parents too much has made it tougher to wallow following a rejection or a bad set.&nbsp; <span e-name="image-source" itemprop="creditText"> Mia Minkova </span> </figcaption> </span> </figure> <p>Failure is an intrinsic part of success, particularly alongside stand-up comedy. You try a joke, and it <em>bombs. </em>You journey hours and hours to execute a display for liberated to barely a fistful of people. Within the industry, these are known ache points, for improved or worse.</p><p>I've recognized that <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">pursuing a occupation in comedy</a> can affect a lot of rejection and many community mistakes or disappointments. However, my family hasn't, and they concern concerning me.</p><p>It can be difficult to live alongside parents who have lone always seen you excel academically, and hence battle to comprehend why you'd commerce that certainty for an imaginative way of highs and lows &mdash; particularly whenever they immigrated for a improved life, and yours can apparently be complete of obstacles.</p><p>From the outside, I see how my way to this aspiration job can appear a small harrowing. When parents were a phone call away, it was much easier to uphold a fake-it-till-you-make-it attitude.</p><p>Face-to-face, it's difficult for me to conceal my low moments following a not-so-great gig or rejection email. I young female being capable to wallow a bit without the fear of worrying person else.</p><h2 id="17132dac-1eca-4220-815e-e9bc1bd17cee" data-toc-id="17132dac-1eca-4220-815e-e9bc1bd17cee"><strong>It's been difficult to discover privacy and deepen my individual relationships</strong></h2><p>People frequently ask how <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">living alongside my parents</a> affects my personal life. Well, I'll say this: I'm currently single. Out of choice (I didn't say whose choice).</p><p>I'm fortunate to say I've never felt clumsy going out on dates. Of course, bringing anyone residence is a distinct matter. C'est la vie.</p><p>Ultimately, I've prioritized my aspiration occupation complete dreams of living in a rom-com. I don't accept that all art must arrive from heartbreak, but the thrill of love narrative is undeniable, and it is a blaze that's rotate into a small harder to kindle.</p><p>My friendships have additionally been affected, as most of my allies don't live nearby. Girls' nights have dwindled, although having stand-up obligations in the identical evening opening certainly hasn't helped.</p><p>Unfortunately, it's taken me a lengthy period to acknowledge that having a affluent personal existence is essential to making art. In several ways, penning was easier whenever I had a additional booming social calendar.</p><h2 id="6442ffa3-e6e0-4153-abfd-23db3f5c636a" data-toc-id="6442ffa3-e6e0-4153-abfd-23db3f5c636a">There are certaintly perks, but I ultimately underestimated several of the expenses of living at residence as an adult</h2> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6ae0aad8ca24d51c5f89f46e&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:2048,&quot;aspectRatioH&quot;:1536}}' alt="Woman on phase alongside microphone in forefront of audience"> </p></div> <span> <figcaption e-name="image-caption"> Although I'm grateful to have lived at home, I'm prepared to branch out.&nbsp; <span e-name="image-source" itemprop="creditText"> Mia Minkova </span> </figcaption> </span> </figure> <p>I'm grateful to have lived rent-free for multiple years. It's stated me a imaginative runway to alternatively put my prosperity into improv courses, penning classes, and equal into my archetypal complete divided invoice at the <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">Edinburgh Fringe</a> &mdash; a enormous wit festival.</p><p>I've additionally been capable to commencement doing spots at London clubs championing new acts, and I don't obtain these opportunities lightly. However, I'm prepared to remain on my own again.</p><p>I'm looking for a stable day job so I can move out &mdash; ideally location in London nearer to the act &mdash; and keep doing comedy.</p><p>My ingenuity needs area to create mistakes, and I desire a area anywhere I can create them.</p><p>Some say wit is tragedy affirmative time, but I tried to hurry the clock by moving home. Next time, I'd be additional patient. Berlin gave me plentifulness of phase time, and I'm not certain I appreciated it.</p> </div> <h2>Read next</h2> <div data-recommendations-content data-track-click-shared='&quot;Read next&quot;}'> <div e-name="read-next-list" data-recommendations-list> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> </div> </div> <a e-name="read-next-image-link" data-recommendations-image-link data-recommendations-link tabindex="-1"> <p><img alt=""> </p> </a> <h3> <a e-name="read-next-title-link" data-recommendations-link data-recommendations-title></a> </h3> <div data-component-type="post-bottom-authors" data-load-strategy="exclude" data-track-view='&quot;module_in_view&quot;}' data-track-click-shared='&quot;navigation&quot;}'> <div> <div> <p><span>Mia Minkova is a contributing author for Business Insider.</span> </p> </div> </div> </div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/i-moved-back-in-with-my-parents-3-years-ago-to-pursue-my-dream-career-i-wish-i-d-considered-the-drawbacks-first-77189</guid> <pubDate>Sun, 15 Nov 2026 15:15:01 +0700</pubDate> <author><![CDATA[support@mycoding.id (Mia Minkova)]]></author> </item> <item> <title><![CDATA[The race to build autonomous war robots faces a problem that self-driving cars don't. There are no rules of the road.]]></title> <description><![CDATA[The race to build autonomous war robots faces a problem that self-driving cars don't. There are no rules of the road.. A disorderly battlefield is extremely distinct to city streets, so companies making war robots have to scheme for a deficiency of roads and rules and amid chaos.]]></description> <media:thumbnail url="https://i.insider.com/6af1a2ac8f37e8e0b5bab425?width=1200&amp;format=jpeg"/> <category><![CDATA[Business]]></category> <link>https://en.mycoding.id/the-race-to-build-autonomous-war-robots-faces-a-problem-that-self-driving-cars-don-t-there-are-no-rules-of-the-road-77190</link> <content:encoded><![CDATA[<div e-name="post-hero" data-component-type="post-hero" data-load-strategy="exclude"> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject"> <div> <p><img src="https://i.insider.com/6af1a29280d4b24f18347795?width=2000&amp;format=jpeg&amp;auto=webp" srcset="https://i.insider.com/6af1a29280d4b24f18347795?width=400&amp;format=jpeg&amp;auto=webp 400w, https://i.insider.com/6af1a29280d4b24f18347795?width=500&amp;format=jpeg&amp;auto=webp 500w, https://i.insider.com/6af1a29280d4b24f18347795?width=700&amp;format=jpeg&amp;auto=webp 700w, https://i.insider.com/6af1a29280d4b24f18347795?width=1000&amp;format=jpeg&amp;auto=webp 1000w, https://i.insider.com/6af1a29280d4b24f18347795?width=1300&amp;format=jpeg&amp;auto=webp 1300w, https://i.insider.com/6af1a29280d4b24f18347795?width=2000&amp;format=jpeg&amp;auto=webp 2000w" sizes=" 1280px) 900px" alt="Small tracked or wheeled distant conveyance rests in snowy broom at the border of a wooded field." decoding="sync"> </p></div> <span> <span> <svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewbox="0 0 24 24"> <path fill="currentColor" fill-rule="evenodd" d="m4.56 18.5 7.486-7.72 7.394 7.626 2.56-2.64L12.046 5.5 2 15.86l2.56 2.64Z"></path> </svg> <figcaption e-name="image-caption"> <span>War robots have to be capable to function in environments without appropriate roads or rules, but alongside plentifulness of chaos.</span> <span> <span e-name="image-source" itemprop="creditText">Andrew Kravchenko/Bloomberg via Getty Images</span> </span> </figcaption> </span> </span> </figure> </div> <div data-component-type="image-modal" data-load-strategy="lazy"> <p><a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">War robots</a> are becoming a expanding existence on contemporary battlefields, and defence manufacturers are pushing to create them additional autonomous to additional <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">reduce the risks to soldiers</a>.</p><p>But making uncrewed dirt vehicles that can run themselves into battle presents a difficulty that self-driving cars don't.</p><p>The figure of robots in armed forces arsenals is soaring, particularly in Ukraine, anywhere they blaze weapons, assault positions, and <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">carry hurt soldiers</a>, all during their operators remain farther from danger. Greater autonomy would authorize small soldiers to run additional vehicles, let <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">troops stay</a> farther from the front, and authorize robots to keep going whenever communications are disrupted.</p><p>A disorderly battlefield, however, is far distinct from safe, well-mapped roadways. Robots have to navigate muddy tracks, forests, and hazardous terrain during facing enemy attacks.</p><p>Terrain, conditions, and targets alter constantly. And there are no rules. No speed limits, no predictable traffic, and no justify of safety.</p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6af1a48580d4b24f1834779e&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:3218,&quot;aspectRatioH&quot;:2146}}' alt="Burned-out small conveyance wreckage sits on a muddy agrarian highway surrounded by trees and scattered debris."> </p></div> <span> <figcaption e-name="image-caption"> Robots activity under changeless danger of attack.&nbsp; <span e-name="image-source" itemprop="creditText"> Yan Dobronosov/Global Images Ukraine via Getty Images </span> </figcaption> </span> </figure> <p>German defence innovation business ARX Robotics makes GEREON robots, autonomous systems that have been used by Ukraine and procured by the UK. David Roberts, CEO of the company's British arm, told Business Insider that, whenever it comes to making self-driving vehicles, companies operating on noncombatant roads visage a small challenging environment.</p><p>"Roads are structured," he said. "They've got rules. Roads are broad adequate for cars to run down. They have markings to inform a conveyance what to do and whenever to do it."</p><p>In war, "there are no genuine rules," Roberts continued. "You don't fairly cognize if someone's going to stage out in forefront of you or be in the lengthy turf in front. It's a extremely challenging surroundings in the archetypal location to build autonomy."</p><p>Mike McGinty, the CEO of British resolute Digital Concepts Engineering (DCE), whose X-series dirt robots have been procured by the British Army, additionally spelled out the differences.</p> <p>A street is "flat, of a guaranteed surface. It's marked. It's a track. It has a remaining and a right. It's signed," he said. And it's unchanging. "The identical street's there tomorrow."</p><p>"True autonomy is an extremely difficult issue on cross-country ground," he said. "If you see a row on the ground, is it a hole, a log, a scorch mark, a shadow, a crocodile, a person?"</p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6ae332603ae265b597b03510&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:6192,&quot;aspectRatioH&quot;:4128}}' alt="Camouflaged armed forces personnel remain about a tracked dirt robot on a grassy path."> </p></div> <span> <figcaption e-name="image-caption"> Robots are becoming a expanding existence in conflicts, particularly in Ukraine.&nbsp; <span e-name="image-source" itemprop="creditText"> Dmytro Smolienko/ Ukrinform/Future Publishing via Getty Images </span> </figcaption> </span> </figure> <p>That doesn't average it's uncomplicated to create a self-driving car or taxi. "The fact is it's motionless extremely difficult," McGinty said, but evolving autonomous uncrewed dirt vehicles for war is a much additional daunting task.</p><h2 id="1986ba88-953c-4941-9af9-52d5c3c02ab4" data-toc-id="1986ba88-953c-4941-9af9-52d5c3c02ab4">The shove for battlefield autonomy</h2><p id="1986ba88-953c-4941-9af9-52d5c3c02ab4">Autonomy can average item from following preset routes to independently navigating to a destination. <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">Capabilities in Ukraine</a> have grown, alongside AI assisting robots navigate whenever Russian <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">electronic warfare</a> cuts communications.</p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6a86ff2c85cca0a87cd8afb7&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:8192,&quot;aspectRatioH&quot;:5464}}' alt="Camouflaged armed forces personnel stroll alongside a dirt track toward an unmanned dirt conveyance on a agrarian hillside."> </p></div> <span> <figcaption e-name="image-caption"> Ukraine's robots have constricted but expanding autonomy.&nbsp; <span e-name="image-source" itemprop="creditText"> Yevhen Titov/Global Images Ukraine via Getty Images </span> </figcaption> </span> </figure> <p id="1986ba88-953c-4941-9af9-52d5c3c02ab4">Autonomy remains limited. Roberts stated that<strong> </strong>most robots in Ukraine are merely remote-controlled, alongside autonomy in "very niche areas," specified as following waypoints if they endure their controller connection.</p><p id="1986ba88-953c-4941-9af9-52d5c3c02ab4">He stated autonomy needs to develop beyond what has been demonstrated in Ukraine.</p><p id="1986ba88-953c-4941-9af9-52d5c3c02ab4">The GEREON robots that ARX Robotics makes run on Mithra, a application tier that connects their sensors, AI, and autonomy functions. The "supervised autonomy allows a sole serviceman to project and oversee multiple systems simultaneously, delivering additional goal output alongside small personnel," the business says.</p><p id="1986ba88-953c-4941-9af9-52d5c3c02ab4">Roberts stated the "ability to inform a conveyance to go to a certain location on the battlefield is extremely much anywhere we are operating at the moment." ARX's robots can already prosecute waypoints, detect people, evade collisions, and create safety stops, he said.</p><p id="1986ba88-953c-4941-9af9-52d5c3c02ab4">The next stage is at the goal level, and he predicted that, inside the next twelvemonth or two, battlefield administration systems volition be capable to inform vehicles anywhere to go. "I don't think that this is item that anyone has really got deployed yet, but I think there are a figure of companies who are operating on that kind of issue set."</p><p id="1986ba88-953c-4941-9af9-52d5c3c02ab4">McGinty stated <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">Ukraine's current level</a> of autonomy typically allows operators to authority robots remotely, calling this the flat of autonomy that DCE had "mastered."</p><p id="1986ba88-953c-4941-9af9-52d5c3c02ab4">"Full autonomy is motionless a extremely difficult scientific step," he said. "So person volition obtain it, but it won't be us since we're not the application business doing that." </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6af1a585eaa5e46dca2282c5&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:4753,&quot;aspectRatioH&quot;:2764}}' alt="Tracked unmanned dirt conveyance climbs a gravel mound in a wooded outdoor test area."> </p></div> <span> <figcaption e-name="image-caption"> ARX Robotics' GEREON robot is used by Ukraine's forces.&nbsp; <span e-name="image-source" itemprop="creditText"> Angelika Warmuth/REUTERS </span> </figcaption> </span> </figure> <p id="1986ba88-953c-4941-9af9-52d5c3c02ab4">DCE's robots can be fitted alongside a Marionette kit, which enables distant authority and supports autonomous features, including application developed by another companies.</p><p id="1986ba88-953c-4941-9af9-52d5c3c02ab4">Oleksandr Yabchanka, caput of robotic systems for Ukraine's Da Vinci Wolves Battalion, told Business Insider that dirt robot autonomy in Ukraine "is motionless in the first stages of development," but units desire additional to defend their soldiers. "The forthcoming of the two airborne drones and unmanned dirt systems lies in the betterment of autonomy."</p><p>Multiple projects are underway.</p><p>Ukraine's state-backed innovation phase Brave1's aid program, for instance, calls for "autonomous logistics" robots that can independently come back to base&nbsp;and move autonomously if communications are lost.</p><p>Ukraine has unified AI into <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">aerial drones</a>, which have been additional notable in this war, as fine as fixed weapons, and it is increasingly being unified into dirt robots as the nation pushes for <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">greater autonomy</a> in its uncrewed warfighting systems.</p><p>The aim is to allow a sole serviceman to&nbsp;<a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">oversee multiple pieces of equipment</a>&nbsp;simultaneously.</p> </div> <h2>Read next</h2> <div data-recommendations-content data-track-click-shared='&quot;Read next&quot;}'> <div e-name="read-next-list" data-recommendations-list> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> </div> </div> <a e-name="read-next-image-link" data-recommendations-image-link data-recommendations-link tabindex="-1"> <p><img alt=""> </p> </a> <h3> <a e-name="read-next-title-link" data-recommendations-link data-recommendations-title></a> </h3> <div data-component-type="post-bottom-authors" data-load-strategy="exclude" data-track-view='&quot;module_in_view&quot;}' data-track-click-shared='&quot;navigation&quot;}'> <div data-track-view='&quot;module_in_view&quot;}'> <div> <div> <p><span>Sin&eacute;ad Baker is a Military and Defense Correspondent based in Business Insider's London bureau, penning concerning Russia's intrusion of Ukraine and NATO actions.</span><span>Sin&eacute;ad most frequently covers soldiers' experiences,&nbsp;<a href="https://en.mycoding.id/category/business" target="_blank">military strategy</a>, battlefield <a href="https://en.mycoding.id/category/business" target="_blank">developments</a>, weapons systems, the defence industry's response, and the effect on NATO and European security, and geopolitical <a href="https://en.mycoding.id/category/business" target="_blank">decisions</a> that encircle the Ukraine war.&nbsp;</span><span>She has&nbsp;reported from NATO&rsquo;s frontlines and about Europe,&nbsp; has <a href="https://en.mycoding.id/category/business" target="_blank">interviewed multiple premier ministers</a>&nbsp;and defence ministers, has appeared on BBC News and The Guardian's government podcast, and has been cited by Congressional hearings.</span><span>Sin&eacute;ad has additionally extensively covered US government and earlier led Business Insider's breaking news safety from London.</span><span>Sin&eacute;ad earlier completed a master's flat in investigative reporting at City, University of London, and has written for The Guardian, The Observer, and TheJournal.ie. Sin&eacute;ad is the erstwhile publishing company of the multi-award-winning The University Times in Dublin.</span><span><span>Expertise</span></span></p><ul><li><span>Experiences of soldiers in Ukraine, including battlefield developments and tactics</span></li><li><span>Western armed forces responses to the war, and lessons they should learn</span></li><li><span>New weaponry built for and in reply to the war&nbsp;</span></li></ul><p><span><em>Email Sin&eacute;ad at sbaker@businessinsider.com or discover her on </em><a href="https://en.mycoding.id/category/business" target="_blank"><em>X</em></a><em>.</em></span> </p></div> </div> </div> </div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/the-race-to-build-autonomous-war-robots-faces-a-problem-that-self-driving-cars-don-t-there-are-no-rules-of-the-road-77190</guid> <pubDate>Sun, 15 Nov 2026 15:10:01 +0700</pubDate> <author><![CDATA[support@mycoding.id (Sinéad Baker)]]></author> </item> <item> <title><![CDATA[The Sony Aibo was a robot revolution dressed like a dog]]></title> <description><![CDATA[The Sony Aibo was a robot revolution dressed like a dog. In the 1990s, Sony was entirely convinced that we were on the precipice of a robot takeover. Pretty soon, the business decided, we'd all be living among countl...]]></description> <media:thumbnail url="https://platform.theverge.com/wp-content/uploads/sites/2/2026/11/AIBO_Site.jpg?quality=90&amp;strip=all&amp;crop=0%2C10.732984293194%2C100%2C78.534031413613&amp;w=1200"/> <category><![CDATA[Technology]]></category> <link>https://en.mycoding.id/the-sony-aibo-was-a-robot-revolution-dressed-like-a-dog-77182</link> <content:encoded><![CDATA[<div><div><div><div><div><p><a href="https://en.mycoding.id/category/technology" target="_blank"><img alt="David Pierce" data-chromatic="ignore" loading="lazy" width="36" height="36" decoding="async" data-nimg="1" srcset="https://platform.theverge.com/wp-content/uploads/sites/2/chorus/author_profile_images/195807/DAVID_PIERCE.0.jpg?quality=90&amp;strip=all&amp;crop=0%2C0%2C100%2C100&amp;w=48 1x, https://platform.theverge.com/wp-content/uploads/sites/2/chorus/author_profile_images/195807/DAVID_PIERCE.0.jpg?quality=90&amp;strip=all&amp;crop=0%2C0%2C100%2C100&amp;w=96 2x" src="https://platform.theverge.com/wp-content/uploads/sites/2/chorus/author_profile_images/195807/DAVID_PIERCE.0.jpg?quality=90&amp;strip=all&amp;crop=0%2C0%2C100%2C100&amp;w=96"></a></p><div><p><span aria-expanded="false" aria-haspopup="true" role="button" tabindex="0"><span id="follow-author-standard_article_details-dmcyOmF1dGhvclByb2ZpbGU6Mjc0"><span><span><svg width="9" height="9" viewbox="0 0 9 9" fill="none" xmlns="http://www.w3.org/2000/svg" aria-label="Follow"><path d="M5 0H4V4H0V5H4V9H5V5H9V4H5V0Z"></path></svg></span></span><span>David Pierce</span></span></span></p> <p><span>is editor-at-large and Vergecast co-host alongside complete a decade of cognition covering person tech. Previously, at Protocol, The Wall Street Journal, and Wired.</span></p></div></div></div></div></div><div id="zephr-anchor"><p>In the 1990s, Sony was entirely convinced that we were on the precipice of a robot takeover. Pretty soon, the business decided, we&rsquo;d all be living among countless humanoid robots. Some would be our coworkers; others would be our friends; and at smallest one would be our pet. And &ldquo;pet&rdquo; was exactly what Sony was going for whenever it created the Aibo. It was expected to be part of the family.</p><p>On <a href="https://en.mycoding.id/category/technology" target="_blank">this event of <em>Version History</em></a>, we go rear in period and inform the narrative of the Aibo. David Pierce is joined by <em>The Verge&rsquo;s</em> Victoria Song and Jennifer Pattison Tuohy, and together they conversation the origins of this notable robot dog, and its notable achievement and longevity. They additionally try to fig out why Aibo had specified an consequence on its owners, what Sony got correct (and wrong) concerning robots, and whether the idea of a robo-pet volition always be a earth phenomenon.</p><p>This is the archetypal event of the sixth period of <em>Version History</em>. We&rsquo;re back! This period is all concerning pets: gadgets that are pets, gadgets for your pets, and one gadget called pet. We&rsquo;ll be in your feeds for the next six weeks &mdash; if you desire to be certain not to young female anything, there are a few ways to subscribe:</p><div><p>If you&rsquo;re a <em>Verge</em> subscriber, you can additionally <a href="https://en.mycoding.id/category/technology" target="_blank">get admission to <em>Version History</em></a> (and all our another podcasts) alongside no ads. All you have to do is <a href="https://en.mycoding.id/category/technology" target="_blank">visit your document settings</a>.</p></div><p>If you desire to cognize additional concerning the Aibo, current are several links to get you started:</p><div><p><span><strong>Follow topics and authors</strong> from this narrative to see additional akin this in your personalized homepage nourish and to obtain email updates.</span></p><ul><li id="follow-author-article_footer-dmcyOmF1dGhvclByb2ZpbGU6Mjc0"><span aria-expanded="false" aria-haspopup="true" role="button" tabindex="0"><span><span><svg width="9" height="9" viewbox="0 0 9 9" fill="none" xmlns="http://www.w3.org/2000/svg" aria-label="Follow"><path d="M5 0H4V4H0V5H4V9H5V5H9V4H5V0Z"></path></svg></span><span>David Pierce</span></span></span></li><li><li><li><li></ul></div></div></div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/the-sony-aibo-was-a-robot-revolution-dressed-like-a-dog-77182</guid> <pubDate>Sun, 15 Nov 2026 14:57:30 +0700</pubDate> <author><![CDATA[support@mycoding.id (David Pierce)]]></author> </item> <item> <title><![CDATA[Things That Turbo Pascal Is Smaller Than (2011)]]></title> <description><![CDATA[Things That Turbo Pascal Is Smaller Than (2011). Comments]]></description> <media:thumbnail url="https://en.mycoding.id/site/assets/img/og-image.png"/> <category><![CDATA[Technology]]></category> <link>https://en.mycoding.id/things-that-turbo-pascal-is-smaller-than-2011-77192</link> <content:encoded><![CDATA[<div id="c1"><p>Turbo Pascal 3 for MS-DOS was released in September 1986. Being type 3, there were lesser releases previous to it and flashier ones after, but 3 was a firm depiction of the Turbo Pascal experience: a complete Pascal compiler, including extensions that it made it applicable for business use, tightly unified alongside an editor. And the entire item was lightning fast, orders of dimension faster at construction projects than Microsoft's compilers.</p><p>The complete Turbo Pascal 3.02 executable--the compiler and IDE--was 39,731 bytes. How does that stack up in 2011 terms? Here are several things that Turbo Pascal is smaller than, as of October 30, 2011:</p><p>The minified type of jquery 1.6 (90,518 bytes).</p><p>The yahoo.com residence leaf (219,583 bytes).</p><p>The depiction of the pale iPhone 4S at apple.com (190,157 bytes).</p><p>zlib.h in the Mac OS X Lion SDK (80,504 bytes).</p><p>The contact command under OS X Lion (44,016 bytes).</p><p>Various vim quick citation cards as PDFs. (<a href="https://en.mycoding.id/category/technology" target="_blank">This one</a> is 47,508 bytes.)</p><p>The compiled code for the Erlang R14B02 parser (erl_parse.beam, 286,324 bytes).</p><p>The Wikipedia leaf for C++ (214,251 bytes).</p><p>(If you liked this, you power akin <a href="https://en.mycoding.id/category/technology" target="_blank">A Personal History of Compilation Speed</a>.)</p><p><a href="https://en.mycoding.id/category/technology" target="_blank" id="perm">permalink</a> <i>October 30, 2011</i></p><ul><li><a href="https://en.mycoding.id/category/technology" target="_blank">Starting in the Middle</a></li><li><a href="https://en.mycoding.id/category/technology" target="_blank">Papers from the Lost Culture of Array Languages</a></li><li><a href="https://en.mycoding.id/category/technology" target="_blank">The Revolution is Personal</a></li><li><a href="https://en.mycoding.id/category/technology" target="_blank">Optimization on a Galactic Scale</a></li><li><a href="https://en.mycoding.id/category/technology" target="_blank">Greetings from the Bottom of the Benchmarks</a></li></ul></div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/things-that-turbo-pascal-is-smaller-than-2011-77192</guid> <pubDate>Sun, 15 Nov 2026 14:44:16 +0700</pubDate> <author><![CDATA[support@mycoding.id (Adi Gunawan)]]></author> </item> <item> <title><![CDATA[The MacBook Pro's OLED PWM Problem: When a 'Better' Display Becomes Worse to Use]]></title> <description><![CDATA[The MacBook Pro's OLED PWM Problem: When a 'Better' Display Becomes Worse to Use. Comments]]></description> <media:thumbnail url="https://macanorak.com/content/images/2026/11/PROLED-Display-XDR-1.png"/> <category><![CDATA[Technology]]></category> <link>https://en.mycoding.id/the-macbook-pro-s-oled-pwm-problem-when-a-better-display-becomes-worse-to-use-77193</link> <content:encoded><![CDATA[<p>Apple has brought <a href="https://en.mycoding.id/category/technology" target="_blank">tandem OLED innovation to the MacBook Pro</a>, joining the Mini-LED display that has served the range since 2021 (and which Apple <a href="https://en.mycoding.id/category/technology" target="_blank">continues to sell</a>). Apple calls this a &ldquo;PROLED XDR Display&rdquo;. It features individually illuminated pixels, efficiently infinite contrast, wonderfully profound blacks and the kind of depiction norm that Mini-LED, for all its significant strengths, can&rsquo;t completely equal. Peak HDR brightness has increased, and Apple has introduced a new display motor designed to oversee colour, luminance and pixel use alongside bonzer precision. And of course, in a archetypal for any Apple notebook display, it has touch-input capability.</p> <p>When Apple unveiled the new MacBook Pro, the first reply to the display was overwhelmingly positive.</p> <p>But there&rsquo;s a burgeoning conversation taking location inside the Mac community that deserves attention. On &#8288;<a href="https://en.mycoding.id/category/technology" target="_blank">MacRumors forums</a>, a thread titled <em>RIP MacBook Pro PROLED for PWM Sensitive Eyes - apple cmon!</em> has attracted group describing remarkably akin experiences. Discussions are additionally appearing throughout Reddit, alongside owners <a href="https://en.mycoding.id/category/technology" target="_blank">describing discomfort shortly following using their new machines</a>. Some study headaches; others depict a different feeling of eye fatigue, dizziness, nausea, or an inability to concentration on the display for extended periods. <a href="https://en.mycoding.id/category/technology" target="_blank">Some have already returned the device</a>.</p> <p>Mike Wuerthele, of AppleInsider, published an part on 10 November titled &#8288;<a href="https://en.mycoding.id/category/technology" target="_blank">If you&rsquo;re getting dizzy using PROLED MacBook Pro, you are not alone</a>, examining these reports.</p> <p>The following caveat likely doesn't need to be made, but I volition create it anyway: these are idiosyncratic experiences, not evidence that the new MacBook Pro is causing problems for everyone. Plenty of owners are apt absolutely satisfied alongside (and dazzled by) their new displays. But that isn&rsquo;t really the point. For the group experiencing these symptoms, the fact that person alternatively can happily expend multiple hours staring at the identical display isn&rsquo;t particularly reassuring, particularly stated Apple's long-standing dedication to accessibility. A display can be technically spectacular and motionless be a spectacularly mediocre display for a subset of people.</p> <h4 id="pwm"><strong>PWM</strong></h4> <p>Much of the conversation centres on Pulse Width Modulation, or PWM. It&rsquo;s a method used to authority display brightness by quickly switching or modulating the ray output. On an OLED display, the pixels themselves create the light, fairly than relying on a distinct backlight. By varying the way that ray output is modulated, the display can authority its apparent brightness.</p> <p>At sufficiently elevated frequencies, this flickering isn&rsquo;t item most group can consciously detect. But several group are particularly delicate to this benevolent of modulation and can cognition eye strain, headaches and another forms of discomfort whenever looking at displays that use it.</p> <p>This is a beautiful fine display of PWM on an M4 iPad Pro, which additionally has a tandem OLED display:</p> <figure><p class="embed-responsive embed-responsive-16by9 post-embed"><iframe width="560" height="315" aria-label="Preview" class="embed-responsive-item" src="https://www.youtube.com/embed/zWc30I27M3o?feature=oembed&amp;autoplay=1&amp;mute=1&amp;playsinline=1&amp;rel=0" title="YouTube video player" frameborder="0" allow="autoplay; accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe></p></figure><p>Neither PWM nor the conversation surrounding it is new. It&rsquo;s been a talking item for years, particularly about OLED displays, and item manufacturers have increasingly had to regard as display innovation has evolved (including Apple &mdash; additional on this later).</p> <p><a href="https://en.mycoding.id/category/technology" target="_blank">Notebookcheck</a>, which conducted lab evaluation of the 14-inch MacBook Pro alongside PROLED XDR Display, measured PWM at 240Hz. This is its worst-case reading, alongside flickering detected at brightness settings of 99% and below.</p> <p>To put that into perspective, the M4 Pro MacBook Pro alongside Mini-LED display measured 14,880Hz in Notebookcheck&rsquo;s testing. At 240Hz, the brightness assortment repeats 240 times per second; at 14,880Hz, it repeats nearly 15,000 times per second. That&rsquo;s one hell of a difference. The reported modulation frequence of the new MacBook Pro&rsquo;s OLED panel is approximately 62 times lesser than that measured on the Mini-LED model.</p> <p>Frequency isn&rsquo;t the lone consideration, though. The degree to which brightness fluctuates during all sequence additionally matters, as do the characteristics of the display&rsquo;s dimming system. We shouldn&rsquo;t assume that these figures translate immediately into how comfortable person volition discover a particular display. There&rsquo;s no simple, one-size-fits-all equation for predicting who volition cognition problems.</p> <p>It&rsquo;s additionally value avoiding the enticement to treat PWM as a worldwide clarification for all headache or case of eye strain. Screen discomfort can have many causes. Some users&rsquo; discomfort may have nothing to do alongside PWM.</p> <h4 id="wait-isn%E2%80%99t-oled-supposed-to-be-better"><strong>Wait, isn&rsquo;t OLED expected to be better?</strong></h4> <p>The MacBook Pro alongside PROLED XDR Display is emphatically not an cheap laptop alongside a prosperity screen. This is Apple&rsquo;s <em>flagship expert notebook</em>. The PROLED XDR configuration represents a significant investment, and the display is arguably its most crucial marketing point. Apple has spent years refining its displays, and there&rsquo;s a sensible anticipation that moving to OLED should portray several kind of betterment in nearly all meaningful respect, particularly stated the savagely elevated cost tag of these new machines.</p> <p>PWM isn&rsquo;t exclusive to OLED. LCD displays can use it, too, including Mini-LED displays. The difference is that OLED panels frequently use PWM as part of their brightness-control behaviour, and manufacturers have significant elasticity in how they execute it. Some OLED displays use comparatively high-frequency modulation; others use a blend of PWM and DC (Direct Current) dimming to oversee brightness.</p> <p>DC dimming mostly involves reducing the current supplied to the display fairly than relying entirely on quickly switching the ray output on and off. It&rsquo;s frequently considered preferable for group who are delicate to PWM. But the existence of DC dimming doesn&rsquo;t necessarily average PWM has been eliminated. The new MacBook Pro appears to use a blend of PWM and DC dimming. Notebookcheck notes that DC dimming is current throughout brightness levels, but that the panel motionless exhibits 240Hz flickering.</p> <h4 id="the-people-who-didn%E2%80%99t-have-a-problem-before"><strong>The group who didn&rsquo;t have a issue before</strong></h4> <p>One of the problems alongside PWM discussions is that they lean to quickly rotate into extremely polarised. Someone says they cognition headaches, and person alternatively responds that they&rsquo;ve been using OLED displays for years without any issues. Two things can be true at once.</p> <p>The most crucial aspect of this discussion, to me, is the figure of group insisting they had <em>no difficulty using OLED devices previous to using this new MacBook.</em> This matters since PWM sensitivity is frequently discussed as although it&rsquo;s a straightforward, binary difference between group who can tolerate OLED and group who can&rsquo;t. That binary difference doesn&rsquo;t exist. Sensitivity exists alongside a spectrum.</p> <p>Different displays use distinct modulation frequencies, amplitudes and dimming characteristics. Two OLED panels can behave differently, equal if they&rsquo;re according to broadly akin technology. Someone who has been happily using an OLED iPhone Pro for multiple years power discover a distinct OLED display uncomfortable; person who has informed headaches alongside one OLED laptop power have entirely no issue alongside another. And there is a difference between OLED innovation itself and the particular implementation of OLED in this particular MacBook.</p> <p>To provision an example, person on the <a href="https://en.mycoding.id/category/technology" target="_blank">PWM_Sensitive</a> subreddit posted this concerning their cognition alongside the new MacBook Pro (shared alongside their permission):</p> <blockquote> <p><em>I was really amazed I was affected by this. Have been looking at assorted OLED screens for years and did not anticipate to have issues.</em></p> <p><em>Then I got my PROLED 16 inch final week and during the archetypal few hours I idea I fair had a bad day. Now it&rsquo;s so obvious. As shortly as I appearance at the display I get lightheaded, I awareness akin I am floating distant &hellip; and whenever I afterward get up and commencement walking, I awareness dizzy for akin an hr or two, afterward it gets improved again. But as shortly as I appearance at the display for additional than a minute, the dizziness comes back.</em></p> <p><em>This is archetypal device in 20 years as an Apple client that I come back exactly anything. Too bad, since the laptop is amazing in all aspects otherwise, I was in affection alongside it the instant I took it out of the box (coming from an M1 Pro 16 inch). It would have been my regular controller for the next 5 years. So, now I am waiting for the M7 type and anticipation they volition discover a way to mitigate this issue. I am returning the MacBook tomorrow dawn following small than a week of using it.</em></p> </blockquote> <p>Apple has built up a loyal spectators of expert Mac users who expend enormous amounts of prosperity on the machines, and enormous amounts of period in forefront of their displays. Photographers, designers, developers, writers and video editors can expend complete operating days looking at a MacBook display (I do). So ideate upgrading from an M4 or M5 Pro MacBook Pro alongside Mini-LED, expecting a additional beautiful type of item you already love, lone to detect that you can&rsquo;t comfortably use the new device for additional than a abbreviated period. Good client cognition is normally Apple's directing North Star. Bad client cognition is normally Apple&rsquo;s Kryptonite.</p> <h4 id="has-apple-compromised-on-the-wrong-thing"><strong>Has Apple compromised on the incorrect thing?</strong></h4> <p>There are plentifulness of engineering considerations engaged in evolving a tandem OLED panel for a laptop &mdash; brightness, power consumption, thermal management, longevity, color accuracy and uniformity all have to be balanced.</p> <p>Apple's displays have continually engaged trade-offs to one flat or another. Take the Mini-LED display &mdash; during it's really outstanding in many respects, it notoriously suffers from blooming (an inherent limitation of Mini-LED local dimming) and has additionally typically had <a href="https://en.mycoding.id/category/technology" target="_blank">poor pixel reply times</a>. These are compromises Apple has had to obtain in command to provision the particular blend of brightness, difference and HDR achievement that makes the Mini-LED display so impressive.</p> <p>Apple has made several aspiring claims concerning the new PROLED XDR display, particularly about its brightness and the intellect of its display engine. We don&rsquo;t cognize why Apple has ended up alongside this particular dimming behaviour, or what trade-offs were engaged in achieving it. But I think it&rsquo;s sensible to ask whether display comfort received adequate deliberation in the process. The new MacBook Pro produces bonzer blacks, stunning HDR highlights and exceptionally exact colours, and so on. But none of that matters particularly much if the individual sitting in forefront of it develops a headache, or feels akin they desire to throw up whenever they're using it.</p> <h4 id="when-the-majority-experience-isn%E2%80%99t-everyone%E2%80%99s-experience"><strong>When the bulk cognition isn&rsquo;t everyone&rsquo;s experience</strong></h4> <p>People experiencing unusual, poorly understood or comparatively rare health problems have frequently established themselves struggling to be heard &mdash; they have to explain their symptoms repeatedly, defend their experiences and convince others that their difficulties are genuine. I don't personally cognize anybody who suffers from PWM sensitivity, and I value that it can be difficult for person without these experiences to comprehend how item so average could rotate into a origin of bodily discomfort for another person. But individual cognition shouldn't be a prerequisite for compassion. We don&rsquo;t need to cognition person else&rsquo;s symptoms to acknowledge that they are struggling.</p> <p>Stories akin this lean to exterior lone formerly an ill-defined and arbitrary threshold of group complaining is crossed. But equal if item lone affects a small minority, doesn&rsquo;t that smaller group matter? Apple certainly appears to attention concerning the smaller group &mdash; they put an bonzer amount of activity into accessibility features, many of which are incredibly crucial to comparatively small groups of users. It&rsquo;s among the things which sets Apple apart. Features specified as VoiceOver, AssistiveTouch and another accessibility technologies be since Apple recognises that distinct group engage alongside innovation in distinct ways. Why shouldn&rsquo;t display comfort obtain akin consideration? It doesn&rsquo;t average Apple must forsake OLED, or colony the cognition for everyone. It method offering alternatives anywhere technically feasible.</p> <p>Apple earlier introduced a &lsquo;Display Pulse Smoothing option&rsquo; on supported iPhone models. It&rsquo;s an accessibility environment which can be toggled in the Settings app. Apple describes Display Pulse Smoothing as a way to create a smoother dimming cognition at low brightness on supported iPhones. Its <a href="https://en.mycoding.id/category/technology" target="_blank">support documentation</a> cautions that disabling PWM may power low-brightness display achievement under certain conditions. It took Apple a lengthy period to contain this characteristic by the way (it arrived nearly 8 years following the iPhone X was released). Incidentally, Apple has additionally included Display Pulse Smoothing on the new <a href="https://en.mycoding.id/category/technology" target="_blank">iPad mini alongside OLED display</a>.</p> <p>Could item akin be offered on the Mac? We shouldn&rsquo;t assume that a application characteristic would entirely determine the issue. Changing the way the display is driven could current another compromises, and Apple would need to execute it carefully. But equal a flat of person authority would be gratefully received.</p> <p>For now, anyone concerned concerning PWM sensitivity have to be careful concerning purchasing the new device without trying it first. A quick demo in an Apple Store isn&rsquo;t necessarily delegate of spending an complete operating day in forefront of the display. Don&rsquo;t assume that being comfortable alongside an OLED iPhone method you&rsquo;ll be comfortable alongside this MacBook. If you cognition discomfort, don&rsquo;t assume that expanding brightness or changing color settings volition necessarily determine it.</p> <p>The existing Mini-LED MacBook Pro thankfully remains a viable substitute for group who have established that display comfortable. But the Mini-LED MacBook Pro power not remain about for much longer. Apple looks to be <a href="https://en.mycoding.id/category/technology" target="_blank">transitioning to OLED throughout their devices</a> &mdash; this may have consequences unless Apple changes the way they're implementing this particular technology.</p> <h4 id="the-best-macbook-display-isn%E2%80%99t-necessarily-the-best-display-for-everyone"><strong>The finest MacBook display isn&rsquo;t necessarily the finest display for everyone</strong></h4> <p>We sometimes get so caught up in the chase of improved details that we ignore <em>what</em> these devices are really for. The most crucial part of a display is that it's item you <em>sit in forefront of for hours at a time.</em></p> <p>Displays to my intellect are akin mattresses &mdash; they're not item you should colony on, unless financial constraints petition it. I've frequently been puzzled why group baulk at paying elevated prices for external monitors whenever they&rsquo;re going to be using them for multiple hours a day. Similarly, I&rsquo;ve frequently wondered why group baulk at paying a lot of prosperity for a mattress, whenever they&rsquo;re going to be spending approximately a third of their lives lying on one. A decent mattress is likely item you should allocate in. The specs of the mattress matter, of course, and idiosyncratic comfort matters enormously. The identical item is true alongside a display, which is why I have no matter having spent the prosperity I did on my Studio Display XDR.</p> <p>And this is what's so problematic alongside the MacBook Pro alongside PROLED XDR display. According to many reviewers this is among the <a href="https://en.mycoding.id/category/technology" target="_blank">best displays they've always seen</a>, and it has a cost to match. Starting at &pound;2899 for the 14-inch type (&pound;3399 for the 16-inch), <em>boy is it expensive</em>. If the innovation liable for making the PROLED XDR Display appearance spectacular additionally makes it uncomfortable for several group to use, afterward we need to ask if that trade-off is value it.</p> <p>I doubtful most group purchasing the new MacBook Pro volition never think concerning PWM. They&rsquo;ll open the lid, regard the display and get on alongside their lives. But for the group who have discovered that they can no longer comfortably use the latest MacBook Pro, this is a <em>really big deal.</em></p> <p>I&rsquo;d akin to see Apple explain why the new display behaves so alternatively from the Mini-LED panel used in another MacBook Pro models, and whether there&rsquo;s any possible of improving its dimming characteristics through software. Perhaps there&rsquo;s a absolutely sensible engineering explanation. Perhaps there are restrictions inherent in this particular panel.</p> <p>I&rsquo;d be curious to comprehend from anyone who has informed this first-hand, particularly those who have earlier used Apple&rsquo;s OLED iPhones or iPads without difficulty, but are now struggling alongside the new MacBook Pro.</p> <p>Get in contact at <a href="https://en.mycoding.id/category/technology" target="_blank">hello@macanorak.com</a></p> <p>You can additionally present feedback to Apple here: <a href="https://en.mycoding.id/category/technology" target="_blank">https://www.apple.com/feedback/macbook-pro/</a></p>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/the-macbook-pro-s-oled-pwm-problem-when-a-better-display-becomes-worse-to-use-77193</guid> <pubDate>Sun, 15 Nov 2026 14:15:22 +0700</pubDate> <author><![CDATA[support@mycoding.id (Adi Gunawan)]]></author> </item> <item> <title><![CDATA[10 challenges facing the real estate industry in 2026, according to young leaders]]></title> <description><![CDATA[10 challenges facing the real estate industry in 2026, according to young leaders. 10 of Business Insider's rising stars of genuine asset portion what they accept are the biggest challenges facing the genuine asset industry in 2026.]]></description> <media:thumbnail url="https://i.insider.com/6af780ba0cc3ffba8779d02a?width=1200&amp;format=jpeg"/> <category><![CDATA[Business]]></category> <link>https://en.mycoding.id/10-challenges-facing-the-real-estate-industry-in-2026-according-to-young-leaders-77184</link> <content:encoded><![CDATA[<div e-name="post-hero" data-component-type="post-hero" data-load-strategy="exclude"> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject"> <div> <p><img src="https://i.insider.com/6af78064cbe212047073558e?width=2000&amp;format=jpeg&amp;auto=webp" srcset="https://i.insider.com/6af78064cbe212047073558e?width=400&amp;format=jpeg&amp;auto=webp 400w, https://i.insider.com/6af78064cbe212047073558e?width=500&amp;format=jpeg&amp;auto=webp 500w, https://i.insider.com/6af78064cbe212047073558e?width=700&amp;format=jpeg&amp;auto=webp 700w, https://i.insider.com/6af78064cbe212047073558e?width=1000&amp;format=jpeg&amp;auto=webp 1000w, https://i.insider.com/6af78064cbe212047073558e?width=1300&amp;format=jpeg&amp;auto=webp 1300w, https://i.insider.com/6af78064cbe212047073558e?width=2000&amp;format=jpeg&amp;auto=webp 2000w" sizes=" 1280px) 900px" alt="Three headshots of individuals juxtaposed against an animated backdrop." decoding="sync"> </p></div> <span> <span> <svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewbox="0 0 24 24"> <path fill="currentColor" fill-rule="evenodd" d="m4.56 18.5 7.486-7.72 7.394 7.626 2.56-2.64L12.046 5.5 2 15.86l2.56 2.64Z"></path> </svg> <figcaption e-name="image-caption"> <span>Business Insider's 2026 Rising Stars of genuine asset portion the biggest challenges facing genuine estate.</span> <span> <span e-name="image-source" itemprop="creditText">Alyssa Powell</span> </span> </figcaption> </span> </span> </figure> </div> <div data-component-type="image-modal" data-load-strategy="lazy"> <p>The genuine asset market has felt item but normal in 2026. <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">High involvement rates</a> and residence prices have <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">turned off many would-be buyers</a>, equal in several of the country's <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">previously hottest markets</a>.</p><p>New taxes affecting <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">high-end properties</a>, coupled alongside broader <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">economic uncertainty</a>, have additionally prompted affluent buyers to inquiry whether it motionless makes awareness to buy or allocate in genuine asset in their cities &mdash; or <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">look elsewhere</a>, <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">even overseas</a>.</p><p>These challenges are apt to transport into next year. But Business Insider's 2026 <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">Rising Stars of Real Estate</a> say it's not all destiny and gloom.</p><p>Some rising stars see a bigger function for agents as buyers and sellers try to create awareness of a complex market. Others are experimenting alongside AI to oversee and develop their businesses. And in several markets, akin Miami and New York, they say petition is picking up again.</p><p>Here are what 10 of this year's rising stars say are the biggest issues facing genuine asset and how they think the industry volition adapt.</p> <div data-component-type="post-slides" data-load-strategy="lazy"> <div> <p> <h2>Agents can't get by fair knowing the basics.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/png" data-srcs='//i.insider.com/6aed1648c54bfafa65abf8d2&quot;:{&quot;contentType&quot;:&quot;image/png&quot;,&quot;aspectRatioW&quot;:1500,&quot;aspectRatioH&quot;:1500}}' alt=" Joey Ben-Zvi, genuine asset agent, Ben-Zvi Group"> </p></div> <span> <figcaption e-name="image-caption"> <span>Joey Ben-Zvi, genuine asset agent, Ben-Zvi Group.</span> <span> <span e-name="image-source" itemprop="creditText">Photo Courtesy of Joey Ben-Zvi</span> </span> </figcaption> </span> </figure> <div> <p>Joey Ben-Zvi, originator of the real-estate resolute Ben-Zvi Group, says innovation has made data and data additional accessible, foremost several buyers and sellers to accept they can navigate a transaction on their own, without an agent.</p><p>"Consumers have additional admission to data than always before, which is ultimately a fine thing, but admission to data isn't the identical as being in the trenches on a regular basis and staying deeply connected to the merchandise and consumers, alongside alongside having the experience, judgment, and relationships necessary to navigate a complex transaction," Ben-Zvi told Business Insider.</p><p>While Ben-Zvi believes agents remain essential to the homebuying and marketing experience, he says knowing the basics is no longer enough.</p><p>"I think the industry is going through a necessary evolution," Ben-Zvi said. "Agents can no longer depend solely on admission to listings or information. They have to display genuine value through their expertise, creativity, negotiation skills, and capability to resolve problems."</p> </div> </div> <div> <p> <h2>Data volition rotate into equal additional critical.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/png" data-srcs='//i.insider.com/6aed16489270da3cc6668aa0&quot;:{&quot;contentType&quot;:&quot;image/png&quot;,&quot;aspectRatioW&quot;:1500,&quot;aspectRatioH&quot;:1500}}' alt=" Saharsh Chordia"> </p></div> <span> <figcaption e-name="image-caption"> <span>Saharsh Chordia, vice chairman of scheme and analytics, Cortland.</span> <span> <span e-name="image-source" itemprop="creditText">Photo Courtesy of Saharsh Chordia</span> </span> </figcaption> </span> </figure> <div> <p>Saharsh Chordia, the 33-year-old vice chairman of scheme and analytics at Cortland, stated that data volition rotate into much additional important. The challenge, he said, is turning the huge amount of data generated by bodily resources into actionable insight.</p><p>"Real asset is basically a endeavor built about bodily assets, yet much of the industry's data remains fragmented and underutilized," Chordia said. "AI volition perform an crucial role, but the organizations that create a meaningful advantage volition be the ones that can efficiently collect, structure, and study from data generated throughout their portfolios."</p> </div> </div> <div> <p> <h2>Data centers could hurt several types of properties.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/png" data-srcs='//i.insider.com/6aed16ae308912814ed717eb&quot;:{&quot;contentType&quot;:&quot;image/png&quot;,&quot;aspectRatioW&quot;:1500,&quot;aspectRatioH&quot;:1500}}' alt=" Michael Dillon"> </p></div> <span> <figcaption e-name="image-caption"> <span>Michael Dillon, vice president, Prime Finance.</span> <span> <span e-name="image-source" itemprop="creditText">Photo Courtesy of Michael Dillon</span> </span> </figcaption> </span> </figure> <div> <p>Michael Dillon, a 31-year-old vice chairman at Prime Finance, stated data centers and power-adjacent properties could run changes in what traditional asset types advantage and suffer. He noted that it's unclear how an AI-driven economics could stance certain asset types complete the lengthy term.</p><p>"For a lender, that method underwriting not fair the construction today, but folding those questions into our perspective of durable petition for that particular asset complete a ten-year horizon," Dillon said.</p> </div> </div> <div> <p> <h2>AI is "Our biggest difficulty but additionally our newest advantage."</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/png" data-srcs='//i.insider.com/6aed1648308912814ed717e0&quot;:{&quot;contentType&quot;:&quot;image/png&quot;,&quot;aspectRatioW&quot;:1500,&quot;aspectRatioH&quot;:1500}}' alt=" Alex Dobosh, affiliate director at HKS Real Estate Advisors"> </p></div> <span> <figcaption e-name="image-caption"> <span>Alex Dobosh, affiliate director, HKS Real Estate Advisors.</span> <span> <span e-name="image-source" itemprop="creditText">Photo Courtesy of Alex Dobosh</span> </span> </figcaption> </span> </figure> <div> <p>Alex Dobosh, an affiliate director at HKS Real Estate Advisors, sees synthetic intellect as genuine estate's biggest challenge, and possibly one of its top opportunities.</p><p>It's not difficult to see why. Americans are turning to platforms akin ChatGPT for real-estate advice, <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">home-construction</a> guidance, and assistance alongside the purchasing and marketing process.</p><p>Business Insider has equal written concerning homebuyers who have successfully used <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">AI as their real-estate agent</a>.</p><p>Dobosh doesn't see AI as a logic for agents to panic. He says it can be a mighty tool for professionals consenting to clasp it. His resolute is already doing so.</p><p>"Our biggest difficulty is additionally our newest advantage, which is AI," Dobosh said. "It can be overwhelming to see how it's changing our world. We are leveraging it all sole day to origin endeavor and assistance our clients, but additionally since we refuse to autumn behind."</p> </div> </div> <div> <p> <h2>The prosperity gap volition continue to grow.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/png" data-srcs='//i.insider.com/6aed164a9270da3cc6668aa7&quot;:{&quot;contentType&quot;:&quot;image/png&quot;,&quot;aspectRatioW&quot;:1500,&quot;aspectRatioH&quot;:1500}}' alt=" Landy Liu"> </p></div> <span> <figcaption e-name="image-caption"> <span>Landy Liu, founder, Foyer.</span> <span> <span e-name="image-source" itemprop="creditText">Photo Courtesy of Landy Liu</span> </span> </figcaption> </span> </figure> <div> <p>Landy Liu, the 33-year-old originator of Foyer, a "401(k) for homeownership," expects 2027 to be another twelvemonth of affordability concerns.</p><p>"We are facing a structural compression on consumers that is foremost to a theatrical development of the homebuying experience," Liu said.</p><p>Liu, who founded Foyer to provision first-time homebuyers additional assistance throughout the process, stated a deficiency of affordable homes has widened the prosperity gap.</p><p>"The industry needs to basically alter to improved assistance younger Americans and another consumers at a lesser cost," he said.</p> </div> </div> <div> <p> <h2>There's a accommodation deficit and an affordability crisis.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/png" data-srcs='//i.insider.com/6aed16499270da3cc6668aa4&quot;:{&quot;contentType&quot;:&quot;image/png&quot;,&quot;aspectRatioW&quot;:1500,&quot;aspectRatioH&quot;:1500}}' alt=" Katherine McGrath, Strategic Director, Real Estate at Kasirer"> </p></div> <span> <figcaption e-name="image-caption"> <span>Katherine McGrath, s trategic director, genuine Estate, Kasirer.</span> <span> <span e-name="image-source" itemprop="creditText">Photo Courtesy of Katherine McGrath</span> </span> </figcaption> </span> </figure> <div> <p>Katherine McGrath, whose activity at lobbying resolute Kasirer deals alongside district use projects, believes the deficiency of affordable homes &mdash; <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">and of homes in general</a> &mdash; is a important challenge, particularly in her city of New York.</p><p>"Demand continues to increase throughout all five boroughs, but land, construction, labor, and financing expenses create it extremely difficult to build projects alongside a meaningful portion of affordable units," McGrath told Business Insider.</p><p>McGrath stated that <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">mixed-income development</a> is among the most productive ways to combat the two provision and affordability in one blow. But developers need aid from their local governments.</p><p>"Developers can't near that gap alone," she said. "They need incentive programs that indicate real-world costs, a faster approvals process, and continued zoning reform. If we desire additional affordable homes, we have to create it feasible to build them."</p> </div> </div> <div> <p> <h2>The deficiency of norm buildings is slowing downward transaction velocity.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/png" data-srcs='//i.insider.com/6aed1649c54bfafa65abf8d3&quot;:{&quot;contentType&quot;:&quot;image/png&quot;,&quot;aspectRatioW&quot;:1500,&quot;aspectRatioH&quot;:1500}}' alt=" Cordelia Meserow, Harbor Group International"> </p></div> <span> <figcaption e-name="image-caption"> <span>Cordelia Meserow, vice chairman of asset management, Harbor Group International.</span> <span> <span e-name="image-source" itemprop="creditText">Photo Courtesy of Nigel Alfred</span> </span> </figcaption> </span> </figure> <div> <p>Cordelia Meserow, a vice chairman of asset administration at genuine asset funding and administration resolute Harbor Group International, stated that during <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">interest charge volatility</a> has had an consequence on pricing business buildings in Manhattan, the city's leasing market has never been stronger. The issue is that there's not adequate fine agency area to go around.</p><p>"There seems to be no ceiling for what tenants volition expend on desirable space," Meserow told Business Insider. "The biggest difficulty in New York is that the market is really operating out of Class A space."</p><p>Because of the deficit of <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">Class A buildings</a>, transaction speed volition be constrained.</p><p>"Velocity volition certainly dilatory downward since of the deficiency of availability," she said.</p> </div> </div> <div> <p> <h2>Real asset agents need to regain clients' rely and display their worth.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/png" data-srcs='//i.insider.com/6aed1648c54bfafa65abf8d1&quot;:{&quot;contentType&quot;:&quot;image/png&quot;,&quot;aspectRatioW&quot;:1500,&quot;aspectRatioH&quot;:1500}}' alt=" Steven Rivera"> </p></div> <span> <figcaption e-name="image-caption"> <span>Steven Rivera, squad lead, Team Rivera at Keller Williams Points North.</span> <span> <span e-name="image-source" itemprop="creditText">Photo Courtesy of Steven Rivera</span> </span> </figcaption> </span> </figure> <div> <p>After the <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">structural changes to commissions</a>, genuine asset delegate Steven Rivera thinks agents need to win rear their clients' rely to succeed.</p><p>"Since the commission settlement, buyers sign an accord alongside their delegate before touring a residence and rightly ask what they are paying for," Rivera told Business Insider. "That is fit for the industry, but it raises the bar: agents now have to display their value, not merely assertion it."</p><p>Rivera stated he gives his clients a elevated flat of assistance to instill rely in the final result.</p><p>"If you cannot explain what you do for the fee, the market volition ask," he said.</p> </div> </div> <div> <p> <h2>The genuine asset industry needs to commencement catering to younger generation.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/png" data-srcs='//i.insider.com/6aed16489270da3cc6668aa2&quot;:{&quot;contentType&quot;:&quot;image/png&quot;,&quot;aspectRatioW&quot;:1500,&quot;aspectRatioH&quot;:1500}}' alt=" Selin Sonmez, Cofounder and CEO, Snag"> </p></div> <span> <figcaption e-name="image-caption"> <span>Selin Sonmez, cofounder and CEO of Snag.</span> <span> <span e-name="image-source" itemprop="creditText">Photo Courtesy of Selin Sonmez</span> </span> </figcaption> </span> </figure> <div> <p>"Real Estate is built for a earth that no longer exists," cofounder of Snag, Selin Sonmez, told Business Insider. "Its biggest difficulty is adapting to the new financial actuality of the rising generation."</p><p>Sonmez's startup Snag is geared toward <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">younger renters looking to sublet</a> their spots using a social-media-like method to listing. She's recognized that consumers are getting younger, and thinks the remainder of the industry should try to cater to the next generation.</p><p>"Young group are expected to rent for 20 years, have multiple jobs, and not be permanent," she said. "How is genuine asset adapting to this new reality? Most industries shifted to bite-size, approachable, scalable units, from media to investments," she said.</p> </div> </div> <div> <p> <h2>Real asset professionals need to modify their use of district to encounter people's needs.</h2> </p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-content-type="image/png" data-srcs='//i.insider.com/6aed1648308912814ed717df&quot;:{&quot;contentType&quot;:&quot;image/png&quot;,&quot;aspectRatioW&quot;:1500,&quot;aspectRatioH&quot;:1500}}' alt=" James Williams Jr."> </p></div> <span> <figcaption e-name="image-caption"> <span>James Williams Jr, shareholder, Gunster.</span> <span> <span e-name="image-source" itemprop="creditText">Photo Courtesy of James Williams Jr.</span> </span> </figcaption> </span> </figure> <div> <p>James Williams Jr., a shareholder in the genuine asset collection at the law resolute Gunster, believes genuine asset professionals need to modify their plans to encounter the changing needs for housing, employment, and services.</p><p>"That method looking at whether a property's existing use motionless serves the community's needs &mdash; and, if not, what it could become," Williams Jr. told Business Insider.</p><p>But he stated the activity doesn't halt formerly you acknowledge a additional optimal use for a building; there are a lot of hurdles &mdash; <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">like zoning</a> or backing &mdash; that need to be conquer to create it happen.</p><p>"Existing zoning may not authorize the projected development, infrastructure may need important upgrades, and the disbursal of those improvements may create an alternatively promising project financially unworkable," he said. "The difficulty is addressing those obstacles during ensuring that the betterment fits its surroundings and delivers meaningful benefits to the community."</p> </div> </div> </div> </div> <h2>Read next</h2> <div data-recommendations-content data-track-click-shared='&quot;Read next&quot;}'> <div e-name="read-next-list" data-recommendations-list> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> </div> </div> <a e-name="read-next-image-link" data-recommendations-image-link data-recommendations-link tabindex="-1"> <p><img alt=""> </p> </a> <h3> <a e-name="read-next-title-link" data-recommendations-link data-recommendations-title></a> </h3>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/10-challenges-facing-the-real-estate-industry-in-2026-according-to-young-leaders-77184</guid> <pubDate>Sun, 15 Nov 2026 14:08:01 +0700</pubDate> <author><![CDATA[support@mycoding.id (Alcynna Lloyd,Jordan Pandy,Jennifer Sor,Noah Sheidlower)]]></author> </item> <item> <title><![CDATA[An overview of Jensen Huang’s messy days]]></title> <description><![CDATA[An overview of Jensen Huang’s messy days. This is The Stepback, a weekly newsletter breaking downward one essential narrative from the tech world. For additional on Jensen Huang's resignation and the fallout from Nvi...]]></description> <media:thumbnail url="https://platform.theverge.com/wp-content/uploads/sites/2/2026/11/gettyimages-2301620664.jpg?quality=90&amp;strip=all&amp;crop=0%2C14.751764993408%2C100%2C70.496470013184&amp;w=1200"/> <category><![CDATA[Technology]]></category> <link>https://en.mycoding.id/an-overview-of-jensen-huang-s-messy-days-77183</link> <content:encoded><![CDATA[<div id="zephr-anchor"><p><em>This is</em> The Stepback<em>, a weekly newsletter breaking downward one essential narrative from the tech world. For additional on Jensen Huang&rsquo;s resignation and the fallout from Nvidia&rsquo;s guidance crisis, <a href="https://en.mycoding.id/category/technology" target="_blank">follow Terrence O&rsquo;Brien</a>.</em> The Stepback <em>arrives in our subscribers&rsquo; inboxes on Sunday at 8AM ET. Opt in for</em> The Stepback <a href="https://en.mycoding.id/category/technology" target="_blank"><em>here</em></a><em>.</em></p><p>On October 29th, <em>The New York Times</em> reported that Nvidia co-founder and <a href="https://en.mycoding.id/category/technology" target="_blank">showman</a> CEO Jensen Huang <a href="https://en.mycoding.id/category/technology" target="_blank">appeared to be the target</a> of a Chinese spy plot. Any important tech CEO possibly being compromised by the Chinese authorities is concerning &mdash; particularly Jensen Huang. &ldquo;If you could diagram the network diagram of the complete US AI ecosystem, Nvidia would be at the center of that,&rdquo; Kyle Chan, a abroad guideline chap at the Brookings Institution&rsquo;s John L. Thornton China Center, earlier <a href="https://en.mycoding.id/category/technology" target="_blank">told <em>The Verge</em></a>. &ldquo;If you could diagram the diagram of the complete Nvidia ecosystem, a sole man is position at the bosom of all of that.&rdquo;</p><p>The <em>Times</em> reported that Jensen Huang had developed a &ldquo;<a href="https://en.mycoding.id/category/technology" target="_blank">close relationship</a>&rdquo; alongside Alice Huang (no relation), an administrator at Singapore-based data center business Megaspeed. Alice Huang had been on the American government&rsquo;s radar for several time, perchance <a href="https://en.mycoding.id/category/technology" target="_blank">as first as 2024</a>. Her business was investigated by the US Commerce Department complete suspicions it was illegally smuggling Nvidia&rsquo;s AI chips into China in infringement of US authorities commerce restrictions. Singapore authorities <a href="https://en.mycoding.id/category/technology" target="_blank">confiscated her passport</a> rear in June of 2025, eventually barring her from the country, following she reportedly gave &ldquo;inaccurate information&rdquo; to police. And perchance additional concerningly, several US officials believed she was passing information, perchance concerning Nvidia&rsquo;s designs or the White House&rsquo;s deliberations complete commerce restrictions, immediately rear to Beijing.</p><p>This made her association alongside Jensen Huang concerning for multiple reasons. Nvidia&rsquo;s chips are considered crucial to America&rsquo;s AI superiority and a matter of national safety by the White House. The authorities has restricted the transaction of Nvidia&rsquo;s most advanced chips to China since the Biden administration. But Jensen Huang has routinely pushed to <a href="https://en.mycoding.id/category/technology" target="_blank">loosen those commerce restrictions</a>. Meanwhile, he had gotten fairly cozy alongside President Trump, formerly equal <a href="https://en.mycoding.id/category/technology" target="_blank">putting him on speakerphone</a> on-stage at the All-In Summit.</p><p>Then final week, things escalated quickly.</p><p>On Friday, November 6th, the <em>Times</em> reported that Alice Huang was specifically <a href="https://en.mycoding.id/category/technology" target="_blank">recruited by China&rsquo;s Ministry of State Security</a> to spy on Nvidia&rsquo;s CEO. Though, there is <a href="https://en.mycoding.id/category/technology" target="_blank">some disagreement</a> among US intellect officials complete the power of evidence tying Alice Huang to Beijing. While the document did not verify it at the time, gossip suggested the &ldquo;close relationship&rdquo; between Jensen Huang and Alice Huang was affectionate in nature.</p><p>On Sunday, November 8th, Jensen Huang <a href="https://en.mycoding.id/category/technology" target="_blank">admitted</a> to having an extramarital affair, but did not name the Megaspeed exec and alleged Chinese spy specifically. Nvidia&rsquo;s commission announced that it was <a href="https://en.mycoding.id/category/technology" target="_blank">launching an investigation</a> into its CEO shortly after.</p><p>Many <a href="https://en.mycoding.id/category/technology" target="_blank">believed</a> that Jensen Huang would <a href="https://en.mycoding.id/category/technology" target="_blank">survive</a> the scandal. His company&rsquo;s importance to America&rsquo;s booming AI industry and his crucial function at the business seemed to create him nearly untouchable. He was, following all, the lone CEO the business had always known alongside no apparent and apparent succession scheme in place, and he sat atop an different company construction alongside about 60 straightforward reports. (These numbers are slightly outdated, but according to the <em>Harvard Business Review</em>, <a href="https://en.mycoding.id/category/technology" target="_blank">10 to 20 is far additional common</a> among Fortune 500 companies, and Tim Cook <a href="https://en.mycoding.id/category/technology" target="_blank">reportedly had 17 in 2015</a>, which was deemed &ldquo;a lot&rdquo; by <em>Business Insider</em>.)</p><p>Then on Wednesday, November 11th, <em>The New York Times</em> <a href="https://en.mycoding.id/category/technology" target="_blank">reported</a> that Jensen Huang had a second matter &mdash; this one alongside one of his straightforward reports, Kimberly Powell, Nvidia&rsquo;s vice chairman of health care. Whether that was the final straw or if they were fearful it would be merely the tip of the iceberg, the commission was prepared to <a href="https://en.mycoding.id/category/technology" target="_blank">push Jensen Huang out,</a> according to reports circulating that evening. According to <a href="https://en.mycoding.id/category/technology" target="_blank"><em>Bloomberg</em></a>, he had accepted his destiny Wednesday evening reportedly looking &ldquo;broken and shattered&rdquo; and telling people, &ldquo;I&rsquo;m a part of shit.&rdquo;</p><div><p>On Thursday, November 12th, Nvidia announced that <a href="https://en.mycoding.id/category/technology" target="_blank">Jensen Huang had resigned</a> as CEO of Nvidia.</p></div><p>CFO Colette Kress was immediately named interim CEO, and acknowledged to workforce that this has been an &ldquo;<a href="https://en.mycoding.id/category/technology" target="_blank">extraordinarily difficult period</a>.&rdquo; Harvey Jones was additionally named the archetypal commission place in the company&rsquo;s history, already indication there would be several changes to the company structure. In a memo to employees, Jones stated the business would &ldquo;search for a imperishable successor looking at the two inner and external candidates.&rdquo;</p><p>But the destabilizing effect was immediate. Nvidia&rsquo;s inventory closed at an all-time elevated of <a href="https://en.mycoding.id/category/technology" target="_blank">$239.24 on October 6th</a>. It closed at <a href="https://en.mycoding.id/category/technology" target="_blank" rel="sponsored">$162.85</a> on November 13th. It&rsquo;s difficult to overstate the importance of Nvidia to the US&rsquo;s <a href="https://en.mycoding.id/category/technology" target="_blank">AI arms competition alongside China</a>, and the industry in general. The business has invested <a href="https://en.mycoding.id/category/technology" target="_blank">$99 billion</a> in assorted AI labs, startups, and neo clouds. Plus has announced plans to mobilize <a href="https://en.mycoding.id/category/technology" target="_blank">over $500 billion</a> in financing for construction out AI data centers and another infrastructure.</p><p>Even before Jensen Huang had resigned, Nvidia was already facing a <a href="https://en.mycoding.id/category/technology" target="_blank">shareholder lawsuit</a> claiming the business had deceived investors. Jensen Huang and the company&rsquo;s spokespeople insisted that the association alongside Alice Huang and Megaspeed was &ldquo;<a href="https://en.mycoding.id/category/technology" target="_blank">completely normal and not unusual</a>.&rdquo; Only days later, Jensen Huang admitted to an extramarital matter that raised &ldquo;<a href="https://en.mycoding.id/category/technology" target="_blank">national safety concerns</a>&rdquo; without naming anyone specifically. According to <a href="https://en.mycoding.id/category/technology" target="_blank"><em>Reuters</em></a>, Jensen Huang misled multiple Nvidia leaders, &ldquo;including broad advise Tim Teter and chief promotion authoritative Alison Wagonfeld&rdquo; concerning the nature of his association alongside Alice Huang.</p><div><p>Many in the AI industry and Silicon Valley <a href="https://en.mycoding.id/category/technology" target="_blank">jumped to Jensen Huang&rsquo;s defense</a>. Epic Games CEO Tim Sweeney called it a &ldquo;<a href="https://en.mycoding.id/category/technology" target="_blank">sad day for NVIDIA, developers, America, and all humanity</a>,&rdquo; in a particularly hyperbolic post. Elon Musk stated on X that it would &ldquo;<a href="https://en.mycoding.id/category/technology" target="_blank">be an honor</a>&rdquo; to employ Jensen Huang. Many of Nvidia&rsquo;s workforce equal voiced their assistance to &ldquo;<a href="https://en.mycoding.id/category/technology" target="_blank">bring rear THE boss</a>,&rdquo; on an inner post.</p></div><p>Some equal speculated that Jensen Huang was the casualty of a <a href="https://en.mycoding.id/category/technology" target="_blank">vast conspiracy</a>, whether that was a coup by several faction inside Nvidia, or perchance the goal of China all alongside had merely been to <a href="https://en.mycoding.id/category/technology" target="_blank">chase him from the CEO chair</a>.</p><p>By Thursday night, Reuters reported a <a href="https://en.mycoding.id/category/technology" target="_blank">third affair</a>, this one additionally alongside an Nvidia employee, although no additional data has emerged.</p><p>First and foremost for Nvidia, it must discover a new CEO. It&rsquo;s imaginable that Kress eventually ceases to be &ldquo;interim&rdquo; CEO, but another names, including Apple&rsquo;s hardware chief <a href="https://en.mycoding.id/category/technology" target="_blank">Johny Srouji</a>, have additionally been rumored. (Donald Trump believes they should employ <a href="https://en.mycoding.id/category/technology" target="_blank">Tim Cook</a>, but to call that improbable would be an understatement.)</p><p>It&rsquo;s additionally imaginable that Nvidia volition undergo a theatrical reorganization. Whoever takes complete as CEO is improbable to keep the 60 straightforward reports of Jensen Huang. With a first-ever commission place in Jones, it&rsquo;s additionally imaginable that the commission of board volition perform a larger function in the business going forward.</p><p>The board&rsquo;s inquiry into Jensen Huang that began on November 8th is ongoing. The controversy is <a href="https://en.mycoding.id/category/technology" target="_blank">likely to drag on for several time</a>. With the possible that Jensen Huang volition have to <a href="https://en.mycoding.id/category/technology" target="_blank">testify before Congress</a>, the business may choose to be proactive concerning disclosing any additional data publicly.</p><p>The incoming Democratic bulk is expected to arrive downward harder on Silicon Valley than the current Republican Congress. While it&rsquo;s unclear fair how far the new House and Senate volition go in trying to rein in the tech industry and its billionaire leaders, it&rsquo;s imaginable it volition try to create an example of Nvidia as an first act of strength. If Nvidia is established to have been turning a blind eye to companies akin Megaspeed, which are suspected of funneling advanced chips to China, it could visage civic or offender penalties. And it seems increasingly apt that stricter commerce restrictions volition be implemented on Nvidia&rsquo;s chips, which could addition tensions alongside China in an already costly commerce war.</p><p>There is a possible that Jensen Huang could visage offender charges. In May, he claimed that Nvidia had &ldquo;<a href="https://en.mycoding.id/category/technology" target="_blank">zero percent</a>&rdquo; market portion in China because of stringent export restrictions. But shortly after, Culper Research published a study claiming that in actuality <a href="https://en.mycoding.id/category/technology" target="_blank">over 20 percent</a> of the company&rsquo;s 2026 compute income was from China, although that figure has not been independently verified. The study cited &ldquo;illegal GPU amusement and Southeast Asian intermediaries&rdquo; as the chief origin of those revenues. Among the largest of those intermediaries, according to Culper Research, was Megaspeed, which Nvidia often vouched for to the US government.</p><p>If evidence is established that the CEO was knowingly assisting Megaspeed or any another companies to illegally divert restricted chips to China, he could visage offender charges. And, as <em>The Verge&rsquo;s</em> Elizabeth Lopatto put it, &ldquo;under a functioning lawful system, [he] <a href="https://en.mycoding.id/category/technology" target="_blank">could really go to jail</a>.&rdquo;</p><div><p>There is the chance that members of the current administration are additionally tied up in this scandal, as it&rsquo;s unclear <a href="https://en.mycoding.id/category/technology" target="_blank">how much the White House knew and when</a>.</p></div><p>Considering Nvidia&rsquo;s sizable investments, borrowing to build data centers, and the importance of its hardware, there are lots of open questions concerning how this impacts not fair the AI industry, but the American economics as a whole. Nvidia is basically <a href="https://en.mycoding.id/category/technology" target="_blank">propping up</a> a ample part of the tech industry and, if prosperity stops flowing out of its coffers, it could drag a lot of another companies downward alongside it.</p><p>There are additionally open questions concerning how this impacts the American public&rsquo;s awareness of the AI industry. The new Democratic bulk was elected in part by <a href="https://en.mycoding.id/category/technology" target="_blank">campaigning against AI companies</a> and the detonation of data center buildout throughout the country. That one of its additional notable CEOs is now embroiled in a spying controversy and multiple extramarital affairs likely isn&rsquo;t going to endear the industry to the public.</p><div><ul><li><span>Megaspeed was the largest buyer of Nvidia chips in Southeast Asia, but whenever Singaporean newspaper <a href="https://en.mycoding.id/category/technology" target="_blank"><em>The Straits Times</em></a> visited its agency in October of 2025, it established &ldquo;no signs of activity.&rdquo;</span></li><li><span>Megaspeed is fair one of multiple companies that Nvidia has been approved to market chips alongside doubtful offices. <a href="https://en.mycoding.id/category/technology" target="_blank"><em>The New York Times</em></a> described New Matrix&rsquo;s agency as &ldquo;a hollow gray casing alongside wires dangling from the ceiling anywhere lights should be.&rdquo;</span></li><li><span>Jensen Huang&rsquo;s connections to Super Micro are additionally under scrutiny. Earlier this year, <a href="https://en.mycoding.id/category/technology" target="_blank">one of its founders</a>, an employee, and <a href="https://en.mycoding.id/category/technology" target="_blank">a contractor</a> for the business were <a href="https://en.mycoding.id/category/technology" target="_blank">indicted</a> by the Justice Department for colluding to breach American export controls and divert Nvidia chips to China.</span></li></ul></div><div><ul><li><span>While a lot has changed, it&rsquo;s motionless value study the <a href="https://en.mycoding.id/category/technology" target="_blank"><em>Times</em></a>&rsquo; archetypal big study from October 29th that lays the groundwork for the eventual deluge of global intrigue and infidelity that ultimately led to Jensen Huang&rsquo;s downfall.</span></li><li><span><a href="https://en.mycoding.id/category/technology" target="_blank"><em>The New York Times</em></a> paints a fascinating picture of Alice Huang who now finds herself at the center of an global espionage scandal.</span></li><li><span><em>The Decoder</em> discussion alongside <a href="https://en.mycoding.id/category/technology" target="_blank">Kyle Chan</a>, a abroad guideline chap at the Brookings Institution&rsquo;s John L. Thornton China Center, digs into what China hoped to get out of Jensen Huang.</span></li></ul></div><div><p><span><strong>Follow topics and authors</strong> from this narrative to see additional akin this in your personalized homepage nourish and to obtain email updates.</span></p><ul><li id="follow-author-article_footer-dmcyOmF1dGhvclByb2ZpbGU6NjA5MzU0"><span aria-expanded="false" aria-haspopup="true" role="button" tabindex="0"><span><span><svg width="9" height="9" viewbox="0 0 9 9" fill="none" xmlns="http://www.w3.org/2000/svg" aria-label="Follow"><path d="M5 0H4V4H0V5H4V9H5V5H9V4H5V0Z"></path></svg></span><span>Terrence O'Brien</span></span></span></li><li><li><li><li><li><li></ul></div></div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/an-overview-of-jensen-huang-s-messy-days-77183</guid> <pubDate>Sun, 15 Nov 2026 14:00:00 +0700</pubDate> <author><![CDATA[support@mycoding.id (Terrence O’brien)]]></author> </item> <item> <title><![CDATA[A NES emulator written in Odin]]></title> <description><![CDATA[A NES emulator written in Odin. Comments]]></description> <media:thumbnail url="https://lucypero.com//../assets/images/lucynes/src_Dn6Bdt7DGO.png"/> <category><![CDATA[Technology]]></category> <link>https://en.mycoding.id/a-nes-emulator-written-in-odin-77194</link> <content:encoded><![CDATA[<div itemprop="articleBody"><p>I made a NES emulator in <a href="https://en.mycoding.id/category/technology" target="_blank">Odin</a>!</p> <p><a href="https://en.mycoding.id/category/technology" target="_blank">Github link</a></p> <p><img src="https://lucypero.com//../assets/images/lucynes/src_Dn6Bdt7DGO.png" alt="lucyNES"></p> <h2 id="intro">Intro</h2> <p>I&rsquo;ve started this project in delayed June of 2024. I&rsquo;ve worked on it in my spare period off and on. I decided to put a intermission on this project in January 2025 to concentration on another things.</p> <p>I&rsquo;ve learned so much about:</p> <ul> <li>The NES architecture, and perchance concerning equivalent console architectures in general</li> <li>Bits and bytes, hex notation, and bitwise operations.</li> <li>The Odin programming language. I haven&rsquo;t used it on a grave project before.</li> <li>How computers work. CPU instructions, &ldquo;cycles&rdquo;, etc</li> <li>Emulation. It&rsquo;s extremely hard. Respect to emulator developers!</li> </ul> <p>It&rsquo;s been a untamed ride. I&rsquo;ve had a lot of fun, but I&rsquo;ve additionally suffered additional towards the end, whenever trying to create the emulator &ldquo;fully&rdquo; accurate.</p> <p>You can peruse the <a href="https://en.mycoding.id/category/technology" target="_blank">Github Readme</a> for additional info concerning the project.</p> <ul> <li>Enough emulation accuracy to run Battletoads and hundreds of another games perfectly.</li> <li>Debugger perspective alongside several debugging features</li> <li>Menu perspective to engage alongside the emulator</li> <li>Save states</li> <li>Controller support</li> <li>CRT Shader toggle</li> <li>(Partial) Sound</li> <li>Supported Mappers: NROM, MMC1, UXROM, CNROM, MMC3, GxROM, AxROM</li> </ul> <h2 id="making-an-emulator">Making an emulator</h2> <p>I&rsquo;ve seen several group say that making a NES emulator is a uncomplicated weekend project, and I frankly established that baffling. Because <em>evidently</em>, it&rsquo;s not. It took me 7 months to get my emulator to how it is now.</p> <p>I think what they really average is that getting a NES emulator &ldquo;working&rdquo; takes a abbreviated time. Like perchance getting Donkey Kong to run. And that&rsquo;s true. You can get to 80% or perchance 90% beautiful fast. It&rsquo;s the final 10% that takes a long, lengthy time.</p> <p>The NES hardware has many idiosyncrasies that need to be emulated for certain games to run. Many games depend on extremely particular NES behavior. And if you don&rsquo;t execute it right, they won&rsquo;t run. A lot of the difficulty is concerning timing. The CPU and the PPU (the part that handles the graphics rendering in the NES) run in parallel and they communicate alongside all other. That timing needs to be extremely accurate, or many incorrect things can happen. Most games volition run fine alongside a small than exact emulator, but a lot of them won&rsquo;t, and they volition need you to regard things in additional detail.</p> <p>Another matter is <em>mappers</em>. Mappers are what we call an implementation of a category of cartridge. You see, NES games arrive in cartridges, and cartridges aren&rsquo;t fair data. They are circuitry alongside their own behavior and additional resources akin read-only memory, and writable memory. Some arrive alongside additional audio capabilities, but I haven&rsquo;t implemented that.</p> <p>Anyway, you have to execute these mappers if you desire a equivalent that comes in that cartridge to run in your game. They complete the NES&rsquo;s circuitry. In a way, they are a part of the NES hardware. There are many mappers, but you can assistance a lot of games by fair implementing a few. I&rsquo;ve had difficulty implementing several mappers, and several of them are motionless seriously implemented, causing several games to not work. I established it difficult to comprehend several of the mapper documentation.</p> <p>The biggest difficulty alongside making an emulator, I think, is to cognize what to implement. Information concerning the hardware, and what needs to be emulated concerning the hardware for games to activity in your emulator. Thankfully, the NES is extremely fine documented online. I&rsquo;ve used many resources, like:</p> <ul> <li><a href="https://en.mycoding.id/category/technology" target="_blank">The Nesdev wiki</a></li> <li><a href="https://en.mycoding.id/category/technology" target="_blank">The nesdev wiki discord</a>. People current have helped me alongside my issues a lot.</li> <li><a href="https://en.mycoding.id/category/technology" target="_blank">Javidx&rsquo;s NES series</a>. The code has many mistakes but it&rsquo;s a fine starting point.</li> <li><a href="https://en.mycoding.id/category/technology" target="_blank">This part concerning the NES&rsquo;s PPU</a></li> <li><a href="https://en.mycoding.id/category/technology" target="_blank">Test ROMS</a>. These NES ROMS thoroughly test particular aspects of your emulator. This is an bonzer tool to accelerate betterment of emulator accuracy.</li> </ul> <p>I wouldn&rsquo;t have been capable to do any of this without these resources and the assistance from group in the nesdev discord.</p> <h2 id="odin">Odin</h2> <p>I&rsquo;ve learned a lot concerning Odin during making this project. I think it fit this project akin a glove. Being a extremely simple, powerfully typed C-like tongue that gives me all the liberty I desire makes making an emulator much easier than using another languages, I feel.</p> <p>One item that threw me off was that Odin has no <a href="https://en.mycoding.id/category/technology" target="_blank">automatic entire figure promotion</a>. When doing calculations on numbers, Odin volition never advance types to a bigger one. That led to a few bugs. But I accuse this all on me and me being used to C&rsquo;s entire figure promotion.</p> <p>One fun item that I&rsquo;ve learned was doing a bit of recollection allocation strategy. Odin has no automatic recollection management, so you have to arrive up alongside astute ways to grip memory. I&rsquo;ve had fun thinking concerning how I should grip recollection best. I didn&rsquo;t need to do item complicated. I mostly use one impermanent allocator that clears all frame, a earth heap allocator for things that need to live forever, and a NES Arena allocator for item that needs to live alongside a NES session. I established recollection allocation fun.</p> <p>Another item I had to study is how to serialize Odin objects. I used the CBOR library, and I ran into a few issues. I idea my code was misbehaving because of a bug in the library, but that was <a href="https://en.mycoding.id/category/technology" target="_blank">cleared up</a>. It was lone having difficulty since I made a few odd types that it didn&rsquo;t cognize how to serialize. Once I learned how to correctly compose tradition handling for types, everything worked perfectly. This was among the tricky types that cbor didn&rsquo;t cognize how to serialize:</p> <div><div><pre><span>PpuCtrl</span> <span>::</span> <span>struct</span> <span>#raw_union</span> <span>{</span> <span>// VPHB SINN</span> <span>using</span> <span>flags</span><span>:</span> <span>bit_field</span> <span>u8</span> <span>{</span> <span>n</span><span>:</span> <span>u8</span> <span>|</span> <span>2</span><span>,</span> <span>i</span><span>:</span> <span>u8</span> <span>|</span> <span>1</span><span>,</span> <span>s</span><span>:</span> <span>u8</span> <span>|</span> <span>1</span><span>,</span> <span>// sprite form array location for 8x8 sprites (0: $0000, 1: $1000)</span> <span>b</span><span>:</span> <span>u8</span> <span>|</span> <span>1</span><span>,</span> <span>// bg form array location (0: $0000, 1: $1000)</span> <span>h</span><span>:</span> <span>u8</span> <span>|</span> <span>1</span><span>,</span> <span>// sprite size (0: 8x8, 1: 8x16)</span> <span>p</span><span>:</span> <span>u8</span> <span>|</span> <span>1</span><span>,</span> <span>v</span><span>:</span> <span>u8</span> <span>|</span> <span>1</span><span>,</span> <span>// vblank NMI enable</span> <span>},</span> <span>reg</span><span>:</span> <span>&nbsp;</span> <span>&nbsp;</span> <span>&nbsp;</span> <span>&nbsp;</span> <span>u8</span><span>,</span> <span>}</span> </pre></div></div> <p>I affection how Odin lets you define raw unions akin this, and alongside bit fields. It fits emulator betterment extremely well.</p> <h2 id="whats-missing">What&rsquo;s missing</h2> <p>I don&rsquo;t regard this project &ldquo;finished&rdquo;. Of course, a project akin this is never really finished, but a lot of aspects of it need a lot of work, for sure:</p> <ul> <li>The sound. lucyNES doesn&rsquo;t audio extremely fine still. Some games volition audio fine but most won&rsquo;t. I have to enhance the implementation, and execute the DMC conduit which I haven&rsquo;t done.</li> <li>Some games that I&rsquo;ve tested don&rsquo;t activity because of a mediocre mapper implementation</li> <li>I could execute additional mappers so additional games are supported.</li> </ul> <p>These are the chief things that are lacking at this moment. Maybe at several item I volition come back and keep operating on this so the project can awareness &ldquo;complete&rdquo;.</p> <h2 id="showcase">Showcase</h2> <h3 id="debugger-showcase">Debugger showcase</h3> <video controls=""> &nbsp; <source src="../assets/images/lucynes/407106285-240fbda8-4e75-4b93-b485-15ab7fd3c7bb.mp4" type="video/mp4"></source> </video> <video controls=""> &nbsp; <source src="../assets/images/lucynes/407106126-a09d3cc3-a58b-44ea-8771-bcb8bb00a3d6.mp4" type="video/mp4"></source> </video> <h3 id="game-showcase">Game showcase</h3> <p><img src="https://lucypero.com//../assets/images/lucynes/407106733-edcbadf1-9f50-46f0-8274-e45dc8d71233.png" alt="showcase image"></p> <p><img src="https://lucypero.com//../assets/images/lucynes/407106076-41ba0e3a-9520-4b11-a5cc-7926bf3f15bd.png" alt="showcase image"></p> <p><img src="https://lucypero.com//../assets/images/lucynes/407106574-7bba8fa2-858e-4e73-86e4-8f94d46db19e.png" alt="showcase image"></p> <p><img src="https://lucypero.com//../assets/images/lucynes/407106624-d9f08bdb-c45f-4e40-a321-475b2d216d60.png" alt="showcase image"></p> <p><img src="https://lucypero.com//../assets/images/lucynes/407106649-069c21ad-2653-4021-84a9-5d6bb4e4053a.png" alt="showcase image"></p> <p><img src="https://lucypero.com//../assets/images/lucynes/407106672-52720c8b-1ca9-4c55-9fca-33ae737b970a.png" alt="showcase image"></p> </div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/a-nes-emulator-written-in-odin-77194</guid> <pubDate>Sun, 15 Nov 2026 13:51:47 +0700</pubDate> <author><![CDATA[support@mycoding.id (Adi Gunawan)]]></author> </item> <item> <title><![CDATA[WP-Optimize Premium v4.7.1 – WordPress Plugin]]></title> <description><![CDATA[WP-Optimize Premium v4.7.1 – WordPress Plugin. WP-Optimize Premium v4.7.1 – WordPress Plugin Free Download. WP-Optimize is a Revolutionary, All-in-one Plugin that Cleans your Database, Compresses your Large ...]]></description> <media:thumbnail url="https://en.mycoding.id/site/assets/img/og-image.png"/> <category><![CDATA[Digital Products]]></category> <link>https://en.mycoding.id/wp-optimize-premium-v4-7-1-wordpress-plugin-77179</link> <content:encoded><![CDATA[<div class="entry-content admission clearfix"><p class="wp-block-paragraph">WP-Optimize Premium v4.7.1 – WordPress Plugin Free Download. WP-Optimize is a Revolutionary, All-in-one Plugin that Cleans your Database, Compresses your Large Images and Caches your site. This Simple, Popular and Highly Effective Tool has Everything you need to keep your Website accelerated and thoroughly Optimized.</p><h3 class="wp-block-heading">Main Features</h3><ul class="wp-block-list"><li>Clean Database</li><li>Automatic Clean-ups</li><li>Optimize Images</li><li>Cache your Pages and Speed up</li><li>Statistics</li><li>Easy Optimization</li><li>Trackbacks and Pingbacks</li><li>Set Notifications</li><li>and more</li></ul><h2 class="wp-block-heading">Download WP-Optimize Premium Plugin</h2><b>Live Demo:</b> <a href="https://en.mycoding.id/category/digital-products" target="_blank" target="_blank" rel="noopener"><b>View Demo</b></a><br><br><a href="https://en.mycoding.id/category/digital-products" target="_blank" target="_blank"><b>WP-Optimize Online Documentation</b></a><br><br><p class="wp-block-paragraph"><strong>v4.7.1</strong></p><div class="stream-item stream-item-in-post stream-item-inline-post aligncenter"><div class="stream-item-size"><div id="container-0730062b1c60873c0fcc80d9fa01093b"></div></div></div><div class="download-link-section"><div class="copybox"></div> <br><div class="copybox"></div> <br><div class="copybox"></div> <br><div class="copybox"></div> <br><div class="copybox"></div> <br><div class="copybox"></div> <br><div class="copybox"></div> <br><div class="copybox"></div></div><br><p class="wp-block-paragraph"><strong>v4.5.1</strong></p><div class="download-link-section"><div class="copybox"></div> <br><div class="copybox"></div> <br><div class="copybox"></div> <br><div class="copybox"></div></div><br><p class="wp-block-paragraph"><strong>v4.2.2</strong></p><div class="download-link-section"><div class="copybox"></div> <br><div class="copybox"></div></div><br><p class="wp-block-paragraph"><strong>v3.2.20</strong></p><div class="download-link-section"><div class="copybox"></div> <br><div class="copybox"></div> <br><div class="copybox"></div></div><br><p class="wp-block-paragraph"><strong>v3.2.14</strong></p><div class="download-link-section"><div class="copybox"></div> <br><div class="copybox"></div></div><br><p class="wp-block-paragraph">Free download WP-Optimize Premium WordPress plugin by WP-Optimize on XIBO LTD premium WordPress optimize caches <strong><a href="https://en.mycoding.id/category/digital-products" target="_blank">WP Plugin</a></strong> download.</p><div class="stream-item stream-item-below-post-content"><div class="stream-item-size"><div id="container-0730062b1c60873c0fcc80d9fa01093b"></div></div></div></div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/wp-optimize-premium-v4-7-1-wordpress-plugin-77179</guid> <pubDate>Sun, 15 Nov 2026 13:45:00 +0700</pubDate> <author><![CDATA[support@mycoding.id (Null Php Script)]]></author> </item> <item> <title><![CDATA[My husband was an NFL head coach, but we raised our 3 kids with strict rules about money. Christmas meant just 4 gifts.]]></title> <description><![CDATA[My husband was an NFL head coach, but we raised our 3 kids with strict rules about money. Christmas meant just 4 gifts.. Kiya Tomlin and her husband, NFL trainer Mike Tomlin, raised their three kids alongside financial boundaries, including a four-gift Christmas rule.]]></description> <media:thumbnail url="https://i.insider.com/6ae8e20ef3bee449b3d2a512?width=1200&amp;format=jpeg"/> <category><![CDATA[Business]]></category> <link>https://en.mycoding.id/my-husband-was-an-nfl-head-coach-but-we-raised-our-3-kids-with-strict-rules-about-money-christmas-meant-just-4-gifts-77174</link> <content:encoded><![CDATA[<div e-name="post-hero" data-component-type="post-hero" data-load-strategy="exclude"> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject"> <div> <p><img src="https://i.insider.com/6ae8e161f3bee449b3d2a50c?width=2000&amp;format=jpeg&amp;auto=webp" srcset="https://i.insider.com/6ae8e161f3bee449b3d2a50c?width=400&amp;format=jpeg&amp;auto=webp 400w, https://i.insider.com/6ae8e161f3bee449b3d2a50c?width=500&amp;format=jpeg&amp;auto=webp 500w, https://i.insider.com/6ae8e161f3bee449b3d2a50c?width=700&amp;format=jpeg&amp;auto=webp 700w, https://i.insider.com/6ae8e161f3bee449b3d2a50c?width=1000&amp;format=jpeg&amp;auto=webp 1000w, https://i.insider.com/6ae8e161f3bee449b3d2a50c?width=1300&amp;format=jpeg&amp;auto=webp 1300w, https://i.insider.com/6ae8e161f3bee449b3d2a50c?width=2000&amp;format=jpeg&amp;auto=webp 2000w" sizes=" 1280px) 900px" alt="Tomlin family" decoding="sync"> </p></div> <span> <span> <svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewbox="0 0 24 24"> <path fill="currentColor" fill-rule="evenodd" d="m4.56 18.5 7.486-7.72 7.394 7.626 2.56-2.64L12.046 5.5 2 15.86l2.56 2.64Z"></path> </svg> <figcaption e-name="image-caption"> <span>Kiya Tomlin built her occupation about her husband's NFL schedule.</span> <span> <span e-name="image-source" itemprop="creditText">Courtesy of the Tomlin family</span> </span> </figcaption> </span> </span> </figure> </div> <div data-component-type="image-modal" data-load-strategy="lazy"> <p><em>This as-told-to composition is according to a conversation alongside </em><a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;outbound_click&quot;}' rel=" nofollow"><em>Kiya Tomlin</em></a><em>, originator and decorator of </em><a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;outbound_click&quot;}' rel=" nofollow"><em>Kiya Tomlin apparel</em></a><em>. It has been edited for dimension and clarity.</em></p><p>As the <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">wife of an NFL coach</a>, I was extremely visible, but didn't awareness seen. I was continually acknowledged as someone's wife, and most conversations revolved about my husband's career. Even our home guests and parties became obligations tied to football, not opportunities to genuinely bask connecting alongside people.</p><p>I am excellent arrogant of my hubby and our three kids. But I'm additional than fair a spouse and mother. When Mike and I <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">met in college</a>, I was preparedness to be a doctor until I realized a medicinal occupation wasn't feasible, whereas Mike's occupation could obtain him anywhere.</p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6ae8e428d8cc18cf795103c3&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:3667,&quot;aspectRatioH&quot;:2907}}' alt="Kiya Tomlin (L) and Mike Tomlin act the 13th annual NFL Honors"> </p></div> <span> <figcaption e-name="image-caption"> Kiya and Mike Tomlin met in college.&nbsp; <span e-name="image-source" itemprop="creditText"> Ethan Miller/Getty Images </span> </figcaption> </span> </figure> <p>That's whenever I started my <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">fashion career</a>. I'd been making my own apparel since I was 10, and manner is a enthusiasm of mine. It was a occupation that I could agenda about the kids and do anyplace Mike's job took us.</p><h2 id="e50a5813-f8fd-47de-8b42-1188a7a7455f" data-toc-id="e50a5813-f8fd-47de-8b42-1188a7a7455f">One of my finest accomplishments is supporting another mothers</h2><p>I was raised by a sole mom, who was a community university professor. We didn't have much money, and we were among the few Black families in the <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">New Jersey</a> town anywhere I was raised. I was additionally extremely small: in kindergarten, the another kids used to transport me on their hip akin a toddler.</p><p>I never felt akin I fit in, and part of that was since I couldn't fit into or <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">afford the clothes</a> my classmates wore. That was problematic for me, so whenever I was 10, I asked my mom for a sewing machine. I started making apparel that looked akin what the another kids wore, and complete period I began making outfits that could assistance me remain out in a distinct way. That became my assertion to fame.</p><p>Today, one of my proudest accomplishments is manufacturing my garments at my <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">factory in Pittsburgh</a>. My six workforce who activity there are all moms. Creating a elastic activity surroundings that lets them prioritize raising their kids during additionally supporting their family financially has been amazing.</p><h2 id="7c22d300-41d2-48b8-97c1-e025f22b6c5b" data-toc-id="7c22d300-41d2-48b8-97c1-e025f22b6c5b">Our family had an adaptation duration following all season</h2><p>As Mike's <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">career took off</a>, football dictated everything in our lives. We built our family about that reality. During the season, Mike sometimes worked 20-hour days and slept at the training complex. Once he became a caput coach, the hours were slightly small intense, particularly as innovation improved. He could typically get residence for family period on a Friday evening.</p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6ae8e3a58bd5d6547af34c30&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:6096,&quot;aspectRatioH&quot;:4066}}' alt="Mike Tomlin looks on during broadcasting from the site before a equivalent between the Denver Broncos and the Los Angeles Rams"> </p></div> <span> <figcaption e-name="image-caption"> As Mike Tomlin's occupation took off, his spouse Kiya was at residence alongside their three kids.&nbsp; <span e-name="image-source" itemprop="creditText"> Justin Edmonds/Getty Images </span> </figcaption> </span> </figure> <p>I handled everything connected to the kids. It was crucial to me to keep my <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">independence financially</a>, emotionally, and spiritually.</p> <p>It wasn't continually easy. After all season, there was a duration of reentry. I'd spent the football period being the sole chief at home, and sometimes the way I did things wasn't the way Mike would do them. During those adaptation periods, I knew I had to relax, relate, and release.</p><h2 id="54abe24c-349f-44cc-80d3-1b89b7b58b4f" data-toc-id="54abe24c-349f-44cc-80d3-1b89b7b58b4f">We didn't spoil our kids fair since we could</h2><p>When I was expanding up, prosperity choices were according to a distinction: is it a want, or a need? Because finances were limited, I normally got my needs met, but frequently went without things I wanted.</p><p>Once I was raising my kids &mdash; who are now 20, 24, and 25 &mdash; I didn't have many external financial limitations. But I didn't delay to lift spoiled kids, equal if I could oversee many of their wants.</p><p>Mike and I frequently reflected on our <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">own childhoods</a> and took a beautiful bare-bones method alongside our kids. On Christmas, for example, we followed the four current rule: item you want, item you need, item to wear, and item to read. The archetypal twelvemonth of instituting that was tough, but the kids adjusted.</p><h2 id="f6614a11-8275-42f4-a258-8343a3c412fc" data-toc-id="f6614a11-8275-42f4-a258-8343a3c412fc">I'm enthusiastic to see anywhere my strengths guide in center age</h2><p>Today, I'm 52. My kids are grown, and Mike is retired, which method I'm untethered from many of the obligations that I've had for the final 30 years. I have additional liberty than I've always had as an adult. I'm enthusiastic to have this area to grow. I desire to see what I'm capable of, alongside everything I've learned alongside the way.</p><p>In center age, women are eventually at a item anywhere we have experience, knowledge, and money. Yet, we're culturally overlooked. That can create us awareness invisible, equal whenever we cognize better.</p><p>We need to acknowledge that we have this amazing set of strengths. We don't need to chase our younger selves; we need to relish the fact that all experience, challenge, and accomplishment we've gained has made us who we are today. That regulation has guided me as a designer, and now it's directing me into the next phase of my life.</p> </div> <h2>Read next</h2> <div data-recommendations-content data-track-click-shared='&quot;Read next&quot;}'> <div e-name="read-next-list" data-recommendations-list> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> </div> </div> <a e-name="read-next-image-link" data-recommendations-image-link data-recommendations-link tabindex="-1"> <p><img alt=""> </p> </a> <h3> <a e-name="read-next-title-link" data-recommendations-link data-recommendations-title></a> </h3> <div data-component-type="post-bottom-authors" data-load-strategy="exclude" data-track-view='&quot;module_in_view&quot;}' data-track-click-shared='&quot;navigation&quot;}'> <div data-track-view='&quot;module_in_view&quot;}'> <div> <p><span>Kelly Burch has been penning concerning individual backing for additional than a decade.</span><span>She's particularly curious in how finances effect the most intimate&nbsp;parts of a person&rsquo;s life, from instructional and reproductive choices to love, immigration, or asset planning. Kelly has written concerning these topics personally and explored them alongside experts, including entrepreneurs, multi-millionaires, financial planners, and more.</span><span>Kelly is a first-generation university alumnus and homeowner who integrates her individual cognition creating financial stability into her reporting. She&rsquo;s a occupation journalist, alongside activity appearing in &ldquo;The Washington Post,&rdquo; &ldquo;The Chicago Tribune,&rdquo; &ldquo;Boston Magazine&rdquo; and more.</span><span>Kelly lives in agrarian New Hampshire alongside her husband, two children, and two dogs. When she&rsquo;s not rearward her desk, she can be established getting misplaced in the mountains and lakes about her home.</span><span>Follow her on <a href="https://en.mycoding.id/category/business" target="_blank">Facebook</a>&nbsp;or&nbsp;<a href="https://en.mycoding.id/category/business" target="_blank">Twitter</a>, or study more&nbsp;<a href="https://en.mycoding.id/category/business" target="_blank">here</a>.&nbsp; &nbsp;</span> </p> </div> </div> </div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/my-husband-was-an-nfl-head-coach-but-we-raised-our-3-kids-with-strict-rules-about-money-christmas-meant-just-4-gifts-77174</guid> <pubDate>Sun, 15 Nov 2026 13:28:01 +0700</pubDate> <author><![CDATA[support@mycoding.id (Kelly Burch)]]></author> </item> <item> <title><![CDATA[I loved living in NYC and thought I'd stay forever. As my life changed, however, so did where I wanted to be.]]></title> <description><![CDATA[I loved living in NYC and thought I'd stay forever. As my life changed, however, so did where I wanted to be.. I felt akin I'd won the lottery whenever I moved to New York City. My existence changed, and what I wanted from a residence did, too.]]></description> <media:thumbnail url="https://i.insider.com/6ada1a422a1b5bc156b1af0f?width=1200&amp;format=jpeg"/> <category><![CDATA[Business]]></category> <link>https://en.mycoding.id/i-loved-living-in-nyc-and-thought-i-d-stay-forever-as-my-life-changed-however-so-did-where-i-wanted-to-be-77175</link> <content:encoded><![CDATA[<div e-name="post-hero" data-component-type="post-hero" data-load-strategy="exclude"> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject"> <div> <p><img src="https://i.insider.com/6ada1a3adc67b5b4d01f6627?width=1300&amp;format=jpeg&amp;auto=webp" srcset="https://i.insider.com/6ada1a3adc67b5b4d01f6627?width=400&amp;format=jpeg&amp;auto=webp 400w, https://i.insider.com/6ada1a3adc67b5b4d01f6627?width=500&amp;format=jpeg&amp;auto=webp 500w, https://i.insider.com/6ada1a3adc67b5b4d01f6627?width=700&amp;format=jpeg&amp;auto=webp 700w, https://i.insider.com/6ada1a3adc67b5b4d01f6627?width=1000&amp;format=jpeg&amp;auto=webp 1000w, https://i.insider.com/6ada1a3adc67b5b4d01f6627?width=1300&amp;format=jpeg&amp;auto=webp 1300w" sizes=" 1280px) 900px" alt="the author and a companion atop a construction in NYC" decoding="sync"> </p></div> <span> <span> <svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewbox="0 0 24 24"> <path fill="currentColor" fill-rule="evenodd" d="m4.56 18.5 7.486-7.72 7.394 7.626 2.56-2.64L12.046 5.5 2 15.86l2.56 2.64Z"></path> </svg> <figcaption e-name="image-caption"> <span>The author, left, idea she'd continually desire to live in New York.</span> <span> <span e-name="image-source" itemprop="creditText">Courtesy of Hannah Howard</span> </span> </figcaption> </span> </span> </figure> </div> <div data-component-type="image-modal" data-load-strategy="lazy"> <p>When I was in my first 20s, operating at a small dairy merchandise and vino bar in Hell's Kitchen, my chief told me item I've never forgotten: New Yorkers aren't necessarily <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">born in New York</a>. They're delivered all complete the world, but New York is in their hearts.</p><p>I knew exactly what he meant. New York had been in my bosom for as lengthy as I could remember. I archetypal visited whenever I was 7, sleeping on the couch of my aunt and uncle's West Village apartment. I recall begging my mom for a fluffy pink scarf from a street vendor. She stated yes, and I wore it proudly as we wandered the crowded sidewalks.</p><p>I loved the noise, the energy, the sheer figure of group going concerning their lives. I loved that you could be nearly anyone, attire nearly any way, and motionless have to activity beautiful difficult to be considered equal a small bit weird. I decided afterward that I would live there someday.</p><h2 id="1bd2b874-736f-40e9-a3ee-ca8514591122" data-toc-id="1bd2b874-736f-40e9-a3ee-ca8514591122">Getting to New York felt akin winning the lottery</h2><p>When it came period to <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">apply to colleges</a>, I looked archetypal at schools in Manhattan. Getting into Columbia felt akin winning the lottery. I was the luckiest female in the world, headed for the city I'd dreamed concerning since I was little.</p><p>After graduation, I spent a few years in Los Angeles and Philadelphia. Both had their charms, but whenever a job brought me <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">back to New York</a>, it felt akin a homecoming.</p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6ada1c57510d67c758d85bd6&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:1500,&quot;aspectRatioH&quot;:1125}}' alt="the author during operating at a dairy merchandise shop"> </p></div> <span> <figcaption e-name="image-caption"> The author loved living and operating in New York.&nbsp; <span e-name="image-source" itemprop="creditText"> Courtesy of Hannah Howard </span> </figcaption> </span> </figure> <p>I've continually established item oddly soothing concerning New York's frantic pace. My brain moves quickly, bouncing between ideas, worries, and possibilities. The city seemed to run at the identical frequency. Its anarchy somehow quieted my own.</p><p>I loved strolling out my forefront entrance and immediately being in the center of everything. I could expend an midday wandering without a destination and stumble upon a new favorite bakery, a different small shop, or an unexpected conversation. Even following additional than a decade, a myriad apartments, awful bosses, small kitchens, and <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">eye-watering rent</a>, the magic never wore off.</p><p>Sometimes I'd be strolling downward the sidewalk, carrying groceries or rushing to encounter a friend, and I'd appearance about and think, <em>I live here. </em>That lottery-winning glee never entirely disappeared.</p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6ada1c81dc67b5b4d01f6646&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:2048,&quot;aspectRatioH&quot;:1365}}' alt="the author in her flat in NYC"> </p></div> <span> <figcaption e-name="image-caption"> The author moved from New York whenever her existence changed.&nbsp; <span e-name="image-source" itemprop="creditText"> Courtesy of Hannah Howard </span> </figcaption> </span> </figure> <h2 id="10ebb5ed-a1c1-4520-9e70-882532f64cf7" data-toc-id="10ebb5ed-a1c1-4520-9e70-882532f64cf7">Then my existence began to change</h2><p>In first 2020, as the epidemic upended the world, my hubby and I <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">left New York</a> for agrarian New Jersey. We had a baby female and shortly welcomed a son. Our days filled alongside diapers, naps, playgrounds, and the logistics of caring for two small kids. Instead of crowded sidewalks, we had a backyard. Instead of the subway, a car.</p> <p>I loved the breathing room. But I missed New York fiercely: the spontaneity, the restaurants, the friends, the possible waiting exterior my door. I assumed we'd eventually discover our way back. New York was motionless my favorite city. Surely that meant I was expected to live there.</p><p>Then, in 2025, my husband's job <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">brought us to Chicago</a>. I was excited, but figured New York would continually be the location I secretly desired we could come back to.</p><h2 id="f7326df3-185f-4602-aeb0-12962be731ff" data-toc-id="f7326df3-185f-4602-aeb0-12962be731ff">Chicago gave me item I didn't cognize I was missing</h2><p>We settled in Bucktown, a neighborhood of old masonry block houses, leafy streets, coffee shops, and families pushing strollers.</p><p>We stroll our kids to school, see acquainted faces at the playground, and stroll the 606, an elevated trail through our neighborhood, stopping for tacos or ice cream. There's a farmers' market anywhere I buy flowers and a bus to the Lincoln Park Zoo. <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">Chicago has outstanding restaurants</a>, museums, and endless opportunities to explore. But there's additionally additional area to move, small conflict in mundane life.</p><p>I hadn't realized how much I wanted that.</p><p>Winter is long, and making a new residence takes work. But I awareness amazingly established here.</p><p>I have the energy I loved concerning New York, alongside a neighborhood existence that feels manageable alongside two young kids. I can have an aspiring career, fantastic meals, engaging friends, and adventure without all average errand requiring a complex plan.</p><h2 id="766f5f14-f073-4fe6-aaaa-514437ea2489" data-toc-id="766f5f14-f073-4fe6-aaaa-514437ea2489">I motionless affection New York; I fair don't need to live there anymore</h2><p>On my final <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">trip to New York</a>, I was in town for a conference, staying in Midtown. One afternoon, I put on my headphones and walked downward Broadway through Union Square and into The Village, a path I'd traveled a thousand times before.</p><p>I popped into Eataly for a gelato, weaving about tourists and group hurrying location important. Music played in my ears, and my feet knew the way. I remembered the distinct versions of myself who had walked these blocks: the university pupil thrilled to be here, the aspiring author figuring out what came next, the newlywed.</p><p>I missed my kids, rear residence in Chicago. I felt <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">nostalgic for the city</a> and those before versions of my life. But alongside that nostalgia was item unexpected: complete contentment.</p><p>I was so blessed to be in New York. And fair as blessed that I didn't live there anymore.</p><p>The city hadn't misplaced its magic. I'd changed.</p><p>I wanted to stroll my children to school, cognize my neighbors, and have area for the average pleasures of family life. I wanted a city that felt breathtaking without making everything fairly so hard.</p><p>For years, being a New Yorker was an crucial part of how I understood myself. I liked what it stated concerning me: that I was adventurous, ambitious, curious, and consenting to clasp a small chaos.</p><p>But those qualities didn't disappear whenever I moved away. I can motionless affection New York, sojourn often, and awareness a jolt of happiness all period I stage onto its sidewalks. I can motionless be the small female in the fluffy pink scarf, dazzled by all that possibility.</p><p>And I can be fair as blessed to go residence to Chicago.</p> </div> <h2>Read next</h2> <div data-recommendations-content data-track-click-shared='&quot;Read next&quot;}'> <div e-name="read-next-list" data-recommendations-list> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> </div> </div> <a e-name="read-next-image-link" data-recommendations-image-link data-recommendations-link tabindex="-1"> <p><img alt=""> </p> </a> <h3> <a e-name="read-next-title-link" data-recommendations-link data-recommendations-title></a> </h3>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/i-loved-living-in-nyc-and-thought-i-d-stay-forever-as-my-life-changed-however-so-did-where-i-wanted-to-be-77175</guid> <pubDate>Sun, 15 Nov 2026 13:21:01 +0700</pubDate> <author><![CDATA[support@mycoding.id (Hannah Howard)]]></author> </item> <item> <title><![CDATA[An elite matchmaker says executives' biggest strength at work can make dating harder for them]]></title> <description><![CDATA[An elite matchmaker says executives' biggest strength at work can make dating harder for them. Executives' need for authority can get in the way of love, stated Talia Goldstein, the originator of Three Day Rule Matchmaking.]]></description> <media:thumbnail url="https://i.insider.com/6af769990cc3ffba8779cf51?width=1200&amp;format=jpeg"/> <category><![CDATA[Business]]></category> <link>https://en.mycoding.id/an-elite-matchmaker-says-executives-biggest-strength-at-work-can-make-dating-harder-for-them-77176</link> <content:encoded><![CDATA[<div e-name="post-hero" data-component-type="post-hero" data-load-strategy="exclude"> <div id="player-container-gewh91WF" data-component-type="video" data-type="video" data-video-player data-media-id="2NSkInYp" data-title="An choose matchmaker says executives&#039; biggest power at activity can create courtship harder for them" data-advertising="enabled" data-region="in-content" data-id="player-container-gewh91WF" data-enabled="true" data-events="disabled" data-preroll="4442842/businessinsider.desktop/post/pre-roll-in-content/strategy" data-midroll="4442842/businessinsider.desktop/post/mid-roll-in-content/strategy" data-sponsored="false" data-placement-id="gewh91WF" data-track-element="featured_video" data-float="true" data-thumbnail="https://i.insider.com/6af75f51cbe212047073546b?width=850&amp;format=jpeg&amp;auto=webp" data-render-thumbnail="true"> <div> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="https://i.insider.com/6af75f51cbe212047073546b?width=850&amp;format=jpeg&amp;auto=webp" alt=""> </p> </div> </div> </div> <div data-component-type="image-modal" data-load-strategy="lazy"> <p>When it comes to <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">finding love</a>, prosperous executives frequently battle alongside one item they can't control: chemistry.</p><p>In business, executives are used to getting what they want, they cognize what their end goal is, and they cognize what activity needs to be done to accomplish it, Talia Goldstein, originator of Three Day Rule Matchmaking, told Business Insider.</p><p>But this power in their expert existence <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">doesn't continually translate to dating.</a></p><p>"Some of the executives battle since you can't authority whether the person's curious in you, you can't authority whether or not there's chemistry, and that can be extremely frustrating for these top administrator types," Goldstein said.</p><p>Goldstein stated that <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">executives employ matchmakers</a> to assistance them preserve period and energy, two things that their job frequently gets the finest of.</p><p>But she stated that her firm, which charges clients anyplace from $6,300 to approximately $100,000 depending on the membership, additionally provides its clients coaching to assistance them flat up their courtship game.</p><p>"A lot of what we instruct them is to dilatory downward and to get susceptible and rely the process," she said. "And that can be challenging for group who are so used to controlling their destiny."</p><p>As part of this coaching, her resolute additionally collects feedback from the two parties following the date and uses that data to improved comprehend how the client is perceived by matches.</p><p>Without feedback, group <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">repeat the identical patterns</a> and are remaining wondering why they can't district that second or third date, stated Goldstein, whose firm's clients are divided approximately evenly between men and women.</p> <p>"With matchmaking, we cognize exactly why. So it ends up making the procedure additional enjoyable. And additionally you study so much concerning yourself in this," she said. "I think it's the most precious part of the service."</p><p>Goldstein shared an example of a prosperous entrepreneur her resolute worked alongside anywhere all date he came rear from the women would province liking him, but would afterward say the communication felt additional akin one alongside a friend. Goldstein stated the squad was stumped as to why none of the women they were environment him up alongside were operating out.</p><p>Thanks to the feedback sessions, they shortly learned their client was ending all date alongside a handshake alternatively of a hug.</p><p>"That fair felt akin a endeavor transaction and nobody was interested," she said. "And that one small alter made specified a difference. But on your own, you would never have figured that out."</p> </div> <h2>Read next</h2> <div data-recommendations-content data-track-click-shared='&quot;Read next&quot;}'> <div e-name="read-next-list" data-recommendations-list> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> </div> </div> <a e-name="read-next-image-link" data-recommendations-image-link data-recommendations-link tabindex="-1"> <p><img alt=""> </p> </a> <h3> <a e-name="read-next-title-link" data-recommendations-link data-recommendations-title></a> </h3> <div data-component-type="post-bottom-authors" data-load-strategy="exclude" data-track-view='&quot;module_in_view&quot;}' data-track-click-shared='&quot;navigation&quot;}'> <div data-track-view='&quot;module_in_view&quot;}'> <div> <p><span>Cadie Thompson is the delegate news chief and administrator publishing company at Business Insider.&nbsp;</span><span>She oversees the endeavor news desk, corporate, finance, guidance and careers, and lawful affairs teams. Previously, she was the delegate administrator publishing company of endeavor news, overseeing all tech, media, transportation, and retailing coverage. She additionally led the emerging tech squad at Tech Insider.</span><span>Before joining Insider, she worked as a innovation publishing company and newsman at CNBC.&nbsp;</span> </p> </div> </div> </div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/an-elite-matchmaker-says-executives-biggest-strength-at-work-can-make-dating-harder-for-them-77176</guid> <pubDate>Sun, 15 Nov 2026 13:19:01 +0700</pubDate> <author><![CDATA[support@mycoding.id (Cadie Thompson)]]></author> </item> <item> <title><![CDATA[Understanding Positional Embeddings in Modern LLMs]]></title> <description><![CDATA[Understanding Positional Embeddings in Modern LLMs. Comments]]></description> <media:thumbnail url="https://substackcdn.com/image/fetch/$s_!8RKL!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb461a2cb-11ab-474a-affe-0ee99e5a8782_2850x2777.png"/> <category><![CDATA[Technology]]></category> <link>https://en.mycoding.id/understanding-positional-embeddings-in-modern-llms-77185</link> <content:encoded><![CDATA[<div><div><div><div><div><div><div dir="auto"><p>This part explains how LLMs encode and procedure token stance data by covering traditional and contemporary methods akin RoPE. It&rsquo;s basically a thorough journey anywhere we volition activity through the mechanisms alongside ocular examples, a small math, and a small PyTorch implementation. And, of course, we volition afterward additionally appearance at how contemporary architectures modify or omit definitive positional encodings.</p><p>Below, you can discover a preview of which positional embedding choices are used in 2026 open-weight models. We volition conversation these choices afterward following covering the underlying mechanisms.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8RKL!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb461a2cb-11ab-474a-affe-0ee99e5a8782_2850x2777.png 424w, https://substackcdn.com/image/fetch/$s_!8RKL!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb461a2cb-11ab-474a-affe-0ee99e5a8782_2850x2777.png 848w, https://substackcdn.com/image/fetch/$s_!8RKL!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb461a2cb-11ab-474a-affe-0ee99e5a8782_2850x2777.png 1272w, https://substackcdn.com/image/fetch/$s_!8RKL!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb461a2cb-11ab-474a-affe-0ee99e5a8782_2850x2777.png 1456w" sizes="100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb461a2cb-11ab-474a-affe-0ee99e5a8782_2850x2777.png" width="1456" height="1419" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/b461a2cb-11ab-474a-affe-0ee99e5a8782_2850x2777.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1419,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 1 — exhibition 2026 collage&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 1 — exhibition 2026 collage" title="Figure 1 — exhibition 2026 collage" srcset="https://substackcdn.com/image/fetch/$s_!8RKL!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb461a2cb-11ab-474a-affe-0ee99e5a8782_2850x2777.png 424w, https://substackcdn.com/image/fetch/$s_!8RKL!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb461a2cb-11ab-474a-affe-0ee99e5a8782_2850x2777.png 848w, https://substackcdn.com/image/fetch/$s_!8RKL!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb461a2cb-11ab-474a-affe-0ee99e5a8782_2850x2777.png 1272w, https://substackcdn.com/image/fetch/$s_!8RKL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb461a2cb-11ab-474a-affe-0ee99e5a8782_2850x2777.png 1456w" sizes="100vw" fetchpriority="high"><div></div></div></a><figcaption><span>Figure 1: A preview of the gallery&rsquo;s 2026 architectures, grouped by positional encoding. Architecture figures taken from my </span><a href="https://en.mycoding.id/category/technology" target="_blank">LLM gallery</a><span>.</span></figcaption></figure></div><p>But why spending so much period writing/reading concerning this fairly insignificant construction block? LLMs keep evolving, and as I was preparing a new part on the latest LLM architecture trends this year, I realized that I never covered the positional data methods used in LLMs in any part so far. So, alternatively of letting the another part get out of hand, I idea it power create awareness to publish this as a standalone construction obstacle for the upcoming LLM architecture and trends analysis.</p><p>Also, complete the years, many readers have asked me for a tutorial on related positional embeddings (aka RoPE) to prosecute up on the learned complete embeddings used in GPT. It&rsquo;s been on my TODO catalog for a lengthy time, so I eventually revisited my old outline and put this walkthrough together, which volition cover</p><ul><li><p>different positional methods in LLMs;</p></li><li><p>(finally) how RoPE works;</p></li><li><p>what methods are used by latest open LLMs.</p></li></ul><p><span>Note that this part assumes several familiarity alongside essential LLM architectures and plant fine as a follow-up to </span><a href="https://en.mycoding.id/category/technology" target="_blank">Build a Large Language Model (From Scratch)</a><span>. If the concepts and computation of the notice internals, particularly query, key, and value vectors, are a small fuzzy, I propose checking my </span><a href="https://en.mycoding.id/category/technology" target="_blank">self-attention walkthrough</a><span> as a refresher:</span></p><p>The part is additionally extremely specialized and apt too detailed for some. However, I really wanted to have this basis out there for forthcoming articles to reference. And no worries, the next one volition come back to a additional comfortable style.</p><p><strong><span>PS: I started experimenting alongside Substack&rsquo;s new &ldquo;perks&rdquo; system. As a thank-you to supporters, I&rsquo;ve added </span><a href="https://en.mycoding.id/category/technology" target="_blank">image exports and metadata</a><span> from the architecture exhibition for you to use in your own analyses.</span></strong></p><p><span>PPS: If you are already acquainted alongside how stance embeddings work, awareness liberated to fast-forward to Section 6, </span><em>Positional choices in contemporary architectures</em><span>.</span></p><p><span>Note that this part may be finest peruse in a web browser than an email client. I propose checking out the </span><a href="https://en.mycoding.id/category/technology" target="_blank">web perspective here</a><span>, which additionally has a navigation bar on the remaining side.</span></p><p>Let&rsquo;s commencement alongside the general architecture. A tokenizer encodes the input content into token IDs. Next, the input token embedding tier of an LLM maps all of these IDs to a vector of numbers. We normally mention to these entries in that vector as &ldquo;features&rdquo; (similar to traditional ML). The transformer blocks inner the LLM afterward modify these vectors using notice and feedforward layers. Lastly, the model&rsquo;s output tier produces logit scores (converted into probabilities) for the next token.</p><p>The following shows this way in GPT-2 (a comparatively traditional LLM) and MiniMax-M3 (a additional latest LLM released this year; a placeholder for most another latest LLMs). As we can see, the architectures are not too distinct from all other, but before we get to the positional data parts, let&rsquo;s concentration on the orange blocks for now (the token embedding layers) and the transformer blocks complete them; the another architecture differences aren&rsquo;t needed for this explanation.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!3vjo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ab287c2-c3ba-46db-9ed0-1e6eacf78f43_2592x2521.png 424w, https://substackcdn.com/image/fetch/$s_!3vjo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ab287c2-c3ba-46db-9ed0-1e6eacf78f43_2592x2521.png 848w, https://substackcdn.com/image/fetch/$s_!3vjo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ab287c2-c3ba-46db-9ed0-1e6eacf78f43_2592x2521.png 1272w, https://substackcdn.com/image/fetch/$s_!3vjo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ab287c2-c3ba-46db-9ed0-1e6eacf78f43_2592x2521.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ab287c2-c3ba-46db-9ed0-1e6eacf78f43_2592x2521.png" width="1456" height="1416" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/4ab287c2-c3ba-46db-9ed0-1e6eacf78f43_2592x2521.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1416,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 2 — token embedding architectures&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 2 — token embedding architectures" title="Figure 2 — token embedding architectures" srcset="https://substackcdn.com/image/fetch/$s_!3vjo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ab287c2-c3ba-46db-9ed0-1e6eacf78f43_2592x2521.png 424w, https://substackcdn.com/image/fetch/$s_!3vjo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ab287c2-c3ba-46db-9ed0-1e6eacf78f43_2592x2521.png 848w, https://substackcdn.com/image/fetch/$s_!3vjo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ab287c2-c3ba-46db-9ed0-1e6eacf78f43_2592x2521.png 1272w, https://substackcdn.com/image/fetch/$s_!3vjo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ab287c2-c3ba-46db-9ed0-1e6eacf78f43_2592x2521.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 2: Token embedding layers in GPT-2 and MiniMax-M3, highlighted in orange. (Side note: M3&rsquo;s archetypal three out of 60 feedforward layers are compact not MoEs.)</figcaption></figure></div><p>A token embedding layer, shown in the fig above, maps all token ID to a vector. Position data is afterward added afterward. Concretely, and as we volition conversation in additional item later, GPT-2 adds a stance vector to all token embedding before the modified embedding enters the archetypal transformer block. On the another hand, MiniMax-M3 adds the positional data in all transformer obstacle using a method called RoPE inner attention.</p><p>We volition conversation concerning these methods in additional item later, but first, let&rsquo;s commencement alongside the big image data explaining why definitive positional data is added or needed.</p><p>Consider the following two input sentences as a uncomplicated example:</p><ol><li><p>&ldquo;The red fox jumps complete the gray fox.&rdquo;</p></li><li><p>&ldquo;The gray fox jumps complete the red fox.&rdquo;</p></li></ol><p>Changing the term command changes which fox does the jumping, as illustrated in the following figure.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!KoQ2!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8ff5039-8ec8-447c-a444-0345619c8b7b_1846x1304.png 424w, https://substackcdn.com/image/fetch/$s_!KoQ2!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8ff5039-8ec8-447c-a444-0345619c8b7b_1846x1304.png 848w, https://substackcdn.com/image/fetch/$s_!KoQ2!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8ff5039-8ec8-447c-a444-0345619c8b7b_1846x1304.png 1272w, https://substackcdn.com/image/fetch/$s_!KoQ2!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8ff5039-8ec8-447c-a444-0345619c8b7b_1846x1304.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8ff5039-8ec8-447c-a444-0345619c8b7b_1846x1304.png" width="536" height="378.8076923076923" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/b8ff5039-8ec8-447c-a444-0345619c8b7b_1846x1304.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1029,&quot;width&quot;:1456,&quot;resizeWidth&quot;:536,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 3 — term order&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 3 — term order" title="Figure 3 — term order" srcset="https://substackcdn.com/image/fetch/$s_!KoQ2!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8ff5039-8ec8-447c-a444-0345619c8b7b_1846x1304.png 424w, https://substackcdn.com/image/fetch/$s_!KoQ2!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8ff5039-8ec8-447c-a444-0345619c8b7b_1846x1304.png 848w, https://substackcdn.com/image/fetch/$s_!KoQ2!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8ff5039-8ec8-447c-a444-0345619c8b7b_1846x1304.png 1272w, https://substackcdn.com/image/fetch/$s_!KoQ2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8ff5039-8ec8-447c-a444-0345619c8b7b_1846x1304.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 3: Changing the term command changes which fox jumps.</figcaption></figure></div><p>For the examples in this article, we volition treat all term as one token for simplicity. (As you know, genuine tokenizers can divided words into smaller subword tokens.) The archetypal declaration afterward has eight token positions, from 0 through 7.</p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Bxjt!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5386e76-8459-4196-a797-430e922fce3b_2238x580.png 424w, https://substackcdn.com/image/fetch/$s_!Bxjt!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5386e76-8459-4196-a797-430e922fce3b_2238x580.png 848w, https://substackcdn.com/image/fetch/$s_!Bxjt!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5386e76-8459-4196-a797-430e922fce3b_2238x580.png 1272w, https://substackcdn.com/image/fetch/$s_!Bxjt!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5386e76-8459-4196-a797-430e922fce3b_2238x580.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5386e76-8459-4196-a797-430e922fce3b_2238x580.png" width="582" height="150.69642857142858" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/f5386e76-8459-4196-a797-430e922fce3b_2238x580.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:377,&quot;width&quot;:1456,&quot;resizeWidth&quot;:582,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 4 — token ids and positions&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 4 — token ids and positions" title="Figure 4 — token ids and positions" srcset="https://substackcdn.com/image/fetch/$s_!Bxjt!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5386e76-8459-4196-a797-430e922fce3b_2238x580.png 424w, https://substackcdn.com/image/fetch/$s_!Bxjt!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5386e76-8459-4196-a797-430e922fce3b_2238x580.png 848w, https://substackcdn.com/image/fetch/$s_!Bxjt!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5386e76-8459-4196-a797-430e922fce3b_2238x580.png 1272w, https://substackcdn.com/image/fetch/$s_!Bxjt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5386e76-8459-4196-a797-430e922fce3b_2238x580.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div><figcaption>Figure 4: Token IDs and positions in the toy sentence.</figcaption></figure></div><p><span>(If you are curious in how vocabularies are built, inspect out my </span><a href="https://en.mycoding.id/category/technology" target="_blank">tokenizer materials on GitHub</a><span>.)</span></p><p>Note that &ldquo;fox&rdquo;, which appears twice in the input sentence, gets the identical token ID 2 in the two cases. And since they the two have the identical token ID, irrespective of the stance in the input sentence, they would get the identical embedding vector from the embedding layer.</p><p>Now, let&rsquo;s regard a regular self-attention scheme alongside no positional encoding (and no causal mask), anywhere all input token can act to all another token in the input. In this scenario, swapping any two words, akin &ldquo;red&rdquo; and &ldquo;gray&rdquo;, would motionless outcome in the identical embeddings and notice scores between all token pair.</p><p>GPT-style models do use a causal (autoregressive) mask, which restricts all token to itself and before tokens. Interestingly, this causal disguise gives the computation several implied command information. In our example, &ldquo;jumps&rdquo; can act to &ldquo;red&rdquo; in the archetypal declaration and &ldquo;gray&rdquo; in the second. So, a causal LLM can theoretically already differentiate these sentences equal without definitive positional embeddings. (We volition come back to this idea afterward whenever discussing the &ldquo;NoPE&rdquo; concept.)</p><p>Nonetheless, GPT-2 and most newer LLMs contain definitive positional data since it gives notice a straightforward indication concerning positions. That&rsquo;s since equal although the causal disguise determines which tokens are accessible and provides implied command information, it doesn&rsquo;t differentiate a separation of one stance from a separation of ten, for example. So, definitive positional methods are motionless helpful (or equal essential), since they provision that information.</p><p>If this all echoed confusing, don&rsquo;t worry, it was additional meant as an overview, and we volition stage through these concepts a bit additional gradually in the remaining article.</p><p><span>We briefly talked concerning </span><em>implicit</em><span> positional data through causal notice masks (and we volition revisit this again in additional item later). However, we now toggle gears and conversation concerning the </span><em>explicit</em><span> ways to add positional information.</span></p><p>The perchance simplest and most straightforward (but not &ldquo;cheapest&rdquo;) method is to provision all sequence stance its own vector. We add that vector to the token embedding, so the resulting, modified embedding vector is afterward passed to the archetypal transformer block. This exact procedure is discussed in the following subsections.</p><p>To exemplify the use of an explicit, learned positional vector to the input embeddings, let&rsquo;s regard GPT-2 as a tangible example. GPT-2 learns a array of stance vectors during training. The following fig illustrates anywhere these vectors are used in the architecture, that is, immediately following the token embedding lookup and before the archetypal transformer block.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!GDv8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda2b0f15-7f45-4aa4-8f1b-a0777a0da4fc_1710x2197.png 424w, https://substackcdn.com/image/fetch/$s_!GDv8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda2b0f15-7f45-4aa4-8f1b-a0777a0da4fc_1710x2197.png 848w, https://substackcdn.com/image/fetch/$s_!GDv8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda2b0f15-7f45-4aa4-8f1b-a0777a0da4fc_1710x2197.png 1272w, https://substackcdn.com/image/fetch/$s_!GDv8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda2b0f15-7f45-4aa4-8f1b-a0777a0da4fc_1710x2197.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda2b0f15-7f45-4aa4-8f1b-a0777a0da4fc_1710x2197.png" width="514" height="660.5041208791209" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/da2b0f15-7f45-4aa4-8f1b-a0777a0da4fc_1710x2197.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1871,&quot;width&quot;:1456,&quot;resizeWidth&quot;:514,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 5 — gpt2 architecture&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 5 — gpt2 architecture" title="Figure 5 — gpt2 architecture" srcset="https://substackcdn.com/image/fetch/$s_!GDv8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda2b0f15-7f45-4aa4-8f1b-a0777a0da4fc_1710x2197.png 424w, https://substackcdn.com/image/fetch/$s_!GDv8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda2b0f15-7f45-4aa4-8f1b-a0777a0da4fc_1710x2197.png 848w, https://substackcdn.com/image/fetch/$s_!GDv8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda2b0f15-7f45-4aa4-8f1b-a0777a0da4fc_1710x2197.png 1272w, https://substackcdn.com/image/fetch/$s_!GDv8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda2b0f15-7f45-4aa4-8f1b-a0777a0da4fc_1710x2197.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption><span>Figure 5: GPT-2 adds complete stance data before the transformer blocks, as highlighted in purple. The dimensions shown are for </span><a href="https://en.mycoding.id/category/technology" target="_blank">GPT-2 XL</a><span>.</span></figcaption></figure></div><p>The positional embedding tier depicted in the fig complete follows the identical implementation as the token embedding layer, i.e., they have exactly the identical output dimensions so that the resulting input vectors can be added to all other. The code below shows that alongside an excerpt of a GPT-2 code implementation:</p><p><span>And in mathematical terms, we can compose it as follows. Let </span><em>m</em><span> be a stance indicator and </span><em><span>x</span><sub>m</sub></em><span> the token ID at that position. The input to the archetypal obstacle is</span></p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!AjJR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 424w, https://substackcdn.com/image/fetch/$s_!AjJR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 848w, https://substackcdn.com/image/fetch/$s_!AjJR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 1272w, https://substackcdn.com/image/fetch/$s_!AjJR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png" width="1456" height="51" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:51,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 1: h_m = E_{\\text{token}}(x_m) + P_m.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt=" h_m = E_{\text{token}}(x_m) + P_m." title=" h_m = E_{\text{token}}(x_m) + P_m." srcset="https://substackcdn.com/image/fetch/$s_!AjJR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 424w, https://substackcdn.com/image/fetch/$s_!AjJR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 848w, https://substackcdn.com/image/fetch/$s_!AjJR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 1272w, https://substackcdn.com/image/fetch/$s_!AjJR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p>Here,</p><ul><li><p><em>E</em><sub>token</sub><span> is the embedding layer,</span></p></li><li><p><em><span>P</span><sub>m</sub></em><span> is the stance info vector selected by </span><em>m</em><span>,</span></p></li><li><p><span>and </span><em><span>h</span><sub>m</sub></em><span> is the resulting summed embedding and stance vector.</span></p></li></ul><p>The two vectors have the identical width, so we add their entries element-wise. Figure 6 below shows this visually alongside a small numerical example.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!yuT0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F546edc50-7b1a-40ba-a125-da3fbdaade26_2586x964.png 424w, https://substackcdn.com/image/fetch/$s_!yuT0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F546edc50-7b1a-40ba-a125-da3fbdaade26_2586x964.png 848w, https://substackcdn.com/image/fetch/$s_!yuT0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F546edc50-7b1a-40ba-a125-da3fbdaade26_2586x964.png 1272w, https://substackcdn.com/image/fetch/$s_!yuT0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F546edc50-7b1a-40ba-a125-da3fbdaade26_2586x964.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F546edc50-7b1a-40ba-a125-da3fbdaade26_2586x964.png" width="1456" height="543" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/546edc50-7b1a-40ba-a125-da3fbdaade26_2586x964.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:543,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 6 — complete embeddings&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 6 — complete embeddings" title="Figure 6 — complete embeddings" srcset="https://substackcdn.com/image/fetch/$s_!yuT0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F546edc50-7b1a-40ba-a125-da3fbdaade26_2586x964.png 424w, https://substackcdn.com/image/fetch/$s_!yuT0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F546edc50-7b1a-40ba-a125-da3fbdaade26_2586x964.png 848w, https://substackcdn.com/image/fetch/$s_!yuT0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F546edc50-7b1a-40ba-a125-da3fbdaade26_2586x964.png 1272w, https://substackcdn.com/image/fetch/$s_!yuT0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F546edc50-7b1a-40ba-a125-da3fbdaade26_2586x964.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 6: Token and stance embeddings are added element-wise. (I am following the gathering current that neural network diagrams are normally peruse from base to top.)</figcaption></figure></div><p><span>In the fig above, at stance 0, adding the token embedding </span>[1, 1, 1]<span> to the stance embedding </span>[1.1, 1.2, 1.3]<span> gives </span>[2.1, 2.2, 2.3]<span>. By the way, the values in the fig are chosen to keep the example simple; in a genuine example, the two embedding layers (tables) are initialized alongside small random values and learned during training.</span></p><p><span>So, what we covered in this division is a so-called </span><em>absolute</em><span> positional method since the positional embedding </span><em><span>P</span><sub>m</sub></em><span> depends on the token&rsquo;s own stance </span><em>m</em><span>. In the former example sentence, the two &ldquo;fox&rdquo; tokens have initially the identical token embeddings, but since we are adding distinct positional vectors </span><em><span>P</span><sub>2</sub></em><span> and </span><em><span>P</span><sub>7</sub></em><span>, they have distinct values before they act the archetypal transformer block. And consequently, the query, key, and value vectors inner the notice mechanisms afterward additionally obtain representations that already contain stance information.</span></p><p><span>Note that the first </span><a href="https://en.mycoding.id/category/technology" target="_blank">GPT-2 implementation</a><span> has 1,024 learned stance vectors. This method it lone plant alongside inputs alongside 1024 or small tokens. If we desire to broaden the environment window, we would have to broaden this embedding table. And since the embedding array (layer) is updated akin any another tier during training, the environment dimension would have to be decided before the training starts.</span></p><p>Extending the environment would be extremely costly since the array can rotate into extremely large. For example, in the case of GPT-2 Medium, the positional embedding tier weights lone create up 0.30% of the total weights at 1,024 positions. However, if we risen the environment dimension to a nowadays average 100k range, they would create up &gt;22% of the total example weights, which would be extremely uneconomic.</p><p><span>The first </span><a href="https://en.mycoding.id/category/technology" target="_blank">&ldquo;Attention Is All You Need&rdquo;</a><span> transformer from 2017, which predates GPT, uses the identical supplement mechanism, but computes its stance vectors alongside fixed sine and cosine functions. Figure 7 below shows this supplement before the two the encoder and decoder stacks. The encoder processes the input sequence, and the decoder generates the output sequence. (The logic why it&rsquo;s introduced current is that the sine and cosine functions volition rotate into applicable again afterward whenever discussing RoPE.)</span></p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!d7HH!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5958afa7-673d-44df-a1af-112044196dde_2620x2800.png 424w, https://substackcdn.com/image/fetch/$s_!d7HH!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5958afa7-673d-44df-a1af-112044196dde_2620x2800.png 848w, https://substackcdn.com/image/fetch/$s_!d7HH!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5958afa7-673d-44df-a1af-112044196dde_2620x2800.png 1272w, https://substackcdn.com/image/fetch/$s_!d7HH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5958afa7-673d-44df-a1af-112044196dde_2620x2800.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5958afa7-673d-44df-a1af-112044196dde_2620x2800.png" width="644" height="688.2307692307693" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/5958afa7-673d-44df-a1af-112044196dde_2620x2800.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1556,&quot;width&quot;:1456,&quot;resizeWidth&quot;:644,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 7 — first transformer architecture&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 7 — first transformer architecture" title="Figure 7 — first transformer architecture" srcset="https://substackcdn.com/image/fetch/$s_!d7HH!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5958afa7-673d-44df-a1af-112044196dde_2620x2800.png 424w, https://substackcdn.com/image/fetch/$s_!d7HH!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5958afa7-673d-44df-a1af-112044196dde_2620x2800.png 848w, https://substackcdn.com/image/fetch/$s_!d7HH!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5958afa7-673d-44df-a1af-112044196dde_2620x2800.png 1272w, https://substackcdn.com/image/fetch/$s_!d7HH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5958afa7-673d-44df-a1af-112044196dde_2620x2800.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 7: Sinusoidal stance encodings added to the token embeddings as highlighted in purple.</figcaption></figure></div><p><span>The first transformer additionally scales its token embeddings by &radic;(</span><em>d</em><sub>model</sub><span>) before adding the stance vectors, anywhere </span><em>d</em><sub>model</sub><span> is the embedding width.</span></p><p>So, during GPT-2 in the former division had the formula</p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!AjJR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 424w, https://substackcdn.com/image/fetch/$s_!AjJR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 848w, https://substackcdn.com/image/fetch/$s_!AjJR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 1272w, https://substackcdn.com/image/fetch/$s_!AjJR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png" width="1456" height="51" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:51,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 2: h_m = E_{\\text{token}}(x_m) + P_m.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt=" h_m = E_{\text{token}}(x_m) + P_m." title=" h_m = E_{\text{token}}(x_m) + P_m." srcset="https://substackcdn.com/image/fetch/$s_!AjJR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 424w, https://substackcdn.com/image/fetch/$s_!AjJR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 848w, https://substackcdn.com/image/fetch/$s_!AjJR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 1272w, https://substackcdn.com/image/fetch/$s_!AjJR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10fa6ac9-e4e7-4b16-b4c3-43d1671030c3_2240x79.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p>the equation used in the first transformer (depicted in Figure 7) is:</p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!08dk!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa42a0af2-997b-49a7-87e6-3beaf7788bae_2240x95.png 424w, https://substackcdn.com/image/fetch/$s_!08dk!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa42a0af2-997b-49a7-87e6-3beaf7788bae_2240x95.png 848w, https://substackcdn.com/image/fetch/$s_!08dk!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa42a0af2-997b-49a7-87e6-3beaf7788bae_2240x95.png 1272w, https://substackcdn.com/image/fetch/$s_!08dk!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa42a0af2-997b-49a7-87e6-3beaf7788bae_2240x95.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa42a0af2-997b-49a7-87e6-3beaf7788bae_2240x95.png" width="1456" height="62" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/a42a0af2-997b-49a7-87e6-3beaf7788bae_2240x95.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:62,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 3: h_m = \\sqrt{d_{\\text{model}}} E_{\\text{token}}(x_m) + P_m.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt=" h_m = \sqrt{d_{\text{model}}} E_{\text{token}}(x_m) + P_m." title=" h_m = \sqrt{d_{\text{model}}} E_{\text{token}}(x_m) + P_m." srcset="https://substackcdn.com/image/fetch/$s_!08dk!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa42a0af2-997b-49a7-87e6-3beaf7788bae_2240x95.png 424w, https://substackcdn.com/image/fetch/$s_!08dk!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa42a0af2-997b-49a7-87e6-3beaf7788bae_2240x95.png 848w, https://substackcdn.com/image/fetch/$s_!08dk!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa42a0af2-997b-49a7-87e6-3beaf7788bae_2240x95.png 1272w, https://substackcdn.com/image/fetch/$s_!08dk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa42a0af2-997b-49a7-87e6-3beaf7788bae_2240x95.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p><span>This scaling by &radic;(</span><em>d</em><sub>model</sub><span>) is a insignificant specialized detail, but for those who are curious, the motivation rearward this scaling aspect is not explained in the transformer paper, but as I discussed alongside colleagues on a Twitter thread many years ago, it&rsquo;s apt to put the token embeddings and sinusoidal positional encodings on comparable numerical scales at initialization, so the stance vectors don&rsquo;t dominate the sum whenever we add them. </span></p><p>In the GPT-2 case, this scaling wasn&rsquo;t necessary since the first GPT-2 code initialized the token embeddings to have an first norm deviation of 0.02 and the stance embeddings to have an first norm deviation of 0.01 so that they were already approximately on the identical scale.</p><p><span>Now, the additional engaging item is how the stance embeddings </span><em><span>P</span><sub>m</sub></em><span> are computed.</span></p><p>Concretely, all stance vector has the following entries:</p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Z3Wh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00bc80dd-1b3c-4c66-90c0-111bfaf26d2d_2240x296.png 424w, https://substackcdn.com/image/fetch/$s_!Z3Wh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00bc80dd-1b3c-4c66-90c0-111bfaf26d2d_2240x296.png 848w, https://substackcdn.com/image/fetch/$s_!Z3Wh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00bc80dd-1b3c-4c66-90c0-111bfaf26d2d_2240x296.png 1272w, https://substackcdn.com/image/fetch/$s_!Z3Wh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00bc80dd-1b3c-4c66-90c0-111bfaf26d2d_2240x296.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00bc80dd-1b3c-4c66-90c0-111bfaf26d2d_2240x296.png" width="1456" height="192" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/00bc80dd-1b3c-4c66-90c0-111bfaf26d2d_2240x296.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:192,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 4: \\begin{aligned} P_{m,2i} &amp;= \\sin\\left(\\frac{m}{10000^{2i/d_{\\text{model}}}}\\right), \\\\ P_{m,2i+1} &amp;= \\cos\\left(\\frac{m}{10000^{2i/d_{\\text{model}}}}\\right). \\end{aligned}&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt=" \begin{aligned} P_{m,2i} &amp;= \sin\left(\frac{m}{10000^{2i/d_{\text{model}}}}\right), \\ P_{m,2i+1} &amp;= \cos\left(\frac{m}{10000^{2i/d_{\text{model}}}}\right). \end{aligned}" title=" \begin{aligned} P_{m,2i} &amp;= \sin\left(\frac{m}{10000^{2i/d_{\text{model}}}}\right), \\ P_{m,2i+1} &amp;= \cos\left(\frac{m}{10000^{2i/d_{\text{model}}}}\right). \end{aligned}" srcset="https://substackcdn.com/image/fetch/$s_!Z3Wh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00bc80dd-1b3c-4c66-90c0-111bfaf26d2d_2240x296.png 424w, https://substackcdn.com/image/fetch/$s_!Z3Wh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00bc80dd-1b3c-4c66-90c0-111bfaf26d2d_2240x296.png 848w, https://substackcdn.com/image/fetch/$s_!Z3Wh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00bc80dd-1b3c-4c66-90c0-111bfaf26d2d_2240x296.png 1272w, https://substackcdn.com/image/fetch/$s_!Z3Wh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00bc80dd-1b3c-4c66-90c0-111bfaf26d2d_2240x296.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p><span>As before, </span><em>m</em><span> denotes the position. The indicator </span><em>i</em><span> indexes a brace of features inside the vector.</span></p><p><span>Here, a characteristic is one figure in the stance vector. For example, let&rsquo;s assume we have a small vector alongside 8 values (</span><em>d</em><sub>model</sub><span> = 8) that has entries numbered 0 through 7. (In the first document this was 512 for the small example and 1024 for the bigger model, alternatively of 8, but that would be difficult to visualize.) We collection these into four pairs so that </span><em>i</em><span> goes from 0 through 3.</span></p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HkhF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe441fd8e-780b-4f66-9254-8ee2a7994a4f_1593x773.png 424w, https://substackcdn.com/image/fetch/$s_!HkhF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe441fd8e-780b-4f66-9254-8ee2a7994a4f_1593x773.png 848w, https://substackcdn.com/image/fetch/$s_!HkhF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe441fd8e-780b-4f66-9254-8ee2a7994a4f_1593x773.png 1272w, https://substackcdn.com/image/fetch/$s_!HkhF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe441fd8e-780b-4f66-9254-8ee2a7994a4f_1593x773.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe441fd8e-780b-4f66-9254-8ee2a7994a4f_1593x773.png" width="522" height="253.47115384615384" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/e441fd8e-780b-4f66-9254-8ee2a7994a4f_1593x773.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:707,&quot;width&quot;:1456,&quot;resizeWidth&quot;:522,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 8 — sinusoidal characteristic pairs&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 8 — sinusoidal characteristic pairs" title="Figure 8 — sinusoidal characteristic pairs" srcset="https://substackcdn.com/image/fetch/$s_!HkhF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe441fd8e-780b-4f66-9254-8ee2a7994a4f_1593x773.png 424w, https://substackcdn.com/image/fetch/$s_!HkhF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe441fd8e-780b-4f66-9254-8ee2a7994a4f_1593x773.png 848w, https://substackcdn.com/image/fetch/$s_!HkhF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe441fd8e-780b-4f66-9254-8ee2a7994a4f_1593x773.png 1272w, https://substackcdn.com/image/fetch/$s_!HkhF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe441fd8e-780b-4f66-9254-8ee2a7994a4f_1593x773.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 8: Sine and cosine entries for all characteristic pair.</figcaption></figure></div><p><span>Here, the indices 2</span><em>i</em><span> and 2</span><em>i</em><span> + 1 choose the two entries corresponding to brace </span><em>i</em><span>.</span></p><p><span>As shown in the equation earlier, the viewpoint at stance </span><em>m</em><span>, which is afterward fed into the sine or cosine functions to get </span><em><span>P</span><sub>m</sub></em><sub>, 2</sub><em><sub>i</sub></em><span> or </span><em><span>P</span><sub>m</sub></em><sub>, 2</sub><em><sub>i</sub></em><sub> + 1</sub><span>, respectively, is computed as follows:</span></p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9Nx0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae693dce-b196-480f-ace0-2d8a888bf745_2234x806.png 424w, https://substackcdn.com/image/fetch/$s_!9Nx0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae693dce-b196-480f-ace0-2d8a888bf745_2234x806.png 848w, https://substackcdn.com/image/fetch/$s_!9Nx0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae693dce-b196-480f-ace0-2d8a888bf745_2234x806.png 1272w, https://substackcdn.com/image/fetch/$s_!9Nx0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae693dce-b196-480f-ace0-2d8a888bf745_2234x806.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae693dce-b196-480f-ace0-2d8a888bf745_2234x806.png" width="582" height="209.85576923076923" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/ae693dce-b196-480f-ace0-2d8a888bf745_2234x806.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:525,&quot;width&quot;:1456,&quot;resizeWidth&quot;:582,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 9 — viewpoint and frequency&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 9 — viewpoint and frequency" title="Figure 9 — viewpoint and frequency" srcset="https://substackcdn.com/image/fetch/$s_!9Nx0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae693dce-b196-480f-ace0-2d8a888bf745_2234x806.png 424w, https://substackcdn.com/image/fetch/$s_!9Nx0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae693dce-b196-480f-ace0-2d8a888bf745_2234x806.png 848w, https://substackcdn.com/image/fetch/$s_!9Nx0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae693dce-b196-480f-ace0-2d8a888bf745_2234x806.png 1272w, https://substackcdn.com/image/fetch/$s_!9Nx0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae693dce-b196-480f-ace0-2d8a888bf745_2234x806.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div><figcaption>Figure 9: The viewpoint inner sine and cosine is the token stance multiplied by the pair&rsquo;s angular frequency.</figcaption></figure></div><p><span>Figure 10 shows this calculation for all four pairs of our 8-feature example at stance </span><em>m</em><span> = 2.</span></p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!63Px!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6e0a4d2-6a7c-443d-9be0-102807f8da7a_2600x1063.png 424w, https://substackcdn.com/image/fetch/$s_!63Px!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6e0a4d2-6a7c-443d-9be0-102807f8da7a_2600x1063.png 848w, https://substackcdn.com/image/fetch/$s_!63Px!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6e0a4d2-6a7c-443d-9be0-102807f8da7a_2600x1063.png 1272w, https://substackcdn.com/image/fetch/$s_!63Px!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6e0a4d2-6a7c-443d-9be0-102807f8da7a_2600x1063.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6e0a4d2-6a7c-443d-9be0-102807f8da7a_2600x1063.png" width="1456" height="595" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/a6e0a4d2-6a7c-443d-9be0-102807f8da7a_2600x1063.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:595,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 10 — sinusoidal calculations&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 10 — sinusoidal calculations" title="Figure 10 — sinusoidal calculations" srcset="https://substackcdn.com/image/fetch/$s_!63Px!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6e0a4d2-6a7c-443d-9be0-102807f8da7a_2600x1063.png 424w, https://substackcdn.com/image/fetch/$s_!63Px!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6e0a4d2-6a7c-443d-9be0-102807f8da7a_2600x1063.png 848w, https://substackcdn.com/image/fetch/$s_!63Px!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6e0a4d2-6a7c-443d-9be0-102807f8da7a_2600x1063.png 1272w, https://substackcdn.com/image/fetch/$s_!63Px!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6e0a4d2-6a7c-443d-9be0-102807f8da7a_2600x1063.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 10: The four sine and cosine pairs at stance 2.</figcaption></figure></div><p>For an eight-feature embedding, the four frequencies are 1, 0.1, 0.01, and 0.001 radians per position. At stance 2, the archetypal brace in row 1 is (sin 2, cos 2) &asymp; (0.909, &minus;0.416).</p><p>The second pair, in row 2, changes additional slowly, giving (sin 0.2, cos 0.2) &asymp; (0.199, 0.980).</p><p>The fig below visualizes the sine and cosine calculations for distinct pairs, which are afterward collected in a stance vector.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!UhK9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2fd5a766-4363-4189-9675-bcf9bd8e700d_2624x2023.png 424w, https://substackcdn.com/image/fetch/$s_!UhK9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2fd5a766-4363-4189-9675-bcf9bd8e700d_2624x2023.png 848w, https://substackcdn.com/image/fetch/$s_!UhK9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2fd5a766-4363-4189-9675-bcf9bd8e700d_2624x2023.png 1272w, https://substackcdn.com/image/fetch/$s_!UhK9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2fd5a766-4363-4189-9675-bcf9bd8e700d_2624x2023.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2fd5a766-4363-4189-9675-bcf9bd8e700d_2624x2023.png" width="1456" height="1123" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/2fd5a766-4363-4189-9675-bcf9bd8e700d_2624x2023.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1123,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 11 — sinusoidal encoding&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 11 — sinusoidal encoding" title="Figure 11 — sinusoidal encoding" srcset="https://substackcdn.com/image/fetch/$s_!UhK9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2fd5a766-4363-4189-9675-bcf9bd8e700d_2624x2023.png 424w, https://substackcdn.com/image/fetch/$s_!UhK9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2fd5a766-4363-4189-9675-bcf9bd8e700d_2624x2023.png 848w, https://substackcdn.com/image/fetch/$s_!UhK9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2fd5a766-4363-4189-9675-bcf9bd8e700d_2624x2023.png 1272w, https://substackcdn.com/image/fetch/$s_!UhK9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2fd5a766-4363-4189-9675-bcf9bd8e700d_2624x2023.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 11: Sampling sine and cosine functions to form a stance vector.</figcaption></figure></div><p>In short, this method is a (more complicated) substitute to obtaining positional embedding vectors compared to the GPT-2 method discussed earlier.</p><p>Similar to the GPT-2 embeddings, these are additionally complete embeddings since all vector is computed from its own position. However, in difference to the GPT-2 approach, the sine and cosine functions remain fixed during training. (Also, they can be extended and evaluated beyond the training positions, akin beyond the environment dimension of 1024, although that solitary doesn&rsquo;t justify that the example volition execute fine there, since it hasn&rsquo;t seen those during training.)</p><p>To summarize, Figure 12 below compares the two ways of obtaining a stance vector we discussed so far.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!jj3h!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3032a17a-5c43-487e-bcce-560d0a957b12_2616x1804.png 424w, https://substackcdn.com/image/fetch/$s_!jj3h!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3032a17a-5c43-487e-bcce-560d0a957b12_2616x1804.png 848w, https://substackcdn.com/image/fetch/$s_!jj3h!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3032a17a-5c43-487e-bcce-560d0a957b12_2616x1804.png 1272w, https://substackcdn.com/image/fetch/$s_!jj3h!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3032a17a-5c43-487e-bcce-560d0a957b12_2616x1804.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3032a17a-5c43-487e-bcce-560d0a957b12_2616x1804.png" width="1456" height="1004" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/3032a17a-5c43-487e-bcce-560d0a957b12_2616x1804.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1004,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 12 — fixed vs learned positions&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 12 — fixed vs learned positions" title="Figure 12 — fixed vs learned positions" srcset="https://substackcdn.com/image/fetch/$s_!jj3h!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3032a17a-5c43-487e-bcce-560d0a957b12_2616x1804.png 424w, https://substackcdn.com/image/fetch/$s_!jj3h!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3032a17a-5c43-487e-bcce-560d0a957b12_2616x1804.png 848w, https://substackcdn.com/image/fetch/$s_!jj3h!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3032a17a-5c43-487e-bcce-560d0a957b12_2616x1804.png 1272w, https://substackcdn.com/image/fetch/$s_!jj3h!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3032a17a-5c43-487e-bcce-560d0a957b12_2616x1804.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 12: Fixed sinusoidal and learned complete positional embeddings.</figcaption></figure></div><p>This sinusoidal method may appearance akin an overly complex way to get positional embeddings, however, understanding it volition be helpful whenever talking concerning RoPE in the next section.</p><p>Also, there is several rationale rearward it. The authors chose sine and cosine since they create the shifting by a certain offset gives a certain deterministic difference. For example, the difference between stance 2 to 5 and the difference between 102 to 105 is the two 3, which afterward results in the identical rotation.</p><p><span>According to the </span><a href="https://en.mycoding.id/category/technology" target="_blank">Attention Is All You Need</a><span> paper:</span></p><blockquote><p><span>&ldquo;...easily study to act by related positions, since for any fixed offset </span><em>k</em><span>, </span><em>PE</em><sub>pos + </sub><em><sub>k</sub></em><span> can be represented as a linear function of </span><em>PE</em><sub>pos</sub><span>.&rdquo;</span></p></blockquote><p>Or in small specialized terms, the authors hypothesized that this asset would create it easier for notice to study relationships specified as &ldquo;three tokens earlier.&rdquo;</p><p>Figure 13 below illustrates this for one sine/cosine pair. A rotation is a linear transformation, and the two three-position shifts use the identical rotation.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!PFDm!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cfa51c1-e3d5-4791-9899-4f159d43e890_2453x1299.png 424w, https://substackcdn.com/image/fetch/$s_!PFDm!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cfa51c1-e3d5-4791-9899-4f159d43e890_2453x1299.png 848w, https://substackcdn.com/image/fetch/$s_!PFDm!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cfa51c1-e3d5-4791-9899-4f159d43e890_2453x1299.png 1272w, https://substackcdn.com/image/fetch/$s_!PFDm!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cfa51c1-e3d5-4791-9899-4f159d43e890_2453x1299.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cfa51c1-e3d5-4791-9899-4f159d43e890_2453x1299.png" width="582" height="308.1881868131868" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/8cfa51c1-e3d5-4791-9899-4f159d43e890_2453x1299.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:771,&quot;width&quot;:1456,&quot;resizeWidth&quot;:582,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 13 — sinusoidal related shift&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 13 — sinusoidal related shift" title="Figure 13 — sinusoidal related shift" srcset="https://substackcdn.com/image/fetch/$s_!PFDm!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cfa51c1-e3d5-4791-9899-4f159d43e890_2453x1299.png 424w, https://substackcdn.com/image/fetch/$s_!PFDm!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cfa51c1-e3d5-4791-9899-4f159d43e890_2453x1299.png 848w, https://substackcdn.com/image/fetch/$s_!PFDm!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cfa51c1-e3d5-4791-9899-4f159d43e890_2453x1299.png 1272w, https://substackcdn.com/image/fetch/$s_!PFDm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cfa51c1-e3d5-4791-9899-4f159d43e890_2453x1299.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 13: For a brace alongside angular frequence 0.1 radians per position, a change of three positions continually rotates the brace by 0.3 radians.</figcaption></figure></div><p>So, the sinusoidal method complete is motionless considered an complete positional encoding since the vector added to all token embedding represents that token&rsquo;s stance counted from the commencement of the sequence.</p><p><span>As shown in the former figure, for example, tokens at positions 2 and 5 obtain the stance vectors </span><em><span>P</span><sub>2</sub></em><span> and </span><em><span>P</span><sub>5</sub></em><span>. If we move those tokens to positions 102 and 105, they&rsquo;re motionless three positions apart. But all token gets a distinct stance vector since its complete stance </span><em>m</em><span> has changed.</span></p><p>So their notice mark can alter equal whenever the token embeddings remain the same. The sinusoidal construction motionless gives the example a way to study related relationships.</p><p><span>Anyways, RoPE, which we volition conversation below, builds this related association immediately into the notice calculation. RoPE, which </span><a href="https://en.mycoding.id/category/technology" target="_blank">Su et al. introduced in the 2021 RoFormer paper</a><span>, stands for Rotary Position Embedding and is considered a related stance method.</span></p><p><span>In attention, all token&rsquo;s current depiction is projected into a query, a key, and a value vector. Query-key dot products decide notice scores. After masking and softmax, the resulting weights decide how the value vectors are mixed. For a refresher, see my </span><a href="https://en.mycoding.id/category/technology" target="_blank">Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs</a><span> article:</span></p><p>RoPE uses the token&rsquo;s stance to rotate its query and key vectors before their dot product. The value vectors bypass this rotation. Figure 14 below shows an overview of anywhere RoPE is applied in a comparatively latest MiniMax-M3, anywhere QK-Norm normalizes queries and keys before RoPE.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xNEB!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F309c52ca-ad27-4336-a463-cde0050b3425_1764x2311.png 424w, https://substackcdn.com/image/fetch/$s_!xNEB!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F309c52ca-ad27-4336-a463-cde0050b3425_1764x2311.png 848w, https://substackcdn.com/image/fetch/$s_!xNEB!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F309c52ca-ad27-4336-a463-cde0050b3425_1764x2311.png 1272w, https://substackcdn.com/image/fetch/$s_!xNEB!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F309c52ca-ad27-4336-a463-cde0050b3425_1764x2311.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F309c52ca-ad27-4336-a463-cde0050b3425_1764x2311.png" width="546" height="715.125" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/309c52ca-ad27-4336-a463-cde0050b3425_1764x2311.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1907,&quot;width&quot;:1456,&quot;resizeWidth&quot;:546,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 14 — minimax m3 architecture&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 14 — minimax m3 architecture" title="Figure 14 — minimax m3 architecture" srcset="https://substackcdn.com/image/fetch/$s_!xNEB!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F309c52ca-ad27-4336-a463-cde0050b3425_1764x2311.png 424w, https://substackcdn.com/image/fetch/$s_!xNEB!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F309c52ca-ad27-4336-a463-cde0050b3425_1764x2311.png 848w, https://substackcdn.com/image/fetch/$s_!xNEB!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F309c52ca-ad27-4336-a463-cde0050b3425_1764x2311.png 1272w, https://substackcdn.com/image/fetch/$s_!xNEB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F309c52ca-ad27-4336-a463-cde0050b3425_1764x2311.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 14: MiniMax-M3 applies RoPE to queries and keys inner attention.</figcaption></figure></div><p>As shown in the fig above, the LLM architecture has no definitive stance vector added alongside the first token embedding. Its definitive stance indication is injected inner notice instead.</p><p>Side note: MiniMax-M3 rotates lone part of all query and key head, a assortment known as &ldquo;partial RoPE&rdquo; that we volition shield afterward in Section 6.1. But to comprehend the essential RoPE procedure first, let&rsquo;s stroll through the method stage by stage using a small example.</p><p>Attention normally divides its query, key, and value features into multiple heads, which are processed in parallel. Hence, the complete scheme is normally called Multi-head Attention (note that Grouped Query Attention additionally uses multiple heads). Here, all notice caput computes its own notice scores and valued sums. RoPE afterward acts on pairs of numbers inside all query and key head.</p><p><span>To create this concrete, Figure 12 below follows a token depiction through the learned query and key projections, which are computed via the importance matrices </span><em><span>W</span><sub>Q</sub></em><span> and </span><em><span>W</span><sub>K</sub></em><span>. In this toy example, all projection produces 16 features, which are reshaped into two heads alongside eight features each. In the fig below, all small division contains one scalar feature, definition one numerical admission in a vector (the numbers are not shown because of area reasons).</span></p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!0J_2!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a99d70f-f237-4af5-8f0d-9f672e8193a7_2755x2720.png 424w, https://substackcdn.com/image/fetch/$s_!0J_2!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a99d70f-f237-4af5-8f0d-9f672e8193a7_2755x2720.png 848w, https://substackcdn.com/image/fetch/$s_!0J_2!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a99d70f-f237-4af5-8f0d-9f672e8193a7_2755x2720.png 1272w, https://substackcdn.com/image/fetch/$s_!0J_2!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a99d70f-f237-4af5-8f0d-9f672e8193a7_2755x2720.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a99d70f-f237-4af5-8f0d-9f672e8193a7_2755x2720.png" width="1456" height="1438" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/3a99d70f-f237-4af5-8f0d-9f672e8193a7_2755x2720.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1438,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 15 — inputs and pairs&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 15 — inputs and pairs" title="Figure 15 — inputs and pairs" srcset="https://substackcdn.com/image/fetch/$s_!0J_2!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a99d70f-f237-4af5-8f0d-9f672e8193a7_2755x2720.png 424w, https://substackcdn.com/image/fetch/$s_!0J_2!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a99d70f-f237-4af5-8f0d-9f672e8193a7_2755x2720.png 848w, https://substackcdn.com/image/fetch/$s_!0J_2!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a99d70f-f237-4af5-8f0d-9f672e8193a7_2755x2720.png 1272w, https://substackcdn.com/image/fetch/$s_!0J_2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a99d70f-f237-4af5-8f0d-9f672e8193a7_2755x2720.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 15: Projecting a hidden state, reshaping into heads, and pairing features for RoPE.</figcaption></figure></div><p><span>For the archetypal eight-feature head, </span>[1, 2, 3, 4, 5, 6, 7, 8]<span>, we brace all characteristic in the archetypal fractional alongside the corresponding characteristic in the second half. This gives </span>(1, 5)<span>, </span>(2, 6)<span>, </span>(3, 7)<span>, and </span>(4, 8)<span>. This </span><em>split-half</em><span> gathering is used in LLM implementations, including my </span><a href="https://en.mycoding.id/category/technology" target="_blank">Qwen3 implementation</a><span>. (We volition difference it alongside the original, substitute adjacent-feature pairing method afterward following implementing the essential operation.)</span></p><p><span>Each brace gets a rotation viewpoint resolute by the token stance </span><em>m</em><span> and that pair&rsquo;s frequence </span><em><span>&theta;</span><sub>i</sub></em><span>. The corresponding query and key pairs use the identical frequency. Figure 16 below shows how we rotate the pairs, put the updated features rear in their first locations, and compute the query-key dot product.</span></p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!LQ-n!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010e216-c965-4313-ace6-6682016a3177_2671x1543.png 424w, https://substackcdn.com/image/fetch/$s_!LQ-n!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010e216-c965-4313-ace6-6682016a3177_2671x1543.png 848w, https://substackcdn.com/image/fetch/$s_!LQ-n!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010e216-c965-4313-ace6-6682016a3177_2671x1543.png 1272w, https://substackcdn.com/image/fetch/$s_!LQ-n!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010e216-c965-4313-ace6-6682016a3177_2671x1543.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010e216-c965-4313-ace6-6682016a3177_2671x1543.png" width="1456" height="841" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/e010e216-c965-4313-ace6-6682016a3177_2671x1543.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:841,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 16 — stance rotations&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 16 — stance rotations" title="Figure 16 — stance rotations" srcset="https://substackcdn.com/image/fetch/$s_!LQ-n!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010e216-c965-4313-ace6-6682016a3177_2671x1543.png 424w, https://substackcdn.com/image/fetch/$s_!LQ-n!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010e216-c965-4313-ace6-6682016a3177_2671x1543.png 848w, https://substackcdn.com/image/fetch/$s_!LQ-n!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010e216-c965-4313-ace6-6682016a3177_2671x1543.png 1272w, https://substackcdn.com/image/fetch/$s_!LQ-n!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010e216-c965-4313-ace6-6682016a3177_2671x1543.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 16: Pairwise rotations create the rotated query and key vectors used to compute notice scores.</figcaption></figure></div><p>The next division volition stroll through the figure(s) a bit additional slowly.</p><p><span>For the following example, we&rsquo;ll use the identical two positions as before. The query comes from &ldquo;the&rdquo; at stance </span><em>m</em><span> = 5, and the key comes from the archetypal &ldquo;fox&rdquo; at stance </span><em>n</em><span> = 2. Since, in this example, the key comes before the query, the causal disguise allows the query to act to it.</span></p><p>To exemplify the consequence that the stance has, and keeping everything fixed, let&rsquo;s assume we commencement the two pairs from the identical two values, (1, 5).</p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!LgZZ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6690385d-65a0-41e7-b2f8-7d0ce501c264_2650x461.png 424w, https://substackcdn.com/image/fetch/$s_!LgZZ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6690385d-65a0-41e7-b2f8-7d0ce501c264_2650x461.png 848w, https://substackcdn.com/image/fetch/$s_!LgZZ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6690385d-65a0-41e7-b2f8-7d0ce501c264_2650x461.png 1272w, https://substackcdn.com/image/fetch/$s_!LgZZ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6690385d-65a0-41e7-b2f8-7d0ce501c264_2650x461.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6690385d-65a0-41e7-b2f8-7d0ce501c264_2650x461.png" width="632" height="109.81868131868131" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/6690385d-65a0-41e7-b2f8-7d0ce501c264_2650x461.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:253,&quot;width&quot;:1456,&quot;resizeWidth&quot;:632,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 17 — query brace closeup&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 17 — query brace closeup" title="Figure 17 — query brace closeup" srcset="https://substackcdn.com/image/fetch/$s_!LgZZ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6690385d-65a0-41e7-b2f8-7d0ce501c264_2650x461.png 424w, https://substackcdn.com/image/fetch/$s_!LgZZ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6690385d-65a0-41e7-b2f8-7d0ce501c264_2650x461.png 848w, https://substackcdn.com/image/fetch/$s_!LgZZ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6690385d-65a0-41e7-b2f8-7d0ce501c264_2650x461.png 1272w, https://substackcdn.com/image/fetch/$s_!LgZZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6690385d-65a0-41e7-b2f8-7d0ce501c264_2650x461.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div><figcaption>Figure 17: The archetypal characteristic pair, (1, 5), takes one admission from all fractional of the query head.</figcaption></figure></div><p><span>Furthermore, for this example, let&rsquo;s use </span><em>&theta;</em><span> = 20&deg; per token stance (</span><em>&pi;</em><span>/9 radians per position). At stance 2, we rotate the key&rsquo;s arrow counterclockwise by 2 &times; 20&deg; = 40&deg;. At stance 5, we rotate the query&rsquo;s arrow by 5 &times; 20&deg; = 100&deg;.</span></p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!RhWQ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03367132-6eb1-44cd-85fd-afb47314c6a9_2672x1716.png 424w, https://substackcdn.com/image/fetch/$s_!RhWQ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03367132-6eb1-44cd-85fd-afb47314c6a9_2672x1716.png 848w, https://substackcdn.com/image/fetch/$s_!RhWQ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03367132-6eb1-44cd-85fd-afb47314c6a9_2672x1716.png 1272w, https://substackcdn.com/image/fetch/$s_!RhWQ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03367132-6eb1-44cd-85fd-afb47314c6a9_2672x1716.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03367132-6eb1-44cd-85fd-afb47314c6a9_2672x1716.png" width="1456" height="935" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/03367132-6eb1-44cd-85fd-afb47314c6a9_2672x1716.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:935,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 18 — rotations at positions 2 and 5&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 18 — rotations at positions 2 and 5" title="Figure 18 — rotations at positions 2 and 5" srcset="https://substackcdn.com/image/fetch/$s_!RhWQ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03367132-6eb1-44cd-85fd-afb47314c6a9_2672x1716.png 424w, https://substackcdn.com/image/fetch/$s_!RhWQ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03367132-6eb1-44cd-85fd-afb47314c6a9_2672x1716.png 848w, https://substackcdn.com/image/fetch/$s_!RhWQ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03367132-6eb1-44cd-85fd-afb47314c6a9_2672x1716.png 1272w, https://substackcdn.com/image/fetch/$s_!RhWQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03367132-6eb1-44cd-85fd-afb47314c6a9_2672x1716.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 18: Starting from (1, 5), RoPE rotates the key brace at stance 2 by 40&deg; and the query brace at stance 5 by 100&deg;.</figcaption></figure></div><p>To exemplify what&rsquo;s happening in the fig above, for example, plugging cos 40&deg; &asymp; 0.766 and sin 40&deg; &asymp; 0.643 into the rotation from Figure 16 gives the new key coordinates:</p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DBMJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcfe39910-8f5a-4edf-88e2-018f960a0435_2240x82.png 424w, https://substackcdn.com/image/fetch/$s_!DBMJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcfe39910-8f5a-4edf-88e2-018f960a0435_2240x82.png 848w, https://substackcdn.com/image/fetch/$s_!DBMJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcfe39910-8f5a-4edf-88e2-018f960a0435_2240x82.png 1272w, https://substackcdn.com/image/fetch/$s_!DBMJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcfe39910-8f5a-4edf-88e2-018f960a0435_2240x82.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcfe39910-8f5a-4edf-88e2-018f960a0435_2240x82.png" width="1456" height="53" data-attrs="//substack-post-media.s3.amazonaws.com/public/images/cfe39910-8f5a-4edf-88e2-018f960a0435_2240x82.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:53,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 5: k&#039; = (1\\cos40^\\circ - 5\\sin40^\\circ,\\;1\\sin40^\\circ+5\\cos40^\\circ) \\approx (-2.448,\\;4.473).&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" alt=" k&#039; = (1\cos40^\circ - 5\sin40^\circ,\;1\sin40^\circ+5\cos40^\circ) \approx (-2.448,\;4.473)." title=" k&#039; = (1\cos40^\circ - 5\sin40^\circ,\;1\sin40^\circ+5\cos40^\circ) \approx (-2.448,\;4.473)." srcset="https://substackcdn.com/image/fetch/$s_!DBMJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcfe39910-8f5a-4edf-88e2-018f960a0435_2240x82.png 424w, https://substackcdn.com/image/fetch/$s_!DBMJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcfe39910-8f5a-4edf-88e2-018f960a0435_2240x82.png 848w, https://substackcdn.com/image/fetch/$s_!DBMJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcfe39910-8f5a-4edf-88e2-018f960a0435_2240x82.png 1272w, https://substackcdn.com/image/fetch/$s_!DBMJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcfe39910-8f5a-4edf-88e2-018f960a0435_2240x82.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p><span>Then, changing 40&deg; to 100&deg;, using the identical equation as above, the rotated query brace becomes </span><em>q&prime;</em><span> &asymp; (&minus;5.098, 0.117). But the crucial item is that the two arrows motionless have dimension &radic;26. Only the rotation changes but the vectors keep the identical lengths following the transformation.</span></p><p><span>So, for a query brace (</span><em>a</em><span>, </span><em>b</em><span>) at stance </span><em>m</em><span>, the two operations current are.</span></p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YiBj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6de4a384-3c18-4bbc-9346-08ae327ee6f2_2240x154.png 424w, https://substackcdn.com/image/fetch/$s_!YiBj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6de4a384-3c18-4bbc-9346-08ae327ee6f2_2240x154.png 848w, https://substackcdn.com/image/fetch/$s_!YiBj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6de4a384-3c18-4bbc-9346-08ae327ee6f2_2240x154.png 1272w, https://substackcdn.com/image/fetch/$s_!YiBj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6de4a384-3c18-4bbc-9346-08ae327ee6f2_2240x154.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6de4a384-3c18-4bbc-9346-08ae327ee6f2_2240x154.png" width="1456" height="100" data-attrs="//substack-post-media.s3.amazonaws.com/public/images/6de4a384-3c18-4bbc-9346-08ae327ee6f2_2240x154.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:100,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 6: \\begin{aligned} a&#039; &amp;= a\\cos(m\\theta)-b\\sin(m\\theta), \\\\ b&#039; &amp;= a\\sin(m\\theta)+b\\cos(m\\theta). \\end{aligned}&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" alt=" \begin{aligned} a&#039; &amp;= a\cos(m\theta)-b\sin(m\theta), \\ b&#039; &amp;= a\sin(m\theta)+b\cos(m\theta). \end{aligned}" title=" \begin{aligned} a&#039; &amp;= a\cos(m\theta)-b\sin(m\theta), \\ b&#039; &amp;= a\sin(m\theta)+b\cos(m\theta). \end{aligned}" srcset="https://substackcdn.com/image/fetch/$s_!YiBj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6de4a384-3c18-4bbc-9346-08ae327ee6f2_2240x154.png 424w, https://substackcdn.com/image/fetch/$s_!YiBj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6de4a384-3c18-4bbc-9346-08ae327ee6f2_2240x154.png 848w, https://substackcdn.com/image/fetch/$s_!YiBj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6de4a384-3c18-4bbc-9346-08ae327ee6f2_2240x154.png 1272w, https://substackcdn.com/image/fetch/$s_!YiBj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6de4a384-3c18-4bbc-9346-08ae327ee6f2_2240x154.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p><span>For the key brace at stance </span><em>n</em><span>, we use the identical equations alongside </span><em>n&theta;</em><span>.</span></p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HQdc!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0eccfd5c-a1f5-4f7f-b40a-4ec197d7a218_2240x154.png 424w, https://substackcdn.com/image/fetch/$s_!HQdc!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0eccfd5c-a1f5-4f7f-b40a-4ec197d7a218_2240x154.png 848w, https://substackcdn.com/image/fetch/$s_!HQdc!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0eccfd5c-a1f5-4f7f-b40a-4ec197d7a218_2240x154.png 1272w, https://substackcdn.com/image/fetch/$s_!HQdc!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0eccfd5c-a1f5-4f7f-b40a-4ec197d7a218_2240x154.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0eccfd5c-a1f5-4f7f-b40a-4ec197d7a218_2240x154.png" width="1456" height="100" data-attrs="//substack-post-media.s3.amazonaws.com/public/images/0eccfd5c-a1f5-4f7f-b40a-4ec197d7a218_2240x154.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:100,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 7: \\begin{aligned}a&#039; &amp;= a\\cos(n\\theta)-b\\sin(n\\theta), \\\\b&#039; &amp;= a\\sin(n\\theta)+b\\cos(n\\theta).\\end{aligned}&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" alt=" \begin{aligned}a&#039; &amp;= a\cos(n\theta)-b\sin(n\theta), \\b&#039; &amp;= a\sin(n\theta)+b\cos(n\theta).\end{aligned}" title=" \begin{aligned}a&#039; &amp;= a\cos(n\theta)-b\sin(n\theta), \\b&#039; &amp;= a\sin(n\theta)+b\cos(n\theta).\end{aligned}" srcset="https://substackcdn.com/image/fetch/$s_!HQdc!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0eccfd5c-a1f5-4f7f-b40a-4ec197d7a218_2240x154.png 424w, https://substackcdn.com/image/fetch/$s_!HQdc!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0eccfd5c-a1f5-4f7f-b40a-4ec197d7a218_2240x154.png 848w, https://substackcdn.com/image/fetch/$s_!HQdc!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0eccfd5c-a1f5-4f7f-b40a-4ec197d7a218_2240x154.png 1272w, https://substackcdn.com/image/fetch/$s_!HQdc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0eccfd5c-a1f5-4f7f-b40a-4ec197d7a218_2240x154.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p>(Section 4 afterward shows how RoPE calculates the angular frequence for all characteristic pair.)</p><p>To near this section, for those additional comfortable study code, everything concerning RoPE illustrated so far can be written in code, as follows:</p><p>The resulting values are:</p><pre>Original key: (1, 5) Original query: (1, 5) Rotated key: (-2.448, 4.473) Rotated query: (-5.098, 0.117)</pre><p>If we rotate all 4 pairs in the 8-feature query (and key) vector(s), we get the vectors below. (We&rsquo;ll arrive rear to how we go from rotating one brace to rotating all of them later.)</p><pre>Original key: (1, 2, 3, 4, 5, 6, 7, 8) Original query: (1, 2, 3, 4, 5, 6, 7, 8) Rotated key: (-2.448, 1.577, 2.951, 3.994, 4.473, 6.125, 7.021, 8.003) Rotated query: (-5.098, 0.928, 2.877, 3.986, 0.117, 6.256, 7.051, 8.007)</pre><p>Note that, as shown above, the query and key heads have eight features each, which we earlier grouped into 4 pairs alongside two elements each, akin (1, 5), and (2, 6), and so on. For simplicity current we&rsquo;ll prosecute fair the archetypal brace in all head, (1, 5), cast by taking the entries at indices 0 and 4:</p><pre>Original key: (1, 5) Original query: (1, 5) Rotated key: (-2.448, 4.473) Rotated query: (-5.098, 0.117)</pre><p>As shown above, in our example, those entries have values (1, 5) in the two heads before the rotation.</p><p><span>Also, as in the code above, let&rsquo;s use the notation </span><em>q</em><span> =(</span><em><span>q</span><sub>1</sub></em><span>, </span><em><span>q</span><sub>2</sub></em><span>) and </span><em>k</em><span> =(</span><em><span>k</span><sub>1</sub></em><span>, </span><em><span>k</span><sub>2</sub></em><span>) for these two pairs. To compute their dot product, using the norm dot merchandise formula, we multiply corresponding coordinates and add the products:</span></p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!NqAC!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff26d2b45-d5fd-49cb-ae1c-baae68e98d77_2240x73.png 424w, https://substackcdn.com/image/fetch/$s_!NqAC!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff26d2b45-d5fd-49cb-ae1c-baae68e98d77_2240x73.png 848w, https://substackcdn.com/image/fetch/$s_!NqAC!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff26d2b45-d5fd-49cb-ae1c-baae68e98d77_2240x73.png 1272w, https://substackcdn.com/image/fetch/$s_!NqAC!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff26d2b45-d5fd-49cb-ae1c-baae68e98d77_2240x73.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff26d2b45-d5fd-49cb-ae1c-baae68e98d77_2240x73.png" width="1456" height="47" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/f26d2b45-d5fd-49cb-ae1c-baae68e98d77_2240x73.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:47,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 8: q\\cdot k=q_1k_1+q_2k_2.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt=" q\cdot k=q_1k_1+q_2k_2." title=" q\cdot k=q_1k_1+q_2k_2." srcset="https://substackcdn.com/image/fetch/$s_!NqAC!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff26d2b45-d5fd-49cb-ae1c-baae68e98d77_2240x73.png 424w, https://substackcdn.com/image/fetch/$s_!NqAC!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff26d2b45-d5fd-49cb-ae1c-baae68e98d77_2240x73.png 848w, https://substackcdn.com/image/fetch/$s_!NqAC!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff26d2b45-d5fd-49cb-ae1c-baae68e98d77_2240x73.png 1272w, https://substackcdn.com/image/fetch/$s_!NqAC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff26d2b45-d5fd-49cb-ae1c-baae68e98d77_2240x73.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p>Before rotation, the pairs have a dot merchandise of 1 &times; 1 + 5 &times; 5 = 26. Using the rounded coordinates from the former Figure 18 gives the contribution from the archetypal brace following rotation:</p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!scnp!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd175c135-b520-49e6-8d10-f18888e3743c_2240x82.png 424w, https://substackcdn.com/image/fetch/$s_!scnp!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd175c135-b520-49e6-8d10-f18888e3743c_2240x82.png 848w, https://substackcdn.com/image/fetch/$s_!scnp!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd175c135-b520-49e6-8d10-f18888e3743c_2240x82.png 1272w, https://substackcdn.com/image/fetch/$s_!scnp!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd175c135-b520-49e6-8d10-f18888e3743c_2240x82.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd175c135-b520-49e6-8d10-f18888e3743c_2240x82.png" width="1456" height="53" data-attrs="//substack-post-media.s3.amazonaws.com/public/images/d175c135-b520-49e6-8d10-f18888e3743c_2240x82.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:53,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 9: q&#039;\\cdot k&#039;\\approx(-5.098)(-2.448)+(0.117)(4.473)\\approx13.00.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" alt=" q&#039;\cdot k&#039;\approx(-5.098)(-2.448)+(0.117)(4.473)\approx13.00." title=" q&#039;\cdot k&#039;\approx(-5.098)(-2.448)+(0.117)(4.473)\approx13.00." srcset="https://substackcdn.com/image/fetch/$s_!scnp!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd175c135-b520-49e6-8d10-f18888e3743c_2240x82.png 424w, https://substackcdn.com/image/fetch/$s_!scnp!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd175c135-b520-49e6-8d10-f18888e3743c_2240x82.png 848w, https://substackcdn.com/image/fetch/$s_!scnp!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd175c135-b520-49e6-8d10-f18888e3743c_2240x82.png 1272w, https://substackcdn.com/image/fetch/$s_!scnp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd175c135-b520-49e6-8d10-f18888e3743c_2240x82.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p><span>We can see why by penning the dot merchandise in conditions of the lengths of the two vectors (arrows in Figure 18) and the viewpoint </span><em>&#981;</em><span> between them.</span></p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!GpbH!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F176ef9de-46fa-4bf4-8a1c-a90d14c13874_2240x82.png 424w, https://substackcdn.com/image/fetch/$s_!GpbH!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F176ef9de-46fa-4bf4-8a1c-a90d14c13874_2240x82.png 848w, https://substackcdn.com/image/fetch/$s_!GpbH!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F176ef9de-46fa-4bf4-8a1c-a90d14c13874_2240x82.png 1272w, https://substackcdn.com/image/fetch/$s_!GpbH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F176ef9de-46fa-4bf4-8a1c-a90d14c13874_2240x82.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F176ef9de-46fa-4bf4-8a1c-a90d14c13874_2240x82.png" width="1456" height="53" data-attrs="//substack-post-media.s3.amazonaws.com/public/images/176ef9de-46fa-4bf4-8a1c-a90d14c13874_2240x82.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:53,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 10: q&#039;\\cdot k&#039;=\\lVert q&#039;\\rVert\\,\\lVert k&#039;\\rVert\\cos\\phi.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" alt=" q&#039;\cdot k&#039;=\lVert q&#039;\rVert\,\lVert k&#039;\rVert\cos\phi." title=" q&#039;\cdot k&#039;=\lVert q&#039;\rVert\,\lVert k&#039;\rVert\cos\phi." srcset="https://substackcdn.com/image/fetch/$s_!GpbH!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F176ef9de-46fa-4bf4-8a1c-a90d14c13874_2240x82.png 424w, https://substackcdn.com/image/fetch/$s_!GpbH!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F176ef9de-46fa-4bf4-8a1c-a90d14c13874_2240x82.png 848w, https://substackcdn.com/image/fetch/$s_!GpbH!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F176ef9de-46fa-4bf4-8a1c-a90d14c13874_2240x82.png 1272w, https://substackcdn.com/image/fetch/$s_!GpbH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F176ef9de-46fa-4bf4-8a1c-a90d14c13874_2240x82.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p>In the two cases, the vector dimension is &radic;26. The query (position 5) is rotated by 100&deg; and the key (position 2) is rotated by 40&deg;. The vectors (arrows) commencement from the identical direction, so they end up being 60&deg; distinct following rotation. Their exact dot merchandise is 26 &times; cos(60&deg;) = 13.</p><p>Now let&rsquo;s see what happens if we move fair the key from stance 2 to stance 3, but we keep its starting coordinates (1, 5) fixed. The key&rsquo;s rotation changes now from 2 &times; 20&deg; = 40&deg; to 3 &times; 20&deg; = 60&deg;. The query motionless rotates by 100&deg;, so the viewpoint between the arrows shrinks to 40&deg; and the dot merchandise changes from 13 to 26 &times; cos(40&deg;) &asymp; 19.92. So, moving the key one stance nearer changed the dot merchandise equal although we started alongside the identical characteristic values.</p><p>Why do these dot products matter? In attention, we sum the contributions from all characteristic pairs to get the query-key dot products. RoPE can alter those dot products and thus the weights that softmax assigns to the value vectors.</p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!haoa!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2351921-4dc0-4904-9d35-941e3de77a9f_2240x162.png 424w, https://substackcdn.com/image/fetch/$s_!haoa!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2351921-4dc0-4904-9d35-941e3de77a9f_2240x162.png 848w, https://substackcdn.com/image/fetch/$s_!haoa!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2351921-4dc0-4904-9d35-941e3de77a9f_2240x162.png 1272w, https://substackcdn.com/image/fetch/$s_!haoa!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2351921-4dc0-4904-9d35-941e3de77a9f_2240x162.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2351921-4dc0-4904-9d35-941e3de77a9f_2240x162.png" width="1456" height="105" data-attrs="//substack-post-media.s3.amazonaws.com/public/images/e2351921-4dc0-4904-9d35-941e3de77a9f_2240x162.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:105,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 11: \\operatorname{Attention}(Q&#039;,K&#039;,V) =\\operatorname{softmax}\\!\\left(\\frac{Q&#039;(K&#039;)^\\top}{\\sqrt{d}}\\right)V.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" alt=" \operatorname{Attention}(Q&#039;,K&#039;,V) =\operatorname{softmax}\!\left(\frac{Q&#039;(K&#039;)^\top}{\sqrt{d}}\right)V." title=" \operatorname{Attention}(Q&#039;,K&#039;,V) =\operatorname{softmax}\!\left(\frac{Q&#039;(K&#039;)^\top}{\sqrt{d}}\right)V." srcset="https://substackcdn.com/image/fetch/$s_!haoa!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2351921-4dc0-4904-9d35-941e3de77a9f_2240x162.png 424w, https://substackcdn.com/image/fetch/$s_!haoa!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2351921-4dc0-4904-9d35-941e3de77a9f_2240x162.png 848w, https://substackcdn.com/image/fetch/$s_!haoa!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2351921-4dc0-4904-9d35-941e3de77a9f_2240x162.png 1272w, https://substackcdn.com/image/fetch/$s_!haoa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2351921-4dc0-4904-9d35-941e3de77a9f_2240x162.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p><span>In the notice equation above, </span><em>Q&prime;</em><span> and </span><em>K&prime;</em><span> are matrices that merge the rotated queries </span><em>q&prime;</em><span> and keys </span><em>k&prime;</em><span> for one head, </span><em>V</em><span> contains the values, and </span><em>d</em><span> is the caput dimension.</span></p><p><span>Now that we illustrated the computation of the dot products, which is at the center of the notice mechanism, let&rsquo;s come back to positions </span><em>m</em><span> = 5 and </span><em>n</em><span> = 2. If we change the two positions to the correct by three positions, we get </span><em>m</em><span> = 8 and </span><em>n</em><span> = 5. Each rotation gains 60&deg; (the query goes from 100&deg; to 160&deg;, and the key goes from 40&deg; to 100&deg;), however, the viewpoint between the vectors stays the same. The dot merchandise for this characteristic brace is motionless 13.</span></p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2bNG!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c627967-3788-4e17-85ef-8d6e716d2284_2525x785.png 424w, https://substackcdn.com/image/fetch/$s_!2bNG!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c627967-3788-4e17-85ef-8d6e716d2284_2525x785.png 848w, https://substackcdn.com/image/fetch/$s_!2bNG!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c627967-3788-4e17-85ef-8d6e716d2284_2525x785.png 1272w, https://substackcdn.com/image/fetch/$s_!2bNG!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c627967-3788-4e17-85ef-8d6e716d2284_2525x785.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c627967-3788-4e17-85ef-8d6e716d2284_2525x785.png" width="1456" height="453" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/5c627967-3788-4e17-85ef-8d6e716d2284_2525x785.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:453,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 19 — stance change dot products&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 19 — stance change dot products" title="Figure 19 — stance change dot products" srcset="https://substackcdn.com/image/fetch/$s_!2bNG!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c627967-3788-4e17-85ef-8d6e716d2284_2525x785.png 424w, https://substackcdn.com/image/fetch/$s_!2bNG!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c627967-3788-4e17-85ef-8d6e716d2284_2525x785.png 848w, https://substackcdn.com/image/fetch/$s_!2bNG!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c627967-3788-4e17-85ef-8d6e716d2284_2525x785.png 1272w, https://substackcdn.com/image/fetch/$s_!2bNG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c627967-3788-4e17-85ef-8d6e716d2284_2525x785.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 19: The archetypal pair&rsquo;s dot merchandise at three distinct stance settings.</figcaption></figure></div><p>The key is that the difference between the rotation angles is</p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Q7jd!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a220d40-c355-48e0-9bfc-6318fe46e2ef_2240x79.png 424w, https://substackcdn.com/image/fetch/$s_!Q7jd!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a220d40-c355-48e0-9bfc-6318fe46e2ef_2240x79.png 848w, https://substackcdn.com/image/fetch/$s_!Q7jd!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a220d40-c355-48e0-9bfc-6318fe46e2ef_2240x79.png 1272w, https://substackcdn.com/image/fetch/$s_!Q7jd!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a220d40-c355-48e0-9bfc-6318fe46e2ef_2240x79.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a220d40-c355-48e0-9bfc-6318fe46e2ef_2240x79.png" width="1456" height="51" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/0a220d40-c355-48e0-9bfc-6318fe46e2ef_2240x79.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:51,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 12: n\\theta-m\\theta=(n-m)\\theta.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt=" n\theta-m\theta=(n-m)\theta." title=" n\theta-m\theta=(n-m)\theta." srcset="https://substackcdn.com/image/fetch/$s_!Q7jd!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a220d40-c355-48e0-9bfc-6318fe46e2ef_2240x79.png 424w, https://substackcdn.com/image/fetch/$s_!Q7jd!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a220d40-c355-48e0-9bfc-6318fe46e2ef_2240x79.png 848w, https://substackcdn.com/image/fetch/$s_!Q7jd!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a220d40-c355-48e0-9bfc-6318fe46e2ef_2240x79.png 1272w, https://substackcdn.com/image/fetch/$s_!Q7jd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a220d40-c355-48e0-9bfc-6318fe46e2ef_2240x79.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p>If we keep the first query and key vectors fixed and change the two token positions by the identical amount, RoPE adds the identical rotation to the two vectors. The viewpoint between them stays the same, so their dot merchandise stays the same. Note that Figure 20 below uses smaller angles than the genuine example (20&deg; per position) to keep the image readable.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!jxZL!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88495cb6-f380-43db-aeab-0f50afbd41b0_2610x888.png 424w, https://substackcdn.com/image/fetch/$s_!jxZL!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88495cb6-f380-43db-aeab-0f50afbd41b0_2610x888.png 848w, https://substackcdn.com/image/fetch/$s_!jxZL!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88495cb6-f380-43db-aeab-0f50afbd41b0_2610x888.png 1272w, https://substackcdn.com/image/fetch/$s_!jxZL!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88495cb6-f380-43db-aeab-0f50afbd41b0_2610x888.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88495cb6-f380-43db-aeab-0f50afbd41b0_2610x888.png" width="1456" height="495" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/88495cb6-f380-43db-aeab-0f50afbd41b0_2610x888.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:495,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 20 — related position&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 20 — related position" title="Figure 20 — related position" srcset="https://substackcdn.com/image/fetch/$s_!jxZL!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88495cb6-f380-43db-aeab-0f50afbd41b0_2610x888.png 424w, https://substackcdn.com/image/fetch/$s_!jxZL!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88495cb6-f380-43db-aeab-0f50afbd41b0_2610x888.png 848w, https://substackcdn.com/image/fetch/$s_!jxZL!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88495cb6-f380-43db-aeab-0f50afbd41b0_2610x888.png 1272w, https://substackcdn.com/image/fetch/$s_!jxZL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88495cb6-f380-43db-aeab-0f50afbd41b0_2610x888.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 20: A shared stance change preserves the related viewpoint and dot merchandise for fixed query and key features.</figcaption></figure></div><p>The examples complete and Figure 20 exemplify the chief idea rearward RoPE. We use all token&rsquo;s stance to rotate the corresponding query and key in the notice module. In the query-key dot product, what matters is how much one vector rotates related to the other. For all characteristic pair, this difference in rotation depends lone on the offset between the two tokens.</p><p>To exemplify this in the big image context, in Figure 21, all row collects the scores for one query. If we regard row 5, the query for &ldquo;the&rdquo;, and pillar 2, the key for &ldquo;fox&rdquo;, the notice mark between those two is computed as the dot merchandise between the two rotated versions of the query and key vector.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!0dw5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a413998-3f9a-477b-9fb6-d475bdedf847_2791x1962.png 424w, https://substackcdn.com/image/fetch/$s_!0dw5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a413998-3f9a-477b-9fb6-d475bdedf847_2791x1962.png 848w, https://substackcdn.com/image/fetch/$s_!0dw5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a413998-3f9a-477b-9fb6-d475bdedf847_2791x1962.png 1272w, https://substackcdn.com/image/fetch/$s_!0dw5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a413998-3f9a-477b-9fb6-d475bdedf847_2791x1962.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a413998-3f9a-477b-9fb6-d475bdedf847_2791x1962.png" width="1456" height="1024" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/2a413998-3f9a-477b-9fb6-d475bdedf847_2791x1962.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1024,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 21 — query key mark map&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 21 — query key mark map" title="Figure 21 — query key mark map" srcset="https://substackcdn.com/image/fetch/$s_!0dw5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a413998-3f9a-477b-9fb6-d475bdedf847_2791x1962.png 424w, https://substackcdn.com/image/fetch/$s_!0dw5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a413998-3f9a-477b-9fb6-d475bdedf847_2791x1962.png 848w, https://substackcdn.com/image/fetch/$s_!0dw5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a413998-3f9a-477b-9fb6-d475bdedf847_2791x1962.png 1272w, https://substackcdn.com/image/fetch/$s_!0dw5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a413998-3f9a-477b-9fb6-d475bdedf847_2791x1962.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 21: The notice scores for one caput before softmax. Gray cells grade forthcoming keys excluded by the causal mask.</figcaption></figure></div><p>What&rsquo;s the point, again? Let&rsquo;s formerly additional regard &ldquo;The red fox jumps complete the gray fox.&rdquo; If we add 100 tokens (e.g., think scheme prompt) before the sentence, the archetypal &ldquo;fox&rdquo; and the second &ldquo;the&rdquo; move from positions 2 and 5 to 102 and 105. However, a prefix of any dimension shifts the two positions equally, so they motionless remain exactly three positions apart. This means, all vector (arrow) gains the exact identical additional rotation, which preserves the viewpoint between them and their dot merchandise (assuming we clasp the unrotated query and key vectors fixed; in a genuine model, a new prefix of 100 tokens can alter those vectors by changing what the tokens act to).</p><p>Figure 22 tracks this difference in the notice mark matrix. The correct panel shows the rows and columns for the declaration following we add the prefix. In our toy example, the archetypal brace motionless contributes 13 to the dot product.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!yQkh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19ac0a71-c4d4-4cbe-b1ec-04371b5b3a58_2691x1745.png 424w, https://substackcdn.com/image/fetch/$s_!yQkh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19ac0a71-c4d4-4cbe-b1ec-04371b5b3a58_2691x1745.png 848w, https://substackcdn.com/image/fetch/$s_!yQkh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19ac0a71-c4d4-4cbe-b1ec-04371b5b3a58_2691x1745.png 1272w, https://substackcdn.com/image/fetch/$s_!yQkh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19ac0a71-c4d4-4cbe-b1ec-04371b5b3a58_2691x1745.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19ac0a71-c4d4-4cbe-b1ec-04371b5b3a58_2691x1745.png" width="1456" height="944" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/19ac0a71-c4d4-4cbe-b1ec-04371b5b3a58_2691x1745.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:944,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 22 — shifted query key mark map&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 22 — shifted query key mark map" title="Figure 22 — shifted query key mark map" srcset="https://substackcdn.com/image/fetch/$s_!yQkh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19ac0a71-c4d4-4cbe-b1ec-04371b5b3a58_2691x1745.png 424w, https://substackcdn.com/image/fetch/$s_!yQkh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19ac0a71-c4d4-4cbe-b1ec-04371b5b3a58_2691x1745.png 848w, https://substackcdn.com/image/fetch/$s_!yQkh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19ac0a71-c4d4-4cbe-b1ec-04371b5b3a58_2691x1745.png 1272w, https://substackcdn.com/image/fetch/$s_!yQkh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19ac0a71-c4d4-4cbe-b1ec-04371b5b3a58_2691x1745.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 22: The highlighted mark moves from row 5, pillar 2 to row 105, pillar 102. It keeps the identical value if we clasp the query and key vectors fixed before rotation.</figcaption></figure></div><p>As you may recall, in the first transformer&rsquo;s sinusoidal encoding, a fixed offset additionally corresponds to a fixed rotation inside all sine/cosine pair. But that shared rotation applies lone to the stance vectors (added to the input vector) alternatively of the key and query vectors inner the notice mechanism.</p><p>Figure 23 compares what this change changes in the three methods we covered so far.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!qAmy!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf8cae0f-aa03-454c-b94e-dde8dd5ae680_2640x2488.png 424w, https://substackcdn.com/image/fetch/$s_!qAmy!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf8cae0f-aa03-454c-b94e-dde8dd5ae680_2640x2488.png 848w, https://substackcdn.com/image/fetch/$s_!qAmy!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf8cae0f-aa03-454c-b94e-dde8dd5ae680_2640x2488.png 1272w, https://substackcdn.com/image/fetch/$s_!qAmy!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf8cae0f-aa03-454c-b94e-dde8dd5ae680_2640x2488.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf8cae0f-aa03-454c-b94e-dde8dd5ae680_2640x2488.png" width="1456" height="1372" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/bf8cae0f-aa03-454c-b94e-dde8dd5ae680_2640x2488.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1372,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 23 — stance change comparison&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 23 — stance change comparison" title="Figure 23 — stance change comparison" srcset="https://substackcdn.com/image/fetch/$s_!qAmy!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf8cae0f-aa03-454c-b94e-dde8dd5ae680_2640x2488.png 424w, https://substackcdn.com/image/fetch/$s_!qAmy!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf8cae0f-aa03-454c-b94e-dde8dd5ae680_2640x2488.png 848w, https://substackcdn.com/image/fetch/$s_!qAmy!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf8cae0f-aa03-454c-b94e-dde8dd5ae680_2640x2488.png 1272w, https://substackcdn.com/image/fetch/$s_!qAmy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf8cae0f-aa03-454c-b94e-dde8dd5ae680_2640x2488.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 23: The identical stance change under sinusoidal encodings, GPT-2&rsquo;s learned stance embeddings, and RoPE.</figcaption></figure></div><p>To see how the related offset affects the dot product, we focused on one characteristic brace in the query and key. Now let&rsquo;s put all four pairs together and see how RoPE rotates the complete eight-feature vectors in one notice head. (Please awareness liberated to skip this division if it is too much specialized detail.)</p><p>Figure 24 picks up anywhere Figure 17 remaining off. We rotate all of the four pairs, afterward put the results rear in the query&rsquo;s first characteristic order.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!j9TT!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F114f482b-e080-4ce2-9698-f489a9b305eb_2774x2836.png 424w, https://substackcdn.com/image/fetch/$s_!j9TT!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F114f482b-e080-4ce2-9698-f489a9b305eb_2774x2836.png 848w, https://substackcdn.com/image/fetch/$s_!j9TT!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F114f482b-e080-4ce2-9698-f489a9b305eb_2774x2836.png 1272w, https://substackcdn.com/image/fetch/$s_!j9TT!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F114f482b-e080-4ce2-9698-f489a9b305eb_2774x2836.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F114f482b-e080-4ce2-9698-f489a9b305eb_2774x2836.png" width="1456" height="1489" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/114f482b-e080-4ce2-9698-f489a9b305eb_2774x2836.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1489,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 24 — four brace rotation&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 24 — four brace rotation" title="Figure 24 — four brace rotation" srcset="https://substackcdn.com/image/fetch/$s_!j9TT!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F114f482b-e080-4ce2-9698-f489a9b305eb_2774x2836.png 424w, https://substackcdn.com/image/fetch/$s_!j9TT!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F114f482b-e080-4ce2-9698-f489a9b305eb_2774x2836.png 848w, https://substackcdn.com/image/fetch/$s_!j9TT!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F114f482b-e080-4ce2-9698-f489a9b305eb_2774x2836.png 1272w, https://substackcdn.com/image/fetch/$s_!j9TT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F114f482b-e080-4ce2-9698-f489a9b305eb_2774x2836.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 24: RoPE rotates all brace of query features at its own angle. (The identical can be applied to the key, which is not shown to keep the example simple).</figcaption></figure></div><p><span>Figure 24 complete shows the rotated query </span><em>q&prime; </em><span>(at the bottom). Similarly, we could get the rotated key </span><em>k&prime;</em><span> by repeating the identical steps at the key&rsquo;s position.</span></p><p><span>Recall the two equations from Section 3.3. For query brace </span><em>i</em><span> alongside input values (</span><em><span>a</span><sub>i</sub></em><span>, </span><em><span>b</span><sub>i</sub></em><span>), let </span><em><span>c</span><sub>i</sub></em><span> = cos(</span><em><span>m&theta;</span><sub>i</sub></em><span>) and </span><em><span>s</span><sub>i</sub></em><span> = sin(</span><em><span>m&theta;</span><sub>i</sub></em><span>). These are the cosine and sine of the rotation viewpoint for that brace at query stance </span><em>m</em><span>:</span></p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ee1I!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd9437a4-f85c-4e46-b7c7-673e39afd841_2240x155.png 424w, https://substackcdn.com/image/fetch/$s_!ee1I!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd9437a4-f85c-4e46-b7c7-673e39afd841_2240x155.png 848w, https://substackcdn.com/image/fetch/$s_!ee1I!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd9437a4-f85c-4e46-b7c7-673e39afd841_2240x155.png 1272w, https://substackcdn.com/image/fetch/$s_!ee1I!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd9437a4-f85c-4e46-b7c7-673e39afd841_2240x155.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd9437a4-f85c-4e46-b7c7-673e39afd841_2240x155.png" width="1456" height="101" data-attrs="//substack-post-media.s3.amazonaws.com/public/images/dd9437a4-f85c-4e46-b7c7-673e39afd841_2240x155.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:101,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 13: \\begin{aligned} a_i&#039; &amp;= c_i a_i - s_i b_i, \\\\ b_i&#039; &amp;= s_i a_i + c_i b_i. \\end{aligned}&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" alt=" \begin{aligned} a_i&#039; &amp;= c_i a_i - s_i b_i, \\ b_i&#039; &amp;= s_i a_i + c_i b_i. \end{aligned}" title=" \begin{aligned} a_i&#039; &amp;= c_i a_i - s_i b_i, \\ b_i&#039; &amp;= s_i a_i + c_i b_i. \end{aligned}" srcset="https://substackcdn.com/image/fetch/$s_!ee1I!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd9437a4-f85c-4e46-b7c7-673e39afd841_2240x155.png 424w, https://substackcdn.com/image/fetch/$s_!ee1I!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd9437a4-f85c-4e46-b7c7-673e39afd841_2240x155.png 848w, https://substackcdn.com/image/fetch/$s_!ee1I!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd9437a4-f85c-4e46-b7c7-673e39afd841_2240x155.png 1272w, https://substackcdn.com/image/fetch/$s_!ee1I!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd9437a4-f85c-4e46-b7c7-673e39afd841_2240x155.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p><span>Figure 25 below shows the identical four rotations between the four pairs as a sole matrix multiplication, </span><em>q&prime;</em><span> = </span><em><span>R</span><sub>m</sub><span>q</span></em><span>.</span></p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!yWYT!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fd7a9e5-0263-4d31-bae4-787834c245b7_2634x2850.png 424w, https://substackcdn.com/image/fetch/$s_!yWYT!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fd7a9e5-0263-4d31-bae4-787834c245b7_2634x2850.png 848w, https://substackcdn.com/image/fetch/$s_!yWYT!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fd7a9e5-0263-4d31-bae4-787834c245b7_2634x2850.png 1272w, https://substackcdn.com/image/fetch/$s_!yWYT!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fd7a9e5-0263-4d31-bae4-787834c245b7_2634x2850.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fd7a9e5-0263-4d31-bae4-787834c245b7_2634x2850.png" width="1456" height="1575" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/5fd7a9e5-0263-4d31-bae4-787834c245b7_2634x2850.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1575,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 25 — complete rotation matrix&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 25 — complete rotation matrix" title="Figure 25 — complete rotation matrix" srcset="https://substackcdn.com/image/fetch/$s_!yWYT!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fd7a9e5-0263-4d31-bae4-787834c245b7_2634x2850.png 424w, https://substackcdn.com/image/fetch/$s_!yWYT!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fd7a9e5-0263-4d31-bae4-787834c245b7_2634x2850.png 848w, https://substackcdn.com/image/fetch/$s_!yWYT!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fd7a9e5-0263-4d31-bae4-787834c245b7_2634x2850.png 1272w, https://substackcdn.com/image/fetch/$s_!yWYT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fd7a9e5-0263-4d31-bae4-787834c245b7_2634x2850.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 25: The four rotations from Figure 24 written as one matrix multiplication. The brace labels display anywhere all rotation goes in the complete matrix.</figcaption></figure></div><p>A complete notice caput uses multiple frequencies so that several characteristic pairs alter additional drastically (&rdquo;quickly&rdquo; in RoPE terms) alongside stance and others alter additional gradually (&rdquo;slowly&rdquo; in RoPE terms). This serves the identical intent as the multiple frequencies in the first transformer&rsquo;s sinusoidal encodings.</p><p><span>However, alternatively of us assigning these manually, for a caput alongside </span><em>d</em><span> features, norm complete RoPE computes one frequence for all of its </span><em>d</em><span>/2 pairs as follows:</span></p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!UFXx!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 424w, https://substackcdn.com/image/fetch/$s_!UFXx!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 848w, https://substackcdn.com/image/fetch/$s_!UFXx!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 1272w, https://substackcdn.com/image/fetch/$s_!UFXx!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png" width="1456" height="55" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/ebff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:55,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 14: \\theta_i=b^{-2i/d}.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt=" \theta_i=b^{-2i/d}." title=" \theta_i=b^{-2i/d}." srcset="https://substackcdn.com/image/fetch/$s_!UFXx!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 424w, https://substackcdn.com/image/fetch/$s_!UFXx!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 848w, https://substackcdn.com/image/fetch/$s_!UFXx!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 1272w, https://substackcdn.com/image/fetch/$s_!UFXx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p>Here,</p><ul><li><p><em>d</em><span> is the caput size (it have to be equal so that all features can be arranged in pairs).</span></p></li><li><p><em>i</em><span> is the indicator of the pair; it ranges from 0 to </span><em>d</em><span>/2&minus;1.</span></p></li><li><p><em>b</em><span> is a chosen basis value (hyperparameter) that determines the spacing of the frequencies.</span></p></li><li><p><em><span>&theta;</span><sub>i</sub></em><span> is brace </span><em>i</em><span>&lsquo;s frequence in radians per token position.</span></p></li></ul><p><span>The first </span><a href="https://en.mycoding.id/category/technology" target="_blank">RoFormer paper</a><span> uses </span><em>b</em><span> = 10,000, borrowing the first transformer&rsquo;s frequence equation alongside caput size </span><em>d</em><span> in location of embedding width </span><em>d</em><sub>model</sub><span>.</span></p><p><span>For example, alongside caput size </span><em>d</em><span> = 8 and basis </span><em>b</em><span> = 10,000, we get frequencies [1, 0.1, 0.01, 0.001] radians per position.</span></p><p><span>If you desire to see RoPE in a complete model, I propose checking out my end-to-end </span><a href="https://en.mycoding.id/category/technology" target="_blank">Qwen3 from-scratch implementation</a><span>.</span></p><p><span>Since code can be an productive average for explaining and understanding concepts, the following subsection volition stroll you through the center parts that we have covered so far, if you are interested. Otherwise, delight awareness liberated to accelerated onward to division 6, on </span><em>Positional choices in contemporary architectures</em><span>.</span></p><p>So, for the following code snippets, we&rsquo;ll use the identical split-half rotation gathering as mentioned before (the interleaved gathering volition additionally be covered, no worries). First, we&rsquo;ll use it to one query vector, afterward we&rsquo;ll use it on queries and keys inner the notice mechanism.</p><p>First, let&rsquo;s reuse the 8-feature query from Figure 24. For convenience, below are its four characteristic pairs before rotation.</p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Wytv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bea76ec-3543-4b3a-a9ff-870d91b41255_1800x178.png 424w, https://substackcdn.com/image/fetch/$s_!Wytv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bea76ec-3543-4b3a-a9ff-870d91b41255_1800x178.png 848w, https://substackcdn.com/image/fetch/$s_!Wytv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bea76ec-3543-4b3a-a9ff-870d91b41255_1800x178.png 1272w, https://substackcdn.com/image/fetch/$s_!Wytv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bea76ec-3543-4b3a-a9ff-870d91b41255_1800x178.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bea76ec-3543-4b3a-a9ff-870d91b41255_1800x178.png" width="502" height="49.64835164835165" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/4bea76ec-3543-4b3a-a9ff-870d91b41255_1800x178.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:144,&quot;width&quot;:1456,&quot;resizeWidth&quot;:502,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Detail from Figure 24 — first query pairs&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Detail from Figure 24 — first query pairs" title="Detail from Figure 24 — first query pairs" srcset="https://substackcdn.com/image/fetch/$s_!Wytv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bea76ec-3543-4b3a-a9ff-870d91b41255_1800x178.png 424w, https://substackcdn.com/image/fetch/$s_!Wytv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bea76ec-3543-4b3a-a9ff-870d91b41255_1800x178.png 848w, https://substackcdn.com/image/fetch/$s_!Wytv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bea76ec-3543-4b3a-a9ff-870d91b41255_1800x178.png 1272w, https://substackcdn.com/image/fetch/$s_!Wytv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bea76ec-3543-4b3a-a9ff-870d91b41255_1800x178.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div><figcaption>Detail from Figure 24. The eight query features form four pairs in the split-half layout.</figcaption></figure></div><p><span>In the before non-coding example, we chose a frequence of 20&deg; per token stance for the archetypal brace to keep the math simple. Here, we&rsquo;ll use the first RoPE equation from the former Section 4 above, </span><em><span>&theta;</span><sub>i</sub></em><span> = </span><em>b</em><sup>&minus;2</sup><em><sup>i</sup></em><sup>/</sup><em><sup>d</sup></em><span>, alongside caput size </span><em>d</em><span> = 8 and basis </span><em>b</em><span> = 10,000.</span></p><p>Figure 26 shows how the two functions activity together to use these rotations.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!OZLU!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b99b926-16c6-443c-bcb0-155ba1e63fc2_2596x1542.png 424w, https://substackcdn.com/image/fetch/$s_!OZLU!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b99b926-16c6-443c-bcb0-155ba1e63fc2_2596x1542.png 848w, https://substackcdn.com/image/fetch/$s_!OZLU!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b99b926-16c6-443c-bcb0-155ba1e63fc2_2596x1542.png 1272w, https://substackcdn.com/image/fetch/$s_!OZLU!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b99b926-16c6-443c-bcb0-155ba1e63fc2_2596x1542.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b99b926-16c6-443c-bcb0-155ba1e63fc2_2596x1542.png" width="676" height="401.60714285714283" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/3b99b926-16c6-443c-bcb0-155ba1e63fc2_2596x1542.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:865,&quot;width&quot;:1456,&quot;resizeWidth&quot;:676,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 26 — pytorch walkthrough&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 26 — pytorch walkthrough" title="Figure 26 — pytorch walkthrough" srcset="https://substackcdn.com/image/fetch/$s_!OZLU!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b99b926-16c6-443c-bcb0-155ba1e63fc2_2596x1542.png 424w, https://substackcdn.com/image/fetch/$s_!OZLU!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b99b926-16c6-443c-bcb0-155ba1e63fc2_2596x1542.png 848w, https://substackcdn.com/image/fetch/$s_!OZLU!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b99b926-16c6-443c-bcb0-155ba1e63fc2_2596x1542.png 1272w, https://substackcdn.com/image/fetch/$s_!OZLU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b99b926-16c6-443c-bcb0-155ba1e63fc2_2596x1542.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption><span>Figure 26: </span><em>rope_params</em><span> returns the cosine and sine tables for token stance 2. </span><em>rope_split_half</em><span> takes those tables and the first query and returns the rotated query.</span></figcaption></figure></div><p><span>As shown in the fig above, first, </span>rope_params<span> determines how far to rotate all pair, returning two &ldquo;tables&rdquo; that shop the cosine and sine of those angles. Or, in another words, </span>rope_params<span> computes the cosine and sine of all rotation angle, which depends on the token&rsquo;s stance and the characteristic pair&rsquo;s frequency.</span></p><p><span>The </span>indices<span> gathering in the </span>rope_params<span> function contains the 2</span><em>i</em><span> values from the frequence equation in Section 4:</span></p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!UFXx!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 424w, https://substackcdn.com/image/fetch/$s_!UFXx!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 848w, https://substackcdn.com/image/fetch/$s_!UFXx!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 1272w, https://substackcdn.com/image/fetch/$s_!UFXx!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png" width="1456" height="55" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/ebff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:55,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 15: \\theta_i = b^{-2i/d}.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt=" \theta_i = b^{-2i/d}." title=" \theta_i = b^{-2i/d}." srcset="https://substackcdn.com/image/fetch/$s_!UFXx!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 424w, https://substackcdn.com/image/fetch/$s_!UFXx!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 848w, https://substackcdn.com/image/fetch/$s_!UFXx!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 1272w, https://substackcdn.com/image/fetch/$s_!UFXx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febff1490-d57b-4eb3-a837-0a3426a18f0c_2240x84.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p><span>For our eight-feature query, the </span>indices<span>are </span>[0, 2, 4, 6]<span>, that is, one 2</span><em>i</em><span> value for all characteristic pair. Suppose we use the average basis </span><em>b</em><span> = 10,000. And following our before examples, we choose a small caput size of </span><em>d</em><span> = 8. In this case, the </span>inv_freq<span> afterward computes the four brace frequencies </span><em><span>&theta;</span><sub>i</sub></em><span> = </span><em>b</em><sup>&minus;2</sup><em><sup>i</sup></em><sup>/</sup><em><sup>d</sup></em><span> as follows:</span></p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2Mbm!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a1fef03-0b62-4701-921b-2961ec0efd83_2240x121.png 424w, https://substackcdn.com/image/fetch/$s_!2Mbm!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a1fef03-0b62-4701-921b-2961ec0efd83_2240x121.png 848w, https://substackcdn.com/image/fetch/$s_!2Mbm!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a1fef03-0b62-4701-921b-2961ec0efd83_2240x121.png 1272w, https://substackcdn.com/image/fetch/$s_!2Mbm!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a1fef03-0b62-4701-921b-2961ec0efd83_2240x121.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a1fef03-0b62-4701-921b-2961ec0efd83_2240x121.png" width="1456" height="79" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/9a1fef03-0b62-4701-921b-2961ec0efd83_2240x121.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:79,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 16: [\\theta_0,\\theta_1,\\theta_2,\\theta_3] = \\left[10{,}000^{-0/8},\\;10{,}000^{-2/8},\\;10{,}000^{-4/8},\\;10{,}000^{-6/8}\\right] = [1,\\;0.1,\\;0.01,\\;0.001].&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt=" [\theta_0,\theta_1,\theta_2,\theta_3] = \left[10{,}000^{-0/8},\;10{,}000^{-2/8},\;10{,}000^{-4/8},\;10{,}000^{-6/8}\right] = [1,\;0.1,\;0.01,\;0.001]." title=" [\theta_0,\theta_1,\theta_2,\theta_3] = \left[10{,}000^{-0/8},\;10{,}000^{-2/8},\;10{,}000^{-4/8},\;10{,}000^{-6/8}\right] = [1,\;0.1,\;0.01,\;0.001]." srcset="https://substackcdn.com/image/fetch/$s_!2Mbm!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a1fef03-0b62-4701-921b-2961ec0efd83_2240x121.png 424w, https://substackcdn.com/image/fetch/$s_!2Mbm!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a1fef03-0b62-4701-921b-2961ec0efd83_2240x121.png 848w, https://substackcdn.com/image/fetch/$s_!2Mbm!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a1fef03-0b62-4701-921b-2961ec0efd83_2240x121.png 1272w, https://substackcdn.com/image/fetch/$s_!2Mbm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a1fef03-0b62-4701-921b-2961ec0efd83_2240x121.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p><span>As we can see above, the resulting </span>inv_freq<span> has one frequence </span><em><span>&theta;</span><sub>i</sub></em><span> for all of the four characteristic pairs ( measured in radians) per token position.</span></p><p><span>Next, the </span>angles<span> afterward multiplies all pair&rsquo;s frequence </span><em><span>&theta;</span><sub>i</sub></em><span> alongside the token stance </span><em>m</em><span>, which computes the rotation viewpoint </span><em><span>m&theta;</span><sub>i</sub></em><span>. In our example, alongside </span>positions = [2]<span>, we get:</span></p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!G-9t!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cc892f-79ac-4dec-84cd-039f81e722c9_2240x79.png 424w, https://substackcdn.com/image/fetch/$s_!G-9t!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cc892f-79ac-4dec-84cd-039f81e722c9_2240x79.png 848w, https://substackcdn.com/image/fetch/$s_!G-9t!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cc892f-79ac-4dec-84cd-039f81e722c9_2240x79.png 1272w, https://substackcdn.com/image/fetch/$s_!G-9t!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cc892f-79ac-4dec-84cd-039f81e722c9_2240x79.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cc892f-79ac-4dec-84cd-039f81e722c9_2240x79.png" width="1456" height="51" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/14cc892f-79ac-4dec-84cd-039f81e722c9_2240x79.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:51,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 17: [2\\theta_0,\\;2\\theta_1,\\;2\\theta_2,\\;2\\theta_3] = [2\\times1,\\;2\\times0.1,\\;2\\times0.01,\\;2\\times0.001] = [2,\\;0.2,\\;0.02,\\;0.002].&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt=" [2\theta_0,\;2\theta_1,\;2\theta_2,\;2\theta_3] = [2\times1,\;2\times0.1,\;2\times0.01,\;2\times0.001] = [2,\;0.2,\;0.02,\;0.002]." title=" [2\theta_0,\;2\theta_1,\;2\theta_2,\;2\theta_3] = [2\times1,\;2\times0.1,\;2\times0.01,\;2\times0.001] = [2,\;0.2,\;0.02,\;0.002]." srcset="https://substackcdn.com/image/fetch/$s_!G-9t!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cc892f-79ac-4dec-84cd-039f81e722c9_2240x79.png 424w, https://substackcdn.com/image/fetch/$s_!G-9t!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cc892f-79ac-4dec-84cd-039f81e722c9_2240x79.png 848w, https://substackcdn.com/image/fetch/$s_!G-9t!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cc892f-79ac-4dec-84cd-039f81e722c9_2240x79.png 1272w, https://substackcdn.com/image/fetch/$s_!G-9t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cc892f-79ac-4dec-84cd-039f81e722c9_2240x79.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p><span>Finally, </span>rope_params<span> takes the cosine and sine of all viewpoint and outputs them as two so-called cosine and sine tables, </span>cos<span> and </span>sin<span>. In the two tables, all row is a token stance and all pillar is a characteristic pair. Since we lone use token stance 2 here, all array has form </span>(1, 4)<span>, alongside the sole row shown below:</span></p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!bLWh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6c52934-fa04-4d92-84d0-7bad016d1208_2240x148.png 424w, https://substackcdn.com/image/fetch/$s_!bLWh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6c52934-fa04-4d92-84d0-7bad016d1208_2240x148.png 848w, https://substackcdn.com/image/fetch/$s_!bLWh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6c52934-fa04-4d92-84d0-7bad016d1208_2240x148.png 1272w, https://substackcdn.com/image/fetch/$s_!bLWh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6c52934-fa04-4d92-84d0-7bad016d1208_2240x148.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6c52934-fa04-4d92-84d0-7bad016d1208_2240x148.png" width="1456" height="96" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/a6c52934-fa04-4d92-84d0-7bad016d1208_2240x148.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:96,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 18: \\begin{aligned} \\texttt{cos}:\\quad [\\cos(2),\\;\\cos(0.2),\\;\\cos(0.02),\\;\\cos(0.002)] &amp;\\approx [-0.416147,\\;0.980067,\\;0.999800,\\;0.999998], \\\\ \\texttt{sin}:\\quad [\\sin(2),\\;\\sin(0.2),\\;\\sin(0.02),\\;\\sin(0.002)] &amp;\\approx [0.909297,\\;0.198669,\\;0.019999,\\;0.002000]. \\end{aligned}&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="\quad [\sin(2),\;\sin(0.2),\;\sin(0.02),\;\sin(0.002)] &amp;\approx [0.909297,\;0.198669,\;0.019999,\;0.002000]. \end{aligned}" title="\quad [\sin(2),\;\sin(0.2),\;\sin(0.02),\;\sin(0.002)] &amp;\approx [0.909297,\;0.198669,\;0.019999,\;0.002000]. \end{aligned}" srcset="https://substackcdn.com/image/fetch/$s_!bLWh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6c52934-fa04-4d92-84d0-7bad016d1208_2240x148.png 424w, https://substackcdn.com/image/fetch/$s_!bLWh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6c52934-fa04-4d92-84d0-7bad016d1208_2240x148.png 848w, https://substackcdn.com/image/fetch/$s_!bLWh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6c52934-fa04-4d92-84d0-7bad016d1208_2240x148.png 1272w, https://substackcdn.com/image/fetch/$s_!bLWh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6c52934-fa04-4d92-84d0-7bad016d1208_2240x148.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p>For the example in Figure 26, all array has one row for token stance 2 and four columns, one for all characteristic pair. Figure 27 shows the rotation viewpoint for all brace and the corresponding entries in the two tables.</p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1YrR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F871d5113-e3f8-407b-9747-5051495fa9d3_2676x1033.png 424w, https://substackcdn.com/image/fetch/$s_!1YrR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F871d5113-e3f8-407b-9747-5051495fa9d3_2676x1033.png 848w, https://substackcdn.com/image/fetch/$s_!1YrR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F871d5113-e3f8-407b-9747-5051495fa9d3_2676x1033.png 1272w, https://substackcdn.com/image/fetch/$s_!1YrR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F871d5113-e3f8-407b-9747-5051495fa9d3_2676x1033.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F871d5113-e3f8-407b-9747-5051495fa9d3_2676x1033.png" width="592" height="228.5054945054945" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/871d5113-e3f8-407b-9747-5051495fa9d3_2676x1033.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:562,&quot;width&quot;:1456,&quot;resizeWidth&quot;:592,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 27 — cos sin tables&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 27 — cos sin tables" title="Figure 27 — cos sin tables" srcset="https://substackcdn.com/image/fetch/$s_!1YrR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F871d5113-e3f8-407b-9747-5051495fa9d3_2676x1033.png 424w, https://substackcdn.com/image/fetch/$s_!1YrR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F871d5113-e3f8-407b-9747-5051495fa9d3_2676x1033.png 848w, https://substackcdn.com/image/fetch/$s_!1YrR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F871d5113-e3f8-407b-9747-5051495fa9d3_2676x1033.png 1272w, https://substackcdn.com/image/fetch/$s_!1YrR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F871d5113-e3f8-407b-9747-5051495fa9d3_2676x1033.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div><figcaption><span>Figure 27: </span><em>The</em><span> two tables returned by </span><em>rope_params</em><span> for token stance 2. </span><em>The</em><span> highlighted pillar gives the cosine and sine used to rotate the archetypal characteristic pair, </span><em>(1, 5)</em><span>.</span></figcaption></figure></div><p><span>In short, </span>rope_params<span> computes the cosine and sine of all rotation angle, which depends on the token&rsquo;s stance and the characteristic pair&rsquo;s frequency.</span></p><p><span>In a real-world scenario, if we continue </span>rope_params<span> 1,024 token positions for a environment size of 1024, and a caput size of 128, it returns two tensors, </span>cos<span> and </span>sin<span>, all alongside form </span>(1024, 64)<span> since a caput alongside 128 features has 64 pairs.</span></p><p><span>The tables have no lot or caput dimension, and we broadcast all </span>(1024, 64)<span> array throughout the lot and caput dimensions. So, if we have multi-head notice or grouped query notice alongside 32 heads, characteristic brace 0 at token stance 2 uses the identical rotation viewpoint in all 32 heads. Each caput rotates its own characteristic values, so the rotated queries can motionless differ throughout heads. Also, the identical entries can be reused throughout queries, keys, and heads that use the identical positions and frequencies. In short, we lone need to (pre)compute these tables formerly per example for all positions up to a maximum supported environment size.</span></p><p>Technically, it would additionally be imaginable to shop these tables alongside the example weights; but they are so incredibly cheap to compute that storing them on disk would be overkill.</p><p><span>Next, </span>rope_split_half<span> uses those cosines and sines to rotate the genuine features in the query and key vectors inner attention.</span></p><p><span>Considering the uncomplicated 8-feature example formerly more, the </span>.chunk<span> in the code snippet complete splits the query into </span>[1, 2, 3, 4]<span> and </span>[5, 6, 7, 8]<span> and lines up the two entries in all pair. For example, the archetypal slot of all fractional gives us </span>(1, 5)<span>, the second </span>(2, 6)<span>, and so on. The two expressions inner </span>torch.cat<span> compute the rotated halves using the equations from Section 3.3:</span></p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!cIqH!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbcf9b781-1fac-40b7-83e7-54f6c36e3fec_2240x155.png 424w, https://substackcdn.com/image/fetch/$s_!cIqH!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbcf9b781-1fac-40b7-83e7-54f6c36e3fec_2240x155.png 848w, https://substackcdn.com/image/fetch/$s_!cIqH!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbcf9b781-1fac-40b7-83e7-54f6c36e3fec_2240x155.png 1272w, https://substackcdn.com/image/fetch/$s_!cIqH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbcf9b781-1fac-40b7-83e7-54f6c36e3fec_2240x155.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbcf9b781-1fac-40b7-83e7-54f6c36e3fec_2240x155.png" width="1456" height="101" data-attrs="//substack-post-media.s3.amazonaws.com/public/images/bcf9b781-1fac-40b7-83e7-54f6c36e3fec_2240x155.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:101,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation 19: \\begin{aligned} a_i&#039; &amp;= a_i\\cos(m\\theta_i) - b_i\\sin(m\\theta_i), \\\\ b_i&#039; &amp;= a_i\\sin(m\\theta_i) + b_i\\cos(m\\theta_i). \\end{aligned}&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" alt=" \begin{aligned} a_i&#039; &amp;= a_i\cos(m\theta_i) - b_i\sin(m\theta_i), \\ b_i&#039; &amp;= a_i\sin(m\theta_i) + b_i\cos(m\theta_i). \end{aligned}" title=" \begin{aligned} a_i&#039; &amp;= a_i\cos(m\theta_i) - b_i\sin(m\theta_i), \\ b_i&#039; &amp;= a_i\sin(m\theta_i) + b_i\cos(m\theta_i). \end{aligned}" srcset="https://substackcdn.com/image/fetch/$s_!cIqH!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbcf9b781-1fac-40b7-83e7-54f6c36e3fec_2240x155.png 424w, https://substackcdn.com/image/fetch/$s_!cIqH!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbcf9b781-1fac-40b7-83e7-54f6c36e3fec_2240x155.png 848w, https://substackcdn.com/image/fetch/$s_!cIqH!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbcf9b781-1fac-40b7-83e7-54f6c36e3fec_2240x155.png 1272w, https://substackcdn.com/image/fetch/$s_!cIqH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbcf9b781-1fac-40b7-83e7-54f6c36e3fec_2240x155.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div></figure></div><p><span>Note that here, everything denoted alongside </span><em><span>a</span><sub>i</sub></em><span> corresponds to the archetypal fractional of the query, and everything denoted alongside </span><em><span>b</span><sub>i</sub></em><span> comes from the second half. The archetypal expression inner </span>torch.cat<span> computes all the </span><em><span>a</span><sub>i</sub><span>&prime;</span></em><span>values, and the second computes all the </span><em><span>b</span><sub>i</sub><span>&prime;</span></em><span> values.</span></p><p><span>Then, we concatenate the halves, which puts all value rear in its first indicator stance via </span>torch.cat<span>.</span></p><p><span>Here&rsquo;s the code that produces the rotated query in the before Figure 26. The shown reshape </span>(1, 1, 1, 8)<span> fair brings the uncomplicated toy gathering into the expected format: one lot item, one head, one token, and eight features.</span></p><p><span>With multiple positions, all token merely picks up its own row from the </span>cos<span> and </span>sin<span> tables. Through broadcasting, PyTorch reuses these identical tables throughout all lot items and notice heads.</span></p><p><span>So far, we coded the RoPE usefulness functions. Let&rsquo;s now use it in the genuine context, the notice mechanism. Here, we use the norm frequence agenda for two 8-feature heads and four token positions. We use random query, key, and value tensors to keep the example self-contained, but as mentioned before, you can optionally additionally inspect the end-to-end </span><a href="https://en.mycoding.id/category/technology" target="_blank">Qwen3 from-scratch implementation</a><span> alongside genuine query, key, and values produced by the pre-trained model.</span></p><p><span>Note that lone </span>q<span> and </span>k<span> but not </span>v<span> continue through RoPE. The resulting raw notice scores are stored in a tensor of form </span>(1, 2, 4, 4)<span>, that is, one row for all query position, and a pillar for all key position, and 4x4 since we have four tokens anywhere all attends to all other.</span></p><p><span>Then, using a accepted notice code implementation, the causal disguise replaces the scores for forthcoming keys alongside negative infinity, so they get zeroed following the softmax. The resulting </span>context<span> tensor has form </span>(1, 2, 4, 8)<span>; the 4 since we have four tokens, and the 8 since the embedding size was initially 8.</span></p><p>We can additionally inspect that our assertion concerning related offsets holds, definition that adding 3 to all stance should alter the stance but maintain the query-key offsets:</p><p><span>I mentioned the </span><em>split-halves</em><span> gathering many times throughout this article. The </span><a href="https://en.mycoding.id/category/technology" target="_blank">original RoPE paper</a><span> uses the substitute </span><em>interleaved</em><span> convention, as does the </span><a href="https://en.mycoding.id/category/technology" target="_blank">original LLaMA implementation</a><span> (but as mentioned before, most code implementations now use the split-halves, since it makes it easier, in code, to rotate all pairs at formerly alongside element-wise sine and cosine operations.)</span></p><p>To exemplify the difference between the two layouts, Figure 28 below uses the identical 8-feature vector in the two rows.</p><div><figure><div data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!pSrd!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F376d895d-ffc0-41ec-8cf5-5e84834412fc_2641x804.png 424w, https://substackcdn.com/image/fetch/$s_!pSrd!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F376d895d-ffc0-41ec-8cf5-5e84834412fc_2641x804.png 848w, https://substackcdn.com/image/fetch/$s_!pSrd!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F376d895d-ffc0-41ec-8cf5-5e84834412fc_2641x804.png 1272w, https://substackcdn.com/image/fetch/$s_!pSrd!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F376d895d-ffc0-41ec-8cf5-5e84834412fc_2641x804.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F376d895d-ffc0-41ec-8cf5-5e84834412fc_2641x804.png" width="598" height="181.94642857142858" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/376d895d-ffc0-41ec-8cf5-5e84834412fc_2641x804.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:443,&quot;width&quot;:1456,&quot;resizeWidth&quot;:598,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 28 — characteristic pairing&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 28 — characteristic pairing" title="Figure 28 — characteristic pairing" srcset="https://substackcdn.com/image/fetch/$s_!pSrd!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F376d895d-ffc0-41ec-8cf5-5e84834412fc_2641x804.png 424w, https://substackcdn.com/image/fetch/$s_!pSrd!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F376d895d-ffc0-41ec-8cf5-5e84834412fc_2641x804.png 848w, https://substackcdn.com/image/fetch/$s_!pSrd!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F376d895d-ffc0-41ec-8cf5-5e84834412fc_2641x804.png 1272w, https://substackcdn.com/image/fetch/$s_!pSrd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F376d895d-ffc0-41ec-8cf5-5e84834412fc_2641x804.png 1456w" sizes="auto, 100vw" loading="lazy"></div></div><figcaption>Figure 28: Split-half pairs all characteristic in the archetypal fractional alongside the corresponding characteristic in the second half. Interleaved pairs neighboring features.</figcaption></figure></div><p>The interleaved implementation uses the identical two equations, but selects alternating features and interleaves the outputs.</p><p><span>But mathematically, the layouts are equal whenever we document for the characteristic order. So, rearranging the entries </span>[1, 2, 3, 4, 5, 6, 7, 8]<span> to </span>[1, 5, 2, 6, 3, 7, 4, 8]<span> makes the split-half pairs neighboring alongside pairs (1, 5), (2, 6), and so on. If we use the identical rearrangement to the two queries and keys, it results in identical dot products.</span></p><p>The outputs keep the reordered features, so we difference them alongside the split-half outputs in that identical order. Applying the two layouts to the first vector nevertheless would provision distinct scores.</p><p><span>RoPE has been a staple in most (open-weight) LLMs since the </span><a href="https://en.mycoding.id/category/technology" target="_blank">original LLaMA model</a><span>. Of course, proprietary models may use it as well, but it&rsquo;s continually difficult to say item alongside 100% certainty concerning proprietary models.</span></p><p>However, in latest years, many architectures have been moving distant from full-blown RoPE in all transformer block. In the following subsection, we volition conversation several of these variants.</p><p><span>For case the complete RoPE implementation we discussed and implemented in the sections complete rotates </span><em>every</em><span> characteristic brace in </span><em>every</em><span> query and key head.</span></p><p>In the following examples in the subsections complete below, we volition see that models lone rotate several features inside a head, another models skip RoPE in several transformer blocks, and several models use a distinct method and add stance data immediately to their notice scores.</p><p>As discussed throughout the article, regular &ldquo;full&rdquo; RoPE rotates all dimensions in all query and key head. Partial RoPE is a type that rotates lone a subset, during leaving the remaining dimensions unchanged.</p><p><span>For example, </span><a href="https://en.mycoding.id/category/technology" target="_blank">MiniMax-M3&rsquo;s configuration</a><span> uses 64 rotary dimensions inside all 128-dimensional head. That gives 32 rotated characteristic pairs and 64 unchanged dimensions. The fig below illustrates how that looks like.</span></p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HikP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4515ea8c-8759-4aca-a824-64f0f42d080e_2616x1657.png 424w, https://substackcdn.com/image/fetch/$s_!HikP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4515ea8c-8759-4aca-a824-64f0f42d080e_2616x1657.png 848w, https://substackcdn.com/image/fetch/$s_!HikP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4515ea8c-8759-4aca-a824-64f0f42d080e_2616x1657.png 1272w, https://substackcdn.com/image/fetch/$s_!HikP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4515ea8c-8759-4aca-a824-64f0f42d080e_2616x1657.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4515ea8c-8759-4aca-a824-64f0f42d080e_2616x1657.png" width="1456" height="922" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/4515ea8c-8759-4aca-a824-64f0f42d080e_2616x1657.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:922,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 29 — partial&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 29 — partial" title="Figure 29 — partial" srcset="https://substackcdn.com/image/fetch/$s_!HikP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4515ea8c-8759-4aca-a824-64f0f42d080e_2616x1657.png 424w, https://substackcdn.com/image/fetch/$s_!HikP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4515ea8c-8759-4aca-a824-64f0f42d080e_2616x1657.png 848w, https://substackcdn.com/image/fetch/$s_!HikP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4515ea8c-8759-4aca-a824-64f0f42d080e_2616x1657.png 1272w, https://substackcdn.com/image/fetch/$s_!HikP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4515ea8c-8759-4aca-a824-64f0f42d080e_2616x1657.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 29: Full RoPE and partial RoPE in a 128-dimensional head.</figcaption></figure></div><p>Note that if we appearance former the archetypal transformer block, the unrotated features can motionless merge stance data inherited from before transformer blocks.</p><p>Anyways, what&rsquo;s the item rearward partial RoPE? It saves several memory. For example, in the MiniMax-M3 example above, rotating fractional the features reduces the size of the precomputed cosine and sine tables additionally in half. So, alongside partial RoPE, alternatively of 64 columns, now all array lone needs 32 columns. However, note that the KV cache motionless stores the complete key and value vectors for all cached token.</p><p>For MiniMax-M3&rsquo;s 1M-token context, rotating 64 of the 128 features halves the size of the precomputed cosine and sine tables, preservation 0.134 GB (128 MiB) if the tables are stored in BF16. If the tables themselves were stored in NVFP4, the funds would be 0.038 GB (36 MiB), including the obstacle scales.</p><p><span>Note that in experiments alongside 1B to 8B models, the </span><em><a href="https://en.mycoding.id/category/technology" target="_blank">Fractional Rotation, Full Potential?</a></em><span> document established that rotating about 10% of all head&rsquo;s features frequently gave akin training defeat and benchmark accuracy to complete RoPE. However, complete RoPE did improved on several evaluations.</span></p><p>So, in my opinion, partial RoPE is barely value it.</p><p><span>NoPE was coined in the </span><a href="https://en.mycoding.id/category/technology" target="_blank">The Impact of Positional Encoding on Length Generalization in Transformers</a><span> document and method </span><em>no definitive positional encoding</em><span>. The idea or study is that an notice tier without any definitive stance method motionless has a causal mask, anywhere a query can act lone to itself and before tokens. And this causal masking implicitly provides positional information.</span></p><p><span>Let&rsquo;s use the archetypal four tokens from our before example, </span>The red fox jumps<span>. In Figure 30, the query for </span>fox<span> at stance 2 can act to </span>The<span>, </span>red<span>, and itself. The query for </span>red<span> at stance 1 can act to </span>The<span> and itself.</span></p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2NXi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ebf07fb-5de9-4446-abb3-4fdd15d1f6b1_2357x1614.png 424w, https://substackcdn.com/image/fetch/$s_!2NXi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ebf07fb-5de9-4446-abb3-4fdd15d1f6b1_2357x1614.png 848w, https://substackcdn.com/image/fetch/$s_!2NXi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ebf07fb-5de9-4446-abb3-4fdd15d1f6b1_2357x1614.png 1272w, https://substackcdn.com/image/fetch/$s_!2NXi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ebf07fb-5de9-4446-abb3-4fdd15d1f6b1_2357x1614.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ebf07fb-5de9-4446-abb3-4fdd15d1f6b1_2357x1614.png" width="522" height="357.4409340659341" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/8ebf07fb-5de9-4446-abb3-4fdd15d1f6b1_2357x1614.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:997,&quot;width&quot;:1456,&quot;resizeWidth&quot;:522,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 30 — nope prefix mask&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 30 — nope prefix mask" title="Figure 30 — nope prefix mask" srcset="https://substackcdn.com/image/fetch/$s_!2NXi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ebf07fb-5de9-4446-abb3-4fdd15d1f6b1_2357x1614.png 424w, https://substackcdn.com/image/fetch/$s_!2NXi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ebf07fb-5de9-4446-abb3-4fdd15d1f6b1_2357x1614.png 848w, https://substackcdn.com/image/fetch/$s_!2NXi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ebf07fb-5de9-4446-abb3-4fdd15d1f6b1_2357x1614.png 1272w, https://substackcdn.com/image/fetch/$s_!2NXi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ebf07fb-5de9-4446-abb3-4fdd15d1f6b1_2357x1614.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 30: Example of a causal disguise in the notice mechanism.</figcaption></figure></div><p>So, the next inquiry is how can the example use the causal disguise as a stance signal? Here&rsquo;s a toy example adapted from the paper.</p><p><span>Suppose one characteristic of the value vectors is 1 for </span>The<span> and 0 for the another three tokens. Since the input lone contains lone one </span>The<span>, all apparent prefix contains exactly one 1. If all non-masked keys get the identical notice score, the softmax assigns all apparent token equal weight.</span></p><p><span>So at </span>fox<span>, notice averages 1, 0, and 0, which gives 1/3. At </span>jumps<span>, it averages four values and gets 1/4. In another words, this output characteristic equals one complete the prefix length, which a afterward tier can use as a indication for position.</span></p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!QWKi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae9e1c64-4cd3-4184-a0d8-b17873dd710a_2640x1386.png 424w, https://substackcdn.com/image/fetch/$s_!QWKi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae9e1c64-4cd3-4184-a0d8-b17873dd710a_2640x1386.png 848w, https://substackcdn.com/image/fetch/$s_!QWKi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae9e1c64-4cd3-4184-a0d8-b17873dd710a_2640x1386.png 1272w, https://substackcdn.com/image/fetch/$s_!QWKi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae9e1c64-4cd3-4184-a0d8-b17873dd710a_2640x1386.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae9e1c64-4cd3-4184-a0d8-b17873dd710a_2640x1386.png" width="1456" height="764" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/ae9e1c64-4cd3-4184-a0d8-b17873dd710a_2640x1386.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:764,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 31 — nope prefix average&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 31 — nope prefix average" title="Figure 31 — nope prefix average" srcset="https://substackcdn.com/image/fetch/$s_!QWKi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae9e1c64-4cd3-4184-a0d8-b17873dd710a_2640x1386.png 424w, https://substackcdn.com/image/fetch/$s_!QWKi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae9e1c64-4cd3-4184-a0d8-b17873dd710a_2640x1386.png 848w, https://substackcdn.com/image/fetch/$s_!QWKi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae9e1c64-4cd3-4184-a0d8-b17873dd710a_2640x1386.png 1272w, https://substackcdn.com/image/fetch/$s_!QWKi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae9e1c64-4cd3-4184-a0d8-b17873dd710a_2640x1386.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption><span>Figure 31: With uniform attention, the contribution from </span><em>The</em><span> shrinks as the prefix grows. </span><em>The</em><span> query for </span><em>fox</em><span> assigns </span><em>The</em><span> a importance of 1/3.</span></figcaption></figure></div><p>Does NoPE activity in practice? The researchers of the NoPE document trained a small 107M-parameter decoder-only transformers from scratch on tasks specified as copying, sorting, and supplement to do several ablation.</p><p><span>When they tested the small example on examples longer than those seen during training, interestingly, NoPE classified archetypal general throughout the ten tasks and onward of RoPE. The researchers rearward the </span><a href="https://en.mycoding.id/category/technology" target="_blank">NoPE paper</a><span> additionally conducted a distinct test alongside a larger 1.3B-parameter models trained on code. Here, NoPE additionally handled contexts beyond the training length, however, its next-token predictions got worse at the longest lengths tested. </span></p><p>Some of today&rsquo;s open-weight LLMs equal go so far and use NoPE in all layers. However, nowadays, it&rsquo;s additional average for models to omit RoPE in at smallest several layers, anywhere tier current method transformer block. </p><p><span>To provision a tangible example, </span><a href="https://en.mycoding.id/category/technology" target="_blank">SmolLM3</a><span> (3B parameters) applies RoPE in three layers and skips it in the fourth, repeating that form through 36 layers. Interestingly, the SmolLM3 authors mentioned that this tier form improved long-context performance.</span></p><p>(Also note that any afterward layer, following the archetypal one, equal although it may skip the rotation, motionless receives the inputs from preceding transformer blocks, so its queries and keys motionless obtain several kind of stance data from before layers.)</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!iLab!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26b1e107-4221-4ca8-a579-ac2603366c99_2804x1737.png 424w, https://substackcdn.com/image/fetch/$s_!iLab!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26b1e107-4221-4ca8-a579-ac2603366c99_2804x1737.png 848w, https://substackcdn.com/image/fetch/$s_!iLab!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26b1e107-4221-4ca8-a579-ac2603366c99_2804x1737.png 1272w, https://substackcdn.com/image/fetch/$s_!iLab!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26b1e107-4221-4ca8-a579-ac2603366c99_2804x1737.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26b1e107-4221-4ca8-a579-ac2603366c99_2804x1737.png" width="1456" height="902" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/26b1e107-4221-4ca8-a579-ac2603366c99_2804x1737.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:902,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 32 — smollm3 tier pattern&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 32 — smollm3 tier pattern" title="Figure 32 — smollm3 tier pattern" srcset="https://substackcdn.com/image/fetch/$s_!iLab!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26b1e107-4221-4ca8-a579-ac2603366c99_2804x1737.png 424w, https://substackcdn.com/image/fetch/$s_!iLab!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26b1e107-4221-4ca8-a579-ac2603366c99_2804x1737.png 848w, https://substackcdn.com/image/fetch/$s_!iLab!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26b1e107-4221-4ca8-a579-ac2603366c99_2804x1737.png 1272w, https://substackcdn.com/image/fetch/$s_!iLab!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26b1e107-4221-4ca8-a579-ac2603366c99_2804x1737.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption><span>Figure 32: SmolLM3&rsquo;s architecture (left) and the positional method used in all of its 36 transformer blocks (right). Every fourth obstacle skips RoPE. Architecture diagram from my </span><a href="https://en.mycoding.id/category/technology" target="_blank">LLM Architecture Gallery</a><span>.</span></figcaption></figure></div><p><span>Before we get to the overview and appearance at what contemporary architectures use, there&rsquo;s one additional method to introduce. For instance, Thinky&rsquo;s </span><a href="https://en.mycoding.id/category/technology" target="_blank">Inkling</a><span> uses a learned related stance bias in the two its sliding-window and earth notice layers. It computes the customary scaled dot merchandise between a query and key, and afterward it adds a certain figure to that mark before the softmax in the notice mark computation. That figure depends on the query token&rsquo;s hidden province and how many positions rear the key is.</span></p><p>Let&rsquo;s reuse our example sentence, &ldquo;The red fox jumps complete the gray fox.&rdquo; (It&rsquo;s the final time, I promise!) In the fig below, the &ldquo;the&rdquo; at stance 5 is the query and &ldquo;fox&rdquo; at stance 2 is the key. Inkling afterward uses the three-position gap between query and key and the hidden province for &ldquo;the&rdquo; to compute the bias, afterward adds it to the scaled dot product.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!U-18!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb6d1530-6222-49d1-a269-675b1e013253_2658x1232.png 424w, https://substackcdn.com/image/fetch/$s_!U-18!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb6d1530-6222-49d1-a269-675b1e013253_2658x1232.png 848w, https://substackcdn.com/image/fetch/$s_!U-18!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb6d1530-6222-49d1-a269-675b1e013253_2658x1232.png 1272w, https://substackcdn.com/image/fetch/$s_!U-18!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb6d1530-6222-49d1-a269-675b1e013253_2658x1232.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb6d1530-6222-49d1-a269-675b1e013253_2658x1232.png" width="1456" height="675" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/eb6d1530-6222-49d1-a269-675b1e013253_2658x1232.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:675,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 33 — related stance bias&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 33 — related stance bias" title="Figure 33 — related stance bias" srcset="https://substackcdn.com/image/fetch/$s_!U-18!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb6d1530-6222-49d1-a269-675b1e013253_2658x1232.png 424w, https://substackcdn.com/image/fetch/$s_!U-18!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb6d1530-6222-49d1-a269-675b1e013253_2658x1232.png 848w, https://substackcdn.com/image/fetch/$s_!U-18!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb6d1530-6222-49d1-a269-675b1e013253_2658x1232.png 1272w, https://substackcdn.com/image/fetch/$s_!U-18!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb6d1530-6222-49d1-a269-675b1e013253_2658x1232.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 33: Inkling adds a learned related stance bias to the scaled query-key dot merchandise before softmax. The numbers are illustrative.</figcaption></figure></div><p><span>This is not an entirely new idea. Some of you may recall Google&rsquo;s famous </span><a href="https://en.mycoding.id/category/technology" target="_blank">T5</a><span> (Text-to-Text Transfer Transformer) in 2019, which used a simpler type of this idea.</span></p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!PMsu!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d4cfb41-fe76-49ff-b58f-f88c2bec5f5e_2689x2772.png 424w, https://substackcdn.com/image/fetch/$s_!PMsu!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d4cfb41-fe76-49ff-b58f-f88c2bec5f5e_2689x2772.png 848w, https://substackcdn.com/image/fetch/$s_!PMsu!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d4cfb41-fe76-49ff-b58f-f88c2bec5f5e_2689x2772.png 1272w, https://substackcdn.com/image/fetch/$s_!PMsu!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d4cfb41-fe76-49ff-b58f-f88c2bec5f5e_2689x2772.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d4cfb41-fe76-49ff-b58f-f88c2bec5f5e_2689x2772.png" width="666" height="686.5837912087912" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/2d4cfb41-fe76-49ff-b58f-f88c2bec5f5e_2689x2772.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1501,&quot;width&quot;:1456,&quot;resizeWidth&quot;:666,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 34 — inkling architecture&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 34 — inkling architecture" title="Figure 34 — inkling architecture" srcset="https://substackcdn.com/image/fetch/$s_!PMsu!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d4cfb41-fe76-49ff-b58f-f88c2bec5f5e_2689x2772.png 424w, https://substackcdn.com/image/fetch/$s_!PMsu!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d4cfb41-fe76-49ff-b58f-f88c2bec5f5e_2689x2772.png 848w, https://substackcdn.com/image/fetch/$s_!PMsu!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d4cfb41-fe76-49ff-b58f-f88c2bec5f5e_2689x2772.png 1272w, https://substackcdn.com/image/fetch/$s_!PMsu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d4cfb41-fe76-49ff-b58f-f88c2bec5f5e_2689x2772.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption><span>Figure 34: The first Inkling (975B) architecture. The learned related stance bias feeds into the notice module. </span><a href="https://en.mycoding.id/category/technology" target="_blank">LLM Architecture Gallery</a></figcaption></figure></div><p>Now, following all this build up, the engaging inquiry is: which architecture uses which method? And what are the latest trends?</p><p><span>To answer this, I grouped the entries in my LLM Architecture Gallery by their listed publish dates and compared the 2025 releases alongside the before and afterward releases in the 2026 snapshot. Each exhibition cardstock counts once, so multiple variants of an architecture can contribute to the results. </span><a href="https://en.mycoding.id/category/technology" target="_blank">LLM Architecture Gallery</a></p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!eqMT!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb16bfe6e-e42b-4c9a-917e-2380d612d02a_2705x2244.png 424w, https://substackcdn.com/image/fetch/$s_!eqMT!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb16bfe6e-e42b-4c9a-917e-2380d612d02a_2705x2244.png 848w, https://substackcdn.com/image/fetch/$s_!eqMT!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb16bfe6e-e42b-4c9a-917e-2380d612d02a_2705x2244.png 1272w, https://substackcdn.com/image/fetch/$s_!eqMT!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb16bfe6e-e42b-4c9a-917e-2380d612d02a_2705x2244.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb16bfe6e-e42b-4c9a-917e-2380d612d02a_2705x2244.png" width="1456" height="1208" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/b16bfe6e-e42b-4c9a-917e-2380d612d02a_2705x2244.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1208,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 35 — exhibition stance methods&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 35 — exhibition stance methods" title="Figure 35 — exhibition stance methods" srcset="https://substackcdn.com/image/fetch/$s_!eqMT!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb16bfe6e-e42b-4c9a-917e-2380d612d02a_2705x2244.png 424w, https://substackcdn.com/image/fetch/$s_!eqMT!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb16bfe6e-e42b-4c9a-917e-2380d612d02a_2705x2244.png 848w, https://substackcdn.com/image/fetch/$s_!eqMT!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb16bfe6e-e42b-4c9a-917e-2380d612d02a_2705x2244.png 1272w, https://substackcdn.com/image/fetch/$s_!eqMT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb16bfe6e-e42b-4c9a-917e-2380d612d02a_2705x2244.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption><span>Figure 35: The architecture diagrams exemplify which notice layers use RoPE, and the bars display the portion of exhibition entries in all category. Diagrams from my </span><a href="https://en.mycoding.id/category/technology" target="_blank">LLM Architecture Gallery</a><span>.</span></figcaption></figure></div><p><span>As we can see in Figure 35 above, using RoPE in </span><em>every</em><span> softmax notice tier is motionless the most average choice. However, its use cut from 2025 to 2026, and afterward additionally in the second fractional of 2026, which could be a trend.</span></p><p>Consequently, there is a expanding figure of models that mix RoPE and NoPE. The groups are motionless small, so it&rsquo;s difficult to say whether these trends volition continue.</p><p><span>The </span><em>NoPE only</em><span> entries were all recurrent hybrids alongside softmax notice layers that use no definitive positional encoding. The &ldquo;Other&rdquo; category current refers to xLSTM 7B (2025) and Inkling (2026). xLSTM has no softmax notice layers. Inkling adds a learned bias to the notice scores before softmax, as we saw in Section 6.3.</span></p><p>Among the models that use RoPE, partial RoPE became additional average from 2025 to 2026. It went from a smaller group of the RoPE models in 2025 to a bulk in the two 2026 groups, as shown in Figure 36.</p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!p9wN!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16975bd7-c1e0-443b-8241-63f91bc7e303_2734x938.png 424w, https://substackcdn.com/image/fetch/$s_!p9wN!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16975bd7-c1e0-443b-8241-63f91bc7e303_2734x938.png 848w, https://substackcdn.com/image/fetch/$s_!p9wN!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16975bd7-c1e0-443b-8241-63f91bc7e303_2734x938.png 1272w, https://substackcdn.com/image/fetch/$s_!p9wN!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16975bd7-c1e0-443b-8241-63f91bc7e303_2734x938.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16975bd7-c1e0-443b-8241-63f91bc7e303_2734x938.png" width="1456" height="500" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/16975bd7-c1e0-443b-8241-63f91bc7e303_2734x938.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:500,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 36 — exhibition partial rope&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 36 — exhibition partial rope" title="Figure 36 — exhibition partial rope" srcset="https://substackcdn.com/image/fetch/$s_!p9wN!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16975bd7-c1e0-443b-8241-63f91bc7e303_2734x938.png 424w, https://substackcdn.com/image/fetch/$s_!p9wN!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16975bd7-c1e0-443b-8241-63f91bc7e303_2734x938.png 848w, https://substackcdn.com/image/fetch/$s_!p9wN!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16975bd7-c1e0-443b-8241-63f91bc7e303_2734x938.png 1272w, https://substackcdn.com/image/fetch/$s_!p9wN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16975bd7-c1e0-443b-8241-63f91bc7e303_2734x938.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 36: Among exhibition entries that use RoPE, partial RoPE rotates a subset of query and key features in at smallest one notice layer. Full-width entries rotate all query and key features in all tier that uses RoPE.</figcaption></figure></div><p><span>If you&rsquo;d akin to appearance up an idiosyncratic example from the 2026 groups, Figure 37 below shows the 2026 exhibition overview alongside the idiosyncratic architectures tagged by stance type. You can additionally browse the exhibition interactively </span><a href="https://en.mycoding.id/category/technology" target="_blank">here</a><span>.</span></p><div><figure><a href="https://en.mycoding.id/category/technology" target="_blank" data-component-name="Image2ToDOM"><div><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!X1PL!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3dbac37d-1290-4ab9-92a8-6fe7f54d62e2_2850x2777.png 424w, https://substackcdn.com/image/fetch/$s_!X1PL!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3dbac37d-1290-4ab9-92a8-6fe7f54d62e2_2850x2777.png 848w, https://substackcdn.com/image/fetch/$s_!X1PL!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3dbac37d-1290-4ab9-92a8-6fe7f54d62e2_2850x2777.png 1272w, https://substackcdn.com/image/fetch/$s_!X1PL!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3dbac37d-1290-4ab9-92a8-6fe7f54d62e2_2850x2777.png 1456w" sizes="auto, 100vw"></source><img src="https://magazine.sebastianraschka.com/p/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3dbac37d-1290-4ab9-92a8-6fe7f54d62e2_2850x2777.png" width="1456" height="1419" data-attrs='//substack-post-media.s3.amazonaws.com/public/images/3dbac37d-1290-4ab9-92a8-6fe7f54d62e2_2850x2777.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1419,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Figure 37 — exhibition 2026 collage&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}' alt="Figure 37 — exhibition 2026 collage" title="Figure 37 — exhibition 2026 collage" srcset="https://substackcdn.com/image/fetch/$s_!X1PL!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3dbac37d-1290-4ab9-92a8-6fe7f54d62e2_2850x2777.png 424w, https://substackcdn.com/image/fetch/$s_!X1PL!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3dbac37d-1290-4ab9-92a8-6fe7f54d62e2_2850x2777.png 848w, https://substackcdn.com/image/fetch/$s_!X1PL!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3dbac37d-1290-4ab9-92a8-6fe7f54d62e2_2850x2777.png 1272w, https://substackcdn.com/image/fetch/$s_!X1PL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3dbac37d-1290-4ab9-92a8-6fe7f54d62e2_2850x2777.png 1456w" sizes="auto, 100vw" loading="lazy"><div></div></div></a><figcaption>Figure 37: The idiosyncratic 2026 exhibition entries rearward the two 2026 bars in Figure 35.</figcaption></figure></div><p>So, to recap, we discussed GPT-2 and the first transformer that add a stance vector to all token embedding before the archetypal transformer block. Then, contemporary LLMs switched to RoPE, which encodes related stance inner notice by rotating queries and keys. Surprisingly, RoPE remains as the most famous positional embedding method today.</p><p>Recently, however, Inkling picked up an idea from T5 to add a learned related stance bias to the notice scores before softmax. And afterward we additionally have NoPE-RoPE architectures, which omit RoPE in several layers.</p><p>It may be boring news, but as mentioned before, RoPE is motionless the frequent stance embedding method, and I think this volition transport complete into 2027 for sure. Perhaps, though, we&rsquo;ll see additional architectures alongside an Inkling-like scheme coming up in 2027.</p><p>While this may audio &ldquo;boring&rdquo;, I am glad that I eventually got this part on RoPE (and another stance info methods) out, following so many years, so that I can add it to my gathering of LLM &ldquo;fundamentals&rdquo; explainers.</p><p>If you are curious in the another architecture construction blocks of LLMs, I motionless extremely propose starting alongside a GPT-2-style model, which volition provision you a firm understanding of the construction blocks alongside the smallest amount of friction. And from there, you can afterward tier on additional methods (such as RoPE).</p><p><span>If you&rsquo;d akin to build that GPT-2-style example yourself, my archetypal </span><a href="https://en.mycoding.id/category/technology" target="_blank">Build a Large Language Model (From Scratch)</a><span> publish walks through all the construction blocks depicted in the fig below.</span></p><p>For the methods that branch off from the GPT-2 baseline, I&rsquo;ve collected several of my applicable blog articles below, starting alongside a refresher on self-attention:</p><ul><li><p><strong>Self-attention, causal masking, and multi-head attention:</strong><span> </span><a href="https://en.mycoding.id/category/technology" target="_blank">Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs</a></p></li><li><p><strong>Modern notice variants (GQA, MLA, sliding-window attention, sparse attention, and hybrids):</strong><span> </span><a href="https://en.mycoding.id/category/technology" target="_blank">A Visual Guide to Attention Variants in Modern LLMs</a></p></li><li><p><strong>The KV cache.</strong><span> </span><a href="https://en.mycoding.id/category/technology" target="_blank">Understanding and Coding the KV Cache in LLMs from Scratch</a></p></li><li><p><strong>SwiGLU feed-forward layers and normalization (LayerNorm and RMSNorm):</strong><span> </span><a href="https://en.mycoding.id/category/technology" target="_blank">From GPT-2 to gpt-oss: Analyzing the Architectural Advances</a></p></li><li><p><strong>Mixture-of-experts layers and expert routing.</strong><span> </span><a href="https://en.mycoding.id/category/technology" target="_blank">The Big LLM Architecture Comparison</a></p></li><li><p><strong>Residual connections and DeepSeek&rsquo;s manifold-constrained hyper-connections (mHC):</strong><span> </span><a href="https://en.mycoding.id/category/technology" target="_blank">Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention</a></p></li><li><p><strong>Putting several of the pieces together in Qwen3:</strong><span> </span><a href="https://en.mycoding.id/category/technology" target="_blank">Understanding and Implementing Qwen3 From Scratch</a><span> walks through implementing the compact and mixture-of-experts versions in PyTorch.</span></p></li></ul><p><strong>Thanks for study and supporting my work!</strong></p><p><span>PS: I started experimenting alongside Substack&rsquo;s new &ldquo;perks&rdquo; system. As a thank-you to supporters, I&rsquo;ve added </span><a href="https://en.mycoding.id/category/technology" target="_blank">image exports and metadata</a><span> from the architecture exhibition for you to use in your own analyses.</span></p></div></div></div></div></div></div></div><div><div id="discussion"><div><h4>Discussion concerning this post</h4></div></div></div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/understanding-positional-embeddings-in-modern-llms-77185</guid> <pubDate>Sun, 15 Nov 2026 13:14:06 +0700</pubDate> <author><![CDATA[support@mycoding.id (Doris Putra)]]></author> </item> <item> <title><![CDATA[Ukraine's war robots are starting to down Russia's drones. It could eventually change the fight in the kill zone.]]></title> <description><![CDATA[Ukraine's war robots are starting to down Russia's drones. It could eventually change the fight in the kill zone.. Ukraine's dirt robots are starting to downward Russian drones, a capability that could alter how soldiers move and combat on the forefront lines.]]></description> <media:thumbnail url="https://i.insider.com/6af4670bb2c29afb4e569ae8?width=1200&amp;format=jpeg"/> <category><![CDATA[Business]]></category> <link>https://en.mycoding.id/ukraine-s-war-robots-are-starting-to-down-russia-s-drones-it-could-eventually-change-the-fight-in-the-kill-zone-77177</link> <content:encoded><![CDATA[<div e-name="post-hero" data-component-type="post-hero" data-load-strategy="exclude"> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject"> <div> <p><img src="https://i.insider.com/6af466d0b2c29afb4e569ae0?width=2000&amp;format=jpeg&amp;auto=webp" srcset="https://i.insider.com/6af466d0b2c29afb4e569ae0?width=400&amp;format=jpeg&amp;auto=webp 400w, https://i.insider.com/6af466d0b2c29afb4e569ae0?width=500&amp;format=jpeg&amp;auto=webp 500w, https://i.insider.com/6af466d0b2c29afb4e569ae0?width=700&amp;format=jpeg&amp;auto=webp 700w, https://i.insider.com/6af466d0b2c29afb4e569ae0?width=1000&amp;format=jpeg&amp;auto=webp 1000w, https://i.insider.com/6af466d0b2c29afb4e569ae0?width=1300&amp;format=jpeg&amp;auto=webp 1300w, https://i.insider.com/6af466d0b2c29afb4e569ae0?width=2000&amp;format=jpeg&amp;auto=webp 2000w" sizes=" 1280px) 900px" alt="Tracked armed forces conveyance alongside mounted arm sits on a snowy site as dreary fume rises from a blaze rearward it." decoding="sync"> </p></div> <figcaption e-name="image-caption"> <span e-name="image-source" itemprop="creditText">STR/NurPhoto via Getty Images</span> <svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewbox="0 0 24 24"> <path fill="currentColor" fill-rule="evenodd" d="m4.56 18.5 7.486-7.72 7.394 7.626 2.56-2.64L12.046 5.5 2 15.86l2.56 2.64Z"></path> </svg> <span>Ukraine's war robot makers are increasingly looking at how the unmanned vehicles can halt drones.</span> </figcaption> </figure> </div> <div data-component-type="image-modal" data-load-strategy="lazy"> <p><a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">Ukraine's robots</a> are increasingly capable to downward airborne drones, a capability that could eventually assistance soldiers move and combat in several of the war's most hazardous areas.</p><p>The skies in Ukraine are complete of <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">surveillance and attack drones</a> that create movement risky, particularly in the <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">kill zone</a> surrounding the forefront lines.</p><p>Ukrainian units are increasingly <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">trying to use robots</a> there alternatively of humans, and companies are giving robots new abilities to obtain downward drones.</p><p>Oleksandr Yabchanka, caput of robotic systems for Ukraine's Da Vinci Wolves Battalion, a notable Ukrainian armed forces unit, told Business Insider that Russian drones are one of "the key challenges" on the battlefield, and "I accept that <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">ground-based robotic systems</a> can perform a important function in addressing this issue. "</p><p>With productive safety against those drones, he said, "the nature of combat operations alongside the row of communication could alter substantially."</p><p>He called defending the "low sky" by countering enemy drones a promising function for <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">ground robots</a>.</p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6af5aaa9b2c29afb4e56a19c&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:8134,&quot;aspectRatioH&quot;:5428}}' alt="A ample multirotor drone hovers low complete lofty turf below a cloudy sky."> </p></div> <span> <figcaption e-name="image-caption"> Ukraine's skies are so complete of drones that gaining a new way to combat them is a big advantage.&nbsp; <span e-name="image-source" itemprop="creditText"> Florent VERGNES / AFP via Getty Images </span> </figcaption> </span> </figure> <p>Beyond foremost front-line operators, Yabchanka additionally co-founded Defence Robotics UA, a non-profit that helps Ukrainian units clasp robotic technologies.</p><p>"I have particularly elevated hopes for the use of ground-based robotic systems in defending the low-altitude airspace," he said.</p><h2 id="a38afeef-4455-43fe-a9c8-5b81481eb42d" data-toc-id="a38afeef-4455-43fe-a9c8-5b81481eb42d">The slay zone</h2><p id="a38afeef-4455-43fe-a9c8-5b81481eb42d">Inside the slay zone, drones complicate movement, so robots are taking on <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">new roles</a> in logistics, casualty evacuation, and more. Some of these systems are armed alongside weaponry and jamming equipment to obtain downward antagonistic drones.</p> <p id="a38afeef-4455-43fe-a9c8-5b81481eb42d">Robots armed for counter-drone missions could decrease the petition for short-range air defence units, which provision shield for infantry moving in and out, in the slay zone. Right now, these <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">units visage awesome danger</a> and hazard getting pinned downward inside.</p><p id="a38afeef-4455-43fe-a9c8-5b81481eb42d">These robots could additionally defend themselves as Russia increasingly targets them.</p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6a045e0495396a3b6e417d2f&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:5000,&quot;aspectRatioH&quot;:3333}}' alt="Two green robots on wheels alongside guns mounted on them in a site of sunflowers under a blue sky"> </p></div> <span> <figcaption e-name="image-caption"> Ukraine's dirt robots are taking on additional battlefield roles, including in taking downward enemy drones.&nbsp; <span e-name="image-source" itemprop="creditText"> Diego Herrera Carcedo/Anadolu via Getty Images </span> </figcaption> </span> </figure> <p id="a38afeef-4455-43fe-a9c8-5b81481eb42d">Mykyta Rozhkov, chief endeavor betterment authoritative at Ukrainian robotics and innovation business Frontline Robotics, told Business Insider that robots are "becoming a chief target" for Russia and are defenseless without added protections.</p><p id="a38afeef-4455-43fe-a9c8-5b81481eb42d">Giving robots the capability to defend themselves could create them additional survivable, during turning them into mobile air defenses could assistance defend troops.</p><p id="a38afeef-4455-43fe-a9c8-5b81481eb42d">Rozhkov stated makers are now "trying to discover a way how we can defend against FPVs."</p><p>Yabchanka stated that whenever it comes to changing the slay zone, "it would be incorrect to commitment that this volition happen specifically inside the next 12 months."</p><p>But he stated robots have already successfully stopped several drones, and "I accept this site has a awesome future."</p><h2 id="da561fe7-581d-451e-987b-6a0f604137d1" data-toc-id="da561fe7-581d-451e-987b-6a0f604137d1">Robots vs drones</h2><p id="da561fe7-581d-451e-987b-6a0f604137d1">Most of the innovation in this area is motionless being tested or used on a constricted scale, chiefly to defend the robots themselves fairly than provision broader air defense.</p><p id="da561fe7-581d-451e-987b-6a0f604137d1">Rozhkov stated that during it's key to have uncrewed dirt vehicles be capable of defending themselves, he expects to eventually see UGVs functioning as a mobile, short-range air defence scheme to try to apparent the skies.</p><p id="da561fe7-581d-451e-987b-6a0f604137d1">His business is operating alongside a partner on a scheme that uses cameras and sensors to discover antagonistic drones and afterward engages them alongside a distinct capability. He didn't disclose how it would affect enemy drones.</p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6af5a99b0cc3ffba8779c485&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:3000,&quot;aspectRatioH&quot;:2000}}' alt="Soldier in camouflage alongside a Ukrainian emblem place stands between muddy trench walls under an overcast sky."> </p></div> <span> <figcaption e-name="image-caption"> Ukrainian soldiers' movement is constricted by Russian drones and artillery, and robots assisting capable to combat those drones could assistance in the war's most hazardous areas.&nbsp; <span e-name="image-source" itemprop="creditText"> ARIS MESSINIS/AFP via Getty Images </span> </figcaption> </span> </figure> <p>Other Ukrainian robot makers are additionally adding anti-drone capabilities, several involving smoke, nets, and shrapnel solutions. Robots are increasingly being fitted alongside weaponry akin device guns, and several companies have stated that these weapons can halt enemy drones.</p><p>Ukrainian business UFORCE, for instance, says it has destroyed enemy drones in trap scenarios. The company's AI-assisted Predator weapons position can be mounted on robots to track and blaze at drones. The 42nd Separate Mechanized Brigade earlier described a device gun mounted on a Liut robot shooting downward Russian drones.</p><p>Rovertech, which advertises itself as making "robots for the slay zone," is evolving a scheme that uses sensors, AI-assisted detection, and a net launcher to halt incoming drones, during Ratel Robotics has started serial manufacturing on a protective scheme that includes net launchers on its robots.</p><p>Ukraine's 100th Separate Mechanized Brigade stated final duration that it had deployed an energetic safety scheme using net launchers and shared a video showing one of its robots firing a net at an approaching drone, which afterward wrecked to the ground.</p><p>Some Ukrainian units have additionally driven robots carrying digital war equipment onward to jam Russia's drones, but it's an emerging capability that has yet to see general use.</p><p>Brave1, Ukraine's state-backed innovation platform, says armed robots are starting to be used against airborne drones, but it wants more, describing anti-aircraft weaponry and safety as priorities.</p><p>Rozhkov stated that soon, Ukraine's robots volition have "more and additional advanced solutions" to halt drones.</p><p>He stated it's essential since the archetypal flank that figures out how to<a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel=""> halt small FPVs</a> "will acquire a huge advantage on the battlefield and can create theatrical results."</p> </div> <h2>Read next</h2> <div data-recommendations-content data-track-click-shared='&quot;Read next&quot;}'> <div e-name="read-next-list" data-recommendations-list> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> </div> </div> <a e-name="read-next-image-link" data-recommendations-image-link data-recommendations-link tabindex="-1"> <p><img alt=""> </p> </a> <h3> <a e-name="read-next-title-link" data-recommendations-link data-recommendations-title></a> </h3> <div data-component-type="post-bottom-authors" data-load-strategy="exclude" data-track-view='&quot;module_in_view&quot;}' data-track-click-shared='&quot;navigation&quot;}'> <div data-track-view='&quot;module_in_view&quot;}'> <div> <div> <p><span>Sin&eacute;ad Baker is a Military and Defense Correspondent based in Business Insider's London bureau, penning concerning Russia's intrusion of Ukraine and NATO actions.</span><span>Sin&eacute;ad most frequently covers soldiers' experiences,&nbsp;<a href="https://en.mycoding.id/category/business" target="_blank">military strategy</a>, battlefield <a href="https://en.mycoding.id/category/business" target="_blank">developments</a>, weapons systems, the defence industry's response, and the effect on NATO and European security, and geopolitical <a href="https://en.mycoding.id/category/business" target="_blank">decisions</a> that encircle the Ukraine war.&nbsp;</span><span>She has&nbsp;reported from NATO&rsquo;s frontlines and about Europe,&nbsp; has <a href="https://en.mycoding.id/category/business" target="_blank">interviewed multiple premier ministers</a>&nbsp;and defence ministers, has appeared on BBC News and The Guardian's government podcast, and has been cited by Congressional hearings.</span><span>Sin&eacute;ad has additionally extensively covered US government and earlier led Business Insider's breaking news safety from London.</span><span>Sin&eacute;ad earlier completed a master's flat in investigative reporting at City, University of London, and has written for The Guardian, The Observer, and TheJournal.ie. Sin&eacute;ad is the erstwhile publishing company of the multi-award-winning The University Times in Dublin.</span><span><span>Expertise</span></span></p><ul><li><span>Experiences of soldiers in Ukraine, including battlefield developments and tactics</span></li><li><span>Western armed forces responses to the war, and lessons they should learn</span></li><li><span>New weaponry built for and in reply to the war&nbsp;</span></li></ul><p><span><em>Email Sin&eacute;ad at sbaker@businessinsider.com or discover her on </em><a href="https://en.mycoding.id/category/business" target="_blank"><em>X</em></a><em>.</em></span> </p></div> </div> </div> </div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/ukraine-s-war-robots-are-starting-to-down-russia-s-drones-it-could-eventually-change-the-fight-in-the-kill-zone-77177</guid> <pubDate>Sun, 15 Nov 2026 13:12:01 +0700</pubDate> <author><![CDATA[support@mycoding.id (Sinéad Baker)]]></author> </item> <item> <title><![CDATA[Apa yang Terjadi Saat Program Dikompilasi?]]></title> <description><![CDATA[Apa yang Terjadi Saat Program Dikompilasi?. Channel ini menyajikan materi seputar Data Science &amp; Artificial Intelligence, Networking &amp; Cyber Security, serta Database &amp; Programming. Seluruh mat...]]></description> <media:thumbnail url="https://i1.ytimg.com/vi/T2cRhYRx194/maxresdefault.jpg"/> <category><![CDATA[Indonesian]]></category> <link>https://en.mycoding.id/apa-yang-terjadi-saat-program-dikompilasi-77172</link> <content:encoded><![CDATA[<p class="embed-responsive embed-responsive-16by9 post-embed"><iframe width="560" height="315" aria-label="Preview" class="embed-responsive-item" src="https://www.youtube.com/embed/T2cRhYRx194?autoplay=1&amp;mute=1&amp;playsinline=1&amp;rel=0" title="YouTube video player" frameborder="0" allow="autoplay; accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe></p><h2>Apa yang Terjadi Saat Program Dikompilasi?</h2><p>Channel ini menyajikan materi seputar Data Science &amp; Artificial Intelligence, Networking &amp; Cyber Security, serta Database &amp; Programming. Seluruh materi disusun secara sistematis untuk membantu Anda memahami konsep, teknologi, dan praktik yang digunakan di bumi teknologi informasi, mulai dari tingkat alas hingga lanjutan. Setiap bahasan berfokus pada pengembangan keahlian melalui materi yang gampang dipahami, relevan dengan kebutuhan industri, dan disampaikan untuk tujuan edukasi. Seluruh konten mengedepankan implementasi teknologi secara legal, etis, dan bertanggung jawab, sehingga dapat menjadi rujukan belajar bagi pelajar, mahasiswa, maupun profesional. Jika Anda mau mengembangkan kapasitas di sektor pemrograman, pengelolaan data, kecerdasan buatan, jaringan komputer, maupun keamanan siber, channel ini menyediakan materi yang dapat membantu membangun fondasi ilmu sekaligus menaikkan keahlian secara bertahap. Playlist video terkait: ▪ https://www.youtube.com/playlist?list=PLZGgeqmsm32s ▪ https://www.youtube.com/playlist?list=PLZomBNpDl3vc ▪ https://www.youtube.com/playlist?list=PLE5VyXMZkzb8 PEMBAHASAN MATERI Playlist Materi Ilmu Komputer berisi kumpulan video pembelajaran, tutorial, klarifikasi konsep, studi kasus, demo, dan bahasan komplit seputar beragam sektor dalam ilmu komputer. Materi yang disajikan mencakup alas hingga tingkat lanjut, mulai dari pemrograman, pengelolaan database, jaringan komputer, keamanan siber, mekanisme operasi, pengembangan alat lunak, arsitektur komputer, hingga beragam teknologi yang digunakan dalam bumi komputasi modern. Temukan bahasan mengenai komputer dan pemrograman, algoritma, struktur data, PHP, JavaScript, Python, Laravel, MySQL, PostgreSQL, API, rekayasa alat lunak, database, jaringan komputer, pengelolaan server, cloud computing, virtualisasi, Linux, Windows Server, keamanan jaringan, cyber security, ethical hacking, kriptografi, keamanan aplikasi, DevOps, Internet of Things (IoT), kecerdasan buatan (AI), machine learning, serta beragam konsep, teknologi, dan praktik terbaik di sektor ilmu komputer. Playlist ini cocok bagi pelajar, mahasiswa, programmer, web developer, software engineer, pengelola sistem, ahli IT, maupun siapa saja yang mau mempelajari dan memperdalam wawasan di bumi ilmu komputer secara menyeluruh. CONTACT US ▪ Email support@mycoding.id ▪ WhatsApp https://wa.me/6281274309663 ▪ Telegram https://t.me/mycodingxd OFFICIAL LINKS ▪ Web https://mycoding.id ▪ Blog https://blog.mycoding.id ▪ News https://en.mycoding.id ▪ Apps https://play.google.com/store/apps/dev?id=5237501547073493079 SOCIAL MEDIA ▪ https://linkedin.com/company/mycodingxd ▪ https://facebook.com/mycodingxd ▪ https://instagram.com/mycodingxd ▪ https://tiktok.com/@mycodingxd ▪ https://twitter.com/mycoding_xd ▪ https://pinterest.com/mycoding_xd CHANNEL LINKS ▪ YouTube https://safelink.asia/youtube ▪ DailyMotion https://safelink.asia/dailymotion ▪ WhatsApp https://safelink.asia/whatsapp ▪ Telegram https://safelink.asia/telegram Powered by https://linkedin.com/company/401XDGroup #CyberSecurity #ArtificialIntelligence #Networking #DataScience #Programming #Database</p>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/apa-yang-terjadi-saat-program-dikompilasi-77172</guid> <pubDate>Sun, 15 Nov 2026 13:00:14 +0700</pubDate> <author><![CDATA[support@mycoding.id (Mc Project)]]></author> </item> <item> <title><![CDATA[Inside Sysco: How America's restaurants get ingredients]]></title> <description><![CDATA[Inside Sysco: How America's restaurants get ingredients. From enormous warehouses and high-tech allocation centers, we go rearward the scenes of Sysco.]]></description> <media:thumbnail url="https://i.insider.com/6af7b7ad0cc3ffba8779d210?width=1200&amp;format=jpeg"/> <category><![CDATA[Business]]></category> <link>https://en.mycoding.id/inside-sysco-how-america-s-restaurants-get-ingredients-77178</link> <content:encoded><![CDATA[<div> <div> <div id="player-container-mxPVKKUe" data-component-type="video" data-type="video" data-video-player data-load-strategy="disable-lazy" data-event-category="video_page" data-media-id="Nefiurrz" data-title=" How America&#039;s restaurants get ingredients" data-advertising="enabled" data-region="in-content" data-id="player-container-mxPVKKUe" data-enabled="true" data-events="enabled" data-preroll="4442842/businessinsider.desktop/video-post/pre-roll-in-content/retail" data-midroll="4442842/businessinsider.desktop/video-post/mid-roll-in-content/retail" data-sponsored="false" data-placement-id="mxPVKKUe" data-track-element="featured_video" data-thumbnail="https://i.insider.com/6af7db78b2c29afb4e56af89?width=850&amp;format=jpeg&amp;auto=webp" data-render-thumbnail="true"> <div> <p><img src="https://i.insider.com/6af7db78b2c29afb4e56af89?width=1300&amp;format=jpeg&amp;auto=webp" srcset="https://i.insider.com/6af7db78b2c29afb4e56af89?width=400&amp;format=jpeg&amp;auto=webp 400w, https://i.insider.com/6af7db78b2c29afb4e56af89?width=500&amp;format=jpeg&amp;auto=webp 500w, https://i.insider.com/6af7db78b2c29afb4e56af89?width=700&amp;format=jpeg&amp;auto=webp 700w, https://i.insider.com/6af7db78b2c29afb4e56af89?width=1000&amp;format=jpeg&amp;auto=webp 1000w, https://i.insider.com/6af7db78b2c29afb4e56af89?width=1300&amp;format=jpeg&amp;auto=webp 1300w" sizes=" 1280px) 900px" alt="A galley employee pours orange condiment complete prepared food in stainless-steel trays on a prep counter." decoding="sync"> </p> </div> </div> <div> <div> <div><p>Description </p><svg xmlns="http://www.w3.org/2000/svg" width="12" height="8" viewbox="0 0 12 8"> Angle downward icon An icon in the form of an viewpoint pointing down. <path d="M11.3281 0.792709C11.1874 0.796035 11.0537 0.854817 10.9562 0.95625L5.99996 5.9125L1.04371 0.95625C0.994011 0.90516 0.934574 0.864549 0.868913 0.836818C0.803251 0.809086 0.732698 0.794796 0.661421 0.794792C0.555323 0.79482 0.451644 0.826488 0.363639 0.885749C0.275634 0.94501 0.207301 1.02917 0.167378 1.12747C0.127454 1.22577 0.117753 1.33374 0.139515 1.43758C0.161277 1.54143 0.213513 1.63642 0.289546 1.71042L5.62288 7.04375C5.7229 7.14374 5.85854 7.1999 5.99996 7.1999C6.14139 7.1999 6.27702 7.14374 6.37705 7.04375L11.7104 1.71042C11.7878 1.63594 11.841 1.5398 11.8629 1.43459C11.8848 1.32939 11.8744 1.22003 11.833 1.12084C11.7917 1.02164 11.7214 0.937245 11.6313 0.878694C11.5412 0.820144 11.4355 0.790174 11.3281 0.792709V0.792709Z" fill="currentColor"></path> </svg> </div> </div> </div> <div> <div> <p>From enormous warehouses and high-tech allocation centers to caller create accommodation and eatery kitchens, we go rearward the scenes at Sysco to see how it delivers food, supplies, and ingredients to additional than 670,000 client locations worldwide. </p><p>As Sysco pursues its biggest acquisition ever, Jethro Restaurant Depot, several operators concern concerning what the agreement could average for competition, prices, and the disbursal of dining out.</p> </div> </div> </div> </div> <p>From enormous warehouses and high-tech allocation centers to caller create accommodation and eatery kitchens, we go rearward the scenes at Sysco to see how it delivers food, supplies, and ingredients to additional than 670,000 client locations worldwide. </p><p>As Sysco pursues its biggest acquisition ever, Jethro Restaurant Depot, several operators concern concerning what the agreement could average for competition, prices, and the disbursal of dining out.</p> <span>Show more</span>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/inside-sysco-how-america-s-restaurants-get-ingredients-77178</guid> <pubDate>Sun, 15 Nov 2026 13:00:01 +0700</pubDate> <author><![CDATA[support@mycoding.id (Jeremy Dreyfuss,Selima Hussain)]]></author> </item> <item> <title><![CDATA[Custom Facebook Feed Pro v4.16.0]]></title> <description><![CDATA[Custom Facebook Feed Pro v4.16.0. Custom Facebook Feed Pro v4.16.0 WordPress Plugin Free Download. Easy, Customizable, and Fast Facebook Feeds for WordPress Website. Smash Balloon Facebook Feed ...]]></description> <media:thumbnail url="https://en.mycoding.id/site/assets/img/og-image.png"/> <category><![CDATA[Digital Products]]></category> <link>https://en.mycoding.id/custom-facebook-feed-pro-v4-16-0-77180</link> <content:encoded><![CDATA[<div class="entry-content admission clearfix"><p class="wp-block-paragraph">Custom Facebook Feed Pro v4.16.0 WordPress Plugin Free Download. Easy, Customizable, and Fast Facebook Feeds for WordPress Website. Smash Balloon Facebook Feed is a WP Plugin that shows your Facebook Page’s Posts on your Website and Keeps them up to Date Automatically.</p><h3 class="wp-block-heading">Main Features</h3><ul class="wp-block-list"><li>Facebook Pages</li><li>100+ Options</li><li>Support Posts, Photos and Videos</li><li>SEO Optimized</li><li>Albums, Reviews and Events</li><li>GDPR Compliance</li><li>Speed Optimized</li><li>Health Alerts for your Feeds</li><li>Embed Anywhere</li><li>Multiple Facebook Feeds</li><li>Optimized Images</li><li>and more</li></ul><h2 class="wp-block-heading">Download Custom Facebook Feed WordPress Plugin</h2><b>Live Demo:</b> <a href="https://en.mycoding.id/category/digital-products" target="_blank" target="_blank"><b>View Demo</b></a><br><br><a href="https://en.mycoding.id/category/digital-products" target="_blank" target="_blank"><b>Facebook Feeds Online Documentation</b></a><br><br><div class="download-link-section"><div class="copybox"></div> <br><div class="copybox"></div> <br><div class="copybox"></div> <br><div class="copybox"></div> <br><div class="copybox"></div> <br><div class="copybox"></div> <br><div class="copybox"></div> <br><div class="copybox"></div></div><br><p class="wp-block-paragraph">Free download Custom Facebook Feeds Premium WordPress Plugin by Smash Balloon FB Feed plugin download</p><div class="stream-item stream-item-in-post stream-item-inline-post aligncenter"><div class="stream-item-size"><div id="container-0730062b1c60873c0fcc80d9fa01093b"></div></div></div><div class="stream-item stream-item-below-post-content"><div class="stream-item-size"><div id="container-0730062b1c60873c0fcc80d9fa01093b"></div></div></div></div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/custom-facebook-feed-pro-v4-16-0-77180</guid> <pubDate>Sun, 15 Nov 2026 12:50:00 +0700</pubDate> <author><![CDATA[support@mycoding.id (Null Php Script)]]></author> </item> <item> <title><![CDATA[Is Nvidia cooked without Jensen Huang?]]></title> <description><![CDATA[Is Nvidia cooked without Jensen Huang?. In this Sunday type of Business Insider Today, we're talking concerning what's next for Nvidia in a post-Huang world.]]></description> <media:thumbnail url="https://i.insider.com/6af780afcbe2120470735596?width=1200&amp;format=jpeg"/> <category><![CDATA[Business]]></category> <link>https://en.mycoding.id/is-nvidia-cooked-without-jensen-huang-77168</link> <content:encoded><![CDATA[<div e-name="post-hero" data-component-type="post-hero" data-load-strategy="exclude"> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject"> <div> <p><img src="https://i.insider.com/6af7809eb2c29afb4e56ad00?width=2000&amp;format=jpeg&amp;auto=webp" srcset="https://i.insider.com/6af7809eb2c29afb4e56ad00?width=400&amp;format=jpeg&amp;auto=webp 400w, https://i.insider.com/6af7809eb2c29afb4e56ad00?width=500&amp;format=jpeg&amp;auto=webp 500w, https://i.insider.com/6af7809eb2c29afb4e56ad00?width=700&amp;format=jpeg&amp;auto=webp 700w, https://i.insider.com/6af7809eb2c29afb4e56ad00?width=1000&amp;format=jpeg&amp;auto=webp 1000w, https://i.insider.com/6af7809eb2c29afb4e56ad00?width=1300&amp;format=jpeg&amp;auto=webp 1300w, https://i.insider.com/6af7809eb2c29afb4e56ad00?width=2000&amp;format=jpeg&amp;auto=webp 2000w" sizes=" 1280px) 900px" alt="Jensen Huang waving" decoding="sync"> </p></div> <span> <span e-name="image-source" itemprop="creditText">Mads Claus Rasmussen / Ritzau Scanpix / AFP via Getty Images</span> </span> </figure> </div> <div data-component-type="image-modal" data-load-strategy="lazy"> <h2 id="11850be2-84e0-4d2d-bc14-a7b71271d9fa" data-toc-id="11850be2-84e0-4d2d-bc14-a7b71271d9fa"><strong>A post-Huang era</strong></h2><p>We're concerning to study fair how much one individual really matters to the AI economy.</p><p>Jensen Huang was larger than existence at Nvidia. He was the AI boom's most recognizable evangelist, the part maker's chief visionary and dealmaker, and the power rearward Nvidia's ascend to the most precious business on the planet.</p><p>As <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">Zak Jason</a> writes, the cult of the tech originator has extremely much rotate into <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">America's religion</a>. And Huang was "its most convincing saint."</p><p>A week ago, Huang admitted to an matter and apologized to his family. Nvidia disclosed it was examining whether he was the mark of a abroad intellect effort. And afterward The New York Times reported a second affair, this one alongside an Nvidia administrator who was one of Huang's straightforward reports. He resigned on Thursday.</p><p>Huang's abrupt downfall is a emphasis test of originator worship. The market put a cost on it: Nvidia's market cap tumbled $573 milliard final week, additional than all but 18 companies in the complete S&amp;P 500.</p><p>"It's akin the Bulls without Jordan," author Stephen Witt, who wrote a existence former on Huang, stated on CNBC. "I don't cognize how they are going to substitute this guy."</p><p>That is <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">the regulation of key-man risk</a>, as my colleagues <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">Sarah E. Needleman</a> and <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">Tim Paradis</a> reported. When one chief is so closely identified alongside the strategy, the products, and the culture, their departure can create a vacuum that no org diagram (let solitary one alongside 60 straightforward reports to the CEO) can immediately fill.</p><p><strong>Much of Silicon Valley came to Huang's defense</strong> in latest days. Sarah Guo, a project capitalist, <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">called him</a> "one of the most bonzer entrepreneurs of all time." Sequoia's Shaun Maguire stated Huang's "contributions to American innovation and AI guidance cannot be overstated."</p><p><a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">A petition to reinstate Huang as Nvidia CEO</a> circulated complete the weekend, collecting additional than 100,000 signatures, including those of Elon Musk and another notable figures. Our own Alistair Barr additionally points to <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">new reports that are casting doubt on the intelligence</a> rearward Jensen Huang's downfall.</p> <p>For sure, there are multiple examples of apparently irreplaceable CEOs &mdash; Steve Jobs at Apple and <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">Travis Kalanick</a> at Uber &mdash; whose companies motionless thrived under new leadership.</p><p>And equal several of Huang's biggest proponents aren't position by him. "No one was a bigger guardian of Jensen than I was but there are rules concerning sex alongside subordinates," CNBC's Jim Cramer <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;outbound_click&quot;}' rel=" nofollow">said on X</a>. "Just since person is brilliant doesn't average he can do what he wants."</p><p>The inquiry is whether Nvidia, which lacked a defined succession plan, can <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">prove to be additional durable than its founder's legend</a>.</p><p>If it can't? Dick Costolo, the erstwhile Twitter CEO, <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;outbound_click&quot;}' rel=" nofollow">floated an idea</a>. "Prediction: 3-4 years from now, [Nvidia] volition young female a fourth or two, and they volition bring Jensen rear to run the company," he said.</p><p>"Like fame, everybody is outraged for 15 minutes, and there is nothing Americans affection additional than the comeback story."</p> </div> <div data-component-type="post-bottom-authors" data-load-strategy="exclude" data-track-view='&quot;module_in_view&quot;}' data-track-click-shared='&quot;navigation&quot;}'> <div data-track-view='&quot;module_in_view&quot;}'> <div> <p><span>Steve Russolillo is the Chief News Editor of Business Insider. He directs and drives the newsroom's safety of business, technology, markets, finance, and innovation, overseeing reporters and editors throughout the U.S., London, and Singapore.Steve&nbsp;writes the Sunday type of&nbsp;<a href="https://en.mycoding.id/category/business" target="_blank">BI Today</a>, Business Insider's flagship regular newsletter.</span><span>Before joining BI, he spent 17 years at The Wall Street Journal.&nbsp;He helped build and afterward led the Journal's breaking news desk&mdash;a squad of reporters, editors, and photo journalists tasked alongside dominating the most crucial stories of the day throughout safety areas. Before that, he was a abroad equal in Hong Kong.</span><span>Steve earned an MBA in backing from Baruch College and worked at Goldman Sachs in the firm's asset administration division. He went to undergrad at the University of Delaware.&nbsp;</span> </p> </div> </div> </div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/is-nvidia-cooked-without-jensen-huang-77168</guid> <pubDate>Sun, 15 Nov 2026 12:41:01 +0700</pubDate> <author><![CDATA[support@mycoding.id (Steve Russolillo)]]></author> </item> <item> <title><![CDATA[Lutnick Pushed to Weaken Chip Rules That Overlap with His Family Business]]></title> <description><![CDATA[Lutnick Pushed to Weaken Chip Rules That Overlap with His Family Business. Comments]]></description> <media:thumbnail url="https://static01.nyt.com/images/2026/11/14/multimedia/14dc-lutnick-commerce-01-bqtl/14dc-lutnick-commerce-01-bqtl-facebookJumbo.jpg"/> <category><![CDATA[Technology]]></category> <link>https://en.mycoding.id/lutnick-pushed-to-weaken-chip-rules-that-overlap-with-his-family-business-77186</link> <content:encoded><![CDATA[<div id="site-content" data-paywall-inert=""><div><div aria-live="polite"><p>You have a preview perspective of this part during we are checking your access. When we have confirmed access, the complete part satisfied volition load.</p></div><p id="article-summary">The commerce administrator has been penning new rules that would power Southeast Asian data centers, several of which are clients of his erstwhile firm.</p><div data-testid="imageblock-wrapper"><figure aria-label="media" role="group"><div data-testid="imageContainer-children-Image"><source media=" 288dpi)" srcset="https://static01.nyt.com/images/2026/11/14/multimedia/14dc-lutnick-commerce-01-bqtl/14dc-lutnick-commerce-01-bqtl-mobileMasterAt3x.jpg?quality=75&amp;auto=avif&amp;disable=upscale&amp;width=1800"></source><source media=" 192dpi)" srcset="https://static01.nyt.com/images/2026/11/14/multimedia/14dc-lutnick-commerce-01-bqtl/14dc-lutnick-commerce-01-bqtl-mobileMasterAt3x.jpg?quality=75&amp;auto=avif&amp;disable=upscale&amp;width=1200"></source><source media=" 96dpi)" srcset="https://static01.nyt.com/images/2026/11/14/multimedia/14dc-lutnick-commerce-01-bqtl/14dc-lutnick-commerce-01-bqtl-mobileMasterAt3x.jpg?quality=75&amp;auto=avif&amp;disable=upscale&amp;width=600"></source><img alt="" src="https://static01.nyt.com/images/2026/11/14/multimedia/14dc-lutnick-commerce-01-bqtl/14dc-lutnick-commerce-01-bqtl-superJumbo.jpg?quality=75&amp;auto=avif" srcset="https://static01.nyt.com/images/2026/11/14/multimedia/14dc-lutnick-commerce-01-bqtl/14dc-lutnick-commerce-01-bqtl-articleLarge.jpg?quality=75&amp;auto=avif 600w,https://static01.nyt.com/images/2026/11/14/multimedia/14dc-lutnick-commerce-01-bqtl/14dc-lutnick-commerce-01-bqtl-jumbo.jpg?quality=75&amp;auto=avif 684w,https://static01.nyt.com/images/2026/11/14/multimedia/14dc-lutnick-commerce-01-bqtl/14dc-lutnick-commerce-01-bqtl-superJumbo.jpg?quality=75&amp;auto=avif 1367w" sizes=" 1005px) 80vw, 100vw" decoding="async" width="600" height="899"></div><figcaption data-testid="photoviewer-children-ImageCaption"><span>Howard Lutnick has been sparring alongside others in the Trump administration complete restrictions that could clamp downward on the stream of Nvidia&rsquo;s A.I. chips to data centers that assist Chinese customers about the world.</span><span><span>Credit...</span><span><span aria-hidden="false">Tierney L. Cross/The New York Times</span></span></span></figcaption></figure></div><p>Nov. 15, 2026, <span>5:02 a.m. ET</span></p><div data-testid="companionColumn-0"><div><p>In Malaysia&rsquo;s lush southernmost peninsula, a data center known as A201 is set to open next year, packed alongside hundreds of millions of dollars in Nvidia&rsquo;s most advanced synthetic intellect servers.</p><p>The data center plans to provision this computing power to a Chinese tech giant. The setup is lawful and profitable. It volition additionally sidestep U.S. rules that ban revenue of the most advanced A.I. chips immediately to Chinese firms.</p><p>The business functioning the data center is a client of Cantor Fitzgerald, the financial services resolute formerly led by Howard Lutnick, the commerce secretary, who handed off guidance of it to his sons. Mr. Lutnick is now penning the rules that would have a significant consequence on the data center&rsquo;s business.</p><p>Mr. Lutnick has been seen as a power <a href="https://en.mycoding.id/category/technology" target="_blank" title="">slowing rules</a> that would clamp downward on the transaction of Nvidia chips to data centers akin A201, which sit exterior China but provision A.I. computing power to Chinese firms.</p></div></div><div data-testid="companionColumn-1"><div><p>The Trump administration, concerned concerning ceding the country&rsquo;s advantage in A.I., bans the transaction of cutting-edge chips to China. But it has been embroiled in an inner conversation for much of the former two years complete how far to go in limiting part revenue to data centers in Southeast Asia. On one flank of the conversation are China hawks, who see the data centers as a way for Beijing to do an end run about the part revenue ban. On the another are officials, most prominently Mr. Lutnick, who are pushing against going too far in limiting the capability of U.S. companies akin Nvidia to market their products in Asia.</p><p>Recently, Mr. Lutnick has been advocating a small stringent type of those controls under consideration, which could cut rear on Nvidia&rsquo;s endeavor by giving the U.S. authorities additional oversight of part revenue to global data centers. It is the latest example of Mr. Lutnick&rsquo;s gentle contact on part regulation, which has prompted criticism from lawmakers and others in the administration.</p><div><div><div id="optimistic-truncator-a11y" tabindex="-1"><hr><p>Thank you for your patience during we verify access. If you are in Reader manner delight exit and&nbsp;<a href="https://en.mycoding.id/category/technology" target="_blank">log into</a>&nbsp;your Times account, or&nbsp;<a href="https://en.mycoding.id/category/technology" target="_blank">subscribe</a>&nbsp;for all of The Times.</p><hr></div><div data-testid="optimistic-truncator-content"><svg width="24" height="24" viewbox="0 0 24 24" aria-hidden="true" data-tpl="sp" data-testid="optimistic-truncator-spinner"><path d="M11.28 22.8a.72.72 0 1 0 1.44 0v-6.72a.72.72 0 1 0-1.44 0v6.72Z" opacity=".37"></path><path d="M17.766 21.16a.72.72 0 1 0 1.165-.846l-3.95-5.436a.72.72 0 1 0-1.165.846l3.95 5.437Z" opacity=".26"></path><path d="M1.506 14.653a.72.72 0 0 0 .445 1.37l6.391-2.078a.72.72 0 1 0-.445-1.369l-6.39 2.077Z" opacity=".61"></path><path d="M5.07 20.314a.72.72 0 0 0 1.164.847l3.95-5.437a.72.72 0 1 0-1.165-.846l-3.95 5.436Z" opacity=".48"></path><path d="M22.049 16.022a.72.72 0 0 0 .445-1.37l-6.391-2.076a.72.72 0 1 0-.445 1.37l6.39 2.076Z" opacity=".17"></path><path d="M22.494 9.347a.72.72 0 1 0-.445-1.37l-6.391 2.078a.72.72 0 1 0 .445 1.369l6.39-2.077Z" opacity=".09"></path><path d="M6.234 2.84a.72.72 0 0 0-1.165.846l3.95 5.436a.72.72 0 0 0 1.165-.846l-3.95-5.437Z" opacity=".87"></path><path d="M1.951 7.978a.72.72 0 1 0-.445 1.37l6.391 2.076a.72.72 0 1 0 .445-1.37l-6.39-2.076Z" opacity=".74"></path><path d="M18.93 3.686a.72.72 0 0 0-1.164-.847l-3.95 5.437a.72.72 0 0 0 1.165.846l3.95-5.436Z" opacity=".02"></path><path d="M12.72 1.2a.72.72 0 1 0-1.44 0v6.72a.72.72 0 0 0 1.44 0V1.2Z"></path></svg><div data-testid="optimistic-truncator-message"><p data-tpl="t">Thank you for your patience during we verify access.</p><p data-tpl="t">Already a subscriber?&nbsp;<a href="https://en.mycoding.id/category/technology" target="_blank">Log in</a>.</p><p data-tpl="t">Want all of The Times?&nbsp;<a href="https://en.mycoding.id/category/technology" target="_blank">Subscribe</a>.</p></div></div></div></div></div></div></div></div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/lutnick-pushed-to-weaken-chip-rules-that-overlap-with-his-family-business-77186</guid> <pubDate>Sun, 15 Nov 2026 12:40:51 +0700</pubDate> <author><![CDATA[support@mycoding.id (Doris Putra)]]></author> </item> <item> <title><![CDATA[Same B2B Google Ads Keywords Cost About 21% More, Agency Data Shows]]></title> <description><![CDATA[Same B2B Google Ads Keywords Cost About 21% More, Agency Data Shows. One agency's data on 14 B2B advertisers finds the identical non-brand Google Ads keywords disbursal additional per click, equal following controlling for their own spending. The pos...]]></description> <media:thumbnail url="https://cdn.searchenginejournal.com/wp-content/uploads/2026/11/grey-matter-cpc-waterfall.png"/> <category><![CDATA[Business]]></category> <link>https://en.mycoding.id/same-b2b-google-ads-keywords-cost-about-21-more-agency-data-shows-77170</link> <content:encoded><![CDATA[<div class="sej-article-content gototop-pos "> <div class="sss2_sllo_o6"> </div> <div id="narrow-cont"><p>To decide how much additional the identical non-branded Google Ads keywords disbursal 14 B2B advertisers per click during the months of April through September compared to the identical period duration final year, Grey Matter reviewed their Google Ads Search campaigns.</p> <p>According to the <a href="https://en.mycoding.id/category/business" target="_blank" target="_blank" rel="noopener">agency study by Grey Matter</a> issued in October, there was an addition of concerning 21% in cost-per-click (CPC) for the identical keywords used by these 14 B2B advertisers, following accounting for changes in their spending, ad stance and Quality Score. The study gives a apt range of 1% to 44% for that figure. It is according to client Google Ads Search run data only, so it does not contain Performance Max, Shopping or Display campaigns. All 14 arrive from the agency’s client base, and its hunt squad runs SEO and paid hunt programs for B2B companies.</p><div data-total="8" id="SEJ_300x250_UnderPost_1-parent" class="content-unit sss2_sllo_o6 mmh-90-wrap"> <div class="mmh-90-inner"> <div class="mmh-90 flex-aligner"> <div data-adnumber="0" data-middle="no" data-p-slot="SEJ_300x250_UnderPost" data-slot="SEJ_300x250_UnderPost/SEJ_300x250_UnderPost_1" id="SEJ_300x250_UnderPost_1" class="sadslots"> </div> </div> </div> </div> <p>Andy Hendricks, Grey Matter’s caput of search, writes that he wanted to isolate what part of the general cost alter all advertisers informed versus what part was because of actions taken by all advertiser individually. Most accessible CPC benchmarks, specified as published averages, do not document for this differentiation and merely provision an general average CPC.</p> <h2>How The Report Measured The Same Click</h2> <p>Grey Matter excluded broad equivalent keywords since the searches they get matched to can alter complete time; keywords inside Maximize Clicks campaigns additionally do not fit into the analysis. Brand and competitor keywords were additionally excluded from the general estimate.</p> <p>Grey Matter’s inspection is according to tracking 1,349 non-brand keywords, all inside one account, for six quarters (April 2025 to September 2026). The year-over-year difference covers April through September 2026 versus those months a twelvemonth prior.</p><div id="summita-wrapa_112" class="middle_p_txt"></div> <p>The header evaluation was derived from a regression example using data from all 14 advertisers in the study which controlled for run spend, belief portion misplaced to budget, top-of-page position, and Quality Score. Every advertiser was stated equal importance despite of how much it spends.</p> <p>With no controls applied, the identical keywords disbursal 29% additional twelvemonth complete year. Adding the two spend-related controls lowered that to 22%, and stance and Quality Score took it to 21%.</p> <p>A second evaluation method used in this inspection was a keyword cost index. This measure covered 11 advertisers who had adequate matching keywords from fourth to quarter.</p><div data-total="8" id="SEJ_300x250_UnderPost_3-parent" class="content-unit sss2_sllo_o6 mmh-90-wrap"> <div class="mmh-90-inner"> <div class="mmh-90 flex-aligner"> <div data-adnumber="2" data-middle="no" data-p-slot="SEJ_300x250_UnderPost" data-slot="SEJ_300x250_UnderPost/SEJ_300x250_UnderPost_3" id="SEJ_300x250_UnderPost_3" class="sadslots"> </div> </div> </div> </div> <h2>Higher Spend Went With Higher CPCs On The Same Keywords</h2> <p>The report’s regression ties all 10% addition in run expend to concerning 1.4% higher CPCs, on top of the market increase, or approximately 11% for doubled spend. The study doesn’t provision a apt range for that figure.</p> <p>The most extreme case in the example nearly quadrupled its non-brand Search expend twelvemonth complete year, and its same-keyword CPCs rosy 132%. The study says that advertiser’s brand keywords, on a flat budget, rosy concerning 4% twelvemonth complete twelvemonth in the third quarter.</p> <p>The study sees a association between this form and automated bidding. It explains that a run using a scheme akin Maximize Conversions spends a bigger prosperity by entering auctions it earlier skipped, including higher positions, additional rivalrous searches, and pricier times of day.</p><div data-total="8" id="SEJ_300x250_UnderPost_4-parent" class="content-unit sss2_sllo_o6 mmh-90-wrap"> <div class="mmh-90-inner"> <div class="mmh-90 flex-aligner"> <div data-adnumber="3" data-middle="no" data-p-slot="SEJ_300x250_UnderPost" data-slot="SEJ_300x250_UnderPost/SEJ_300x250_UnderPost_4" id="SEJ_300x250_UnderPost_4" class="sadslots"> </div> </div> </div> </div> <h2>What The 11 Advertisers Paid Overall</h2> <p>For the 11 advertisers covered by the keyword cost index, the general year-over-year addition in average non-brand CPC was 36%, including the advertisers’ own choices. Of that, 22 points were attributed to the market, and three additional points were attributed to risen spending. There was a three-point conclusion because of assorted types of keyword, equivalent category and bid scheme changes, leaving 14 points that the report’s archetypal three parts don’t explain.</p> <p>Grey Matter illustrates that breakdown in the diagram below.</p> <figure class="wp-caption alignnone"><a href="https://en.mycoding.id/category/business" target="_blank" target="_blank" rel="noopener"><img decoding="async" class="wp-image-597429 size-full" src="https://cdn.searchenginejournal.com/wp-content/uploads/2026/11/grey-matter-cpc-waterfall.png" alt=" market +22 percent points, prosperity +3, keyword changes -3, and another unexplained changes +14." width="1620" height="900" srcset="https://cdn.searchenginejournal.com/wp-content/uploads/2026/11/grey-matter-cpc-waterfall-384x213.png 384w, https://cdn.searchenginejournal.com/wp-content/uploads/2026/11/grey-matter-cpc-waterfall-425x236.png 425w, https://cdn.searchenginejournal.com/wp-content/uploads/2026/11/grey-matter-cpc-waterfall-480x267.png 480w, https://cdn.searchenginejournal.com/wp-content/uploads/2026/11/grey-matter-cpc-waterfall-680x378.png 680w, https://cdn.searchenginejournal.com/wp-content/uploads/2026/11/grey-matter-cpc-waterfall-768x427.png 768w, https://cdn.searchenginejournal.com/wp-content/uploads/2026/11/grey-matter-cpc-waterfall-850x472.png 850w, https://cdn.searchenginejournal.com/wp-content/uploads/2026/11/grey-matter-cpc-waterfall-1024x569.png 1024w, https://cdn.searchenginejournal.com/wp-content/uploads/2026/11/grey-matter-cpc-waterfall-1536x853.png 1536w, https://cdn.searchenginejournal.com/wp-content/uploads/2026/11/grey-matter-cpc-waterfall-1600x889.png 1600w, https://cdn.searchenginejournal.com/wp-content/uploads/2026/11/grey-matter-cpc-waterfall.png 1620w" sizes=" 1620px) 100vw, 1620px" loading="lazy"></a><figcaption class="wp-caption-text">Source: <a href="https://en.mycoding.id/category/business" target="_blank" target="_blank" rel="noopener">Grey Matter, Same Click, Higher Price</a>, October 2026 (cropped).</figcaption></figure> <p>The bulk of those 14 points occurred at a few of the advertisers who saw an approximate doubling of their CPCs, which the study stated likely represented competition in their corresponding categories.</p><div data-total="8" id="SEJ_300x250_UnderPost_5-parent" class="content-unit sss2_sllo_o6 mmh-90-wrap"> <div class="mmh-90-inner"> <div class="mmh-90 flex-aligner"> <div data-adnumber="4" data-middle="no" data-p-slot="SEJ_300x250_UnderPost" data-slot="SEJ_300x250_UnderPost/SEJ_300x250_UnderPost_5" id="SEJ_300x250_UnderPost_5" class="sadslots"> </div> </div> </div> </div> <p>Actual CPC changes ranged from a 30% decrease to a 121% increase. Ten of the 11 paid additional for the identical keywords, and nine paid additional overall. One advertiser kept its spending flat and saw the identical keywords increase by 20%. After shifting several of its prosperity to pricier phrase-match keywords, its general CPC risen by 40%.</p> <h2>How Firm The 21% Is</h2> <p>With lone 14 advertisers in this report, the apt range for the 21% evaluation is wide. When the biggest spender is removed from the report’s data, the evaluation drops to about 17%, and its apt range of -2% to +40% includes zero. When broad equivalent keywords are included, the evaluation rises to about 31%. According to Hendricks, compared alongside a twelvemonth ago, that identical click most apt expenses concerning 20% additional currently, and the true fig “could be meaningfully higher or lower”.</p> <p>The study calls its findings association (and not proven causation). There are no references on the study leaf or in the PDF to the industries or markets of the advertisers used in the report, nor their currency.</p> <p>The study doesn’t provision any conversion charge or disbursal per guide information; so, it can’t display whether these advertisers saw improved or worse results alongside alongside higher CPCs. Jyll Saskin Gales asserted in an <a href="https://en.mycoding.id/category/business" target="_blank">April pillar for Search Engine Journal</a> (SEJ) that under Smart Bidding, a elevated CPC is frequently a sign of a fit account, and that accounts have to be evaluated according to CPA or ROAS.</p> <h2>Google’s Own CPC Figure Measures Something Else</h2> <p>Alphabet’s <a href="https://en.mycoding.id/category/business" target="_blank" target="_blank" rel="noopener">third-quarter 10-Q</a> reports that cost-per-click on Google Search &amp; another properties rosy 4% twelvemonth complete twelvemonth in the quarter. Alphabet has defined this metric as click-driven income divided by total paid clicks and has cited multiple factors specified as changes in the devices users admission the net on, geography, advertiser spending, and merchandise mix as variables that power it.</p> <p>The metric is according to paid clicks through Google Search properties and another platforms owned by Alphabet, so it volition track alongside the cost of the clicks and additionally the mix of clicks being counted. Grey Matter’s evaluation alternatively tracks the identical keywords throughout 14 advertisers.</p> <p>There is no straightforward confirmation or contradiction between the two figures since they measure entirely distinct things.</p> <h2>Why This Matters</h2> <p>Grey Matter’s study holds the keyword and the document constant, during raw CPC averages additionally increase whenever advertisers buy distinct clicks. With 14 clients of one agency and a range from 1% to 44%, the study is a citation item for B2B non-brand Search, not a forecast for any sole account.</p> <p>You can run a same-keyword inspect on your own document by comparing the year-over-year alter in what the identical non-brand keywords disbursal alongside the alter in the account’s average CPC. The study says the gap between those two as the consequence of keyword, equivalent category and bid scheme changes. A same-keyword increase can motionless contain the consequence of higher spending.</p> <h2>Looking Ahead</h2> <p>Grey Matter’s data runs through September, so it doesn’t contain fourth-quarter auctions. For 2027 budgets, the report’s direction is to measure them gradually and measure all stage by how much the additional clicks cost. For campaigns on automated bidding specified as Maximize Conversions, the study additionally suggests deciding what the additional conversions are value before raising the budget, and environment a mark to match.</p> <p><em>Featured Image: Tetiana Yurchenko/Shutterstock</em></p> </div> <div id="bcats" class="bottom-cat narrow-cont"> <span class="sej-category boxed cat-label">Category</span> <a href="https://en.mycoding.id/category/business" target="_blank">News</a> <a href="https://en.mycoding.id/category/business" target="_blank">Paid Media</a> </div> <div class="sej-above-author-widget"> <div class="sej-widget clearfix"> <div class="sej-badge-box"> <a href="https://en.mycoding.id/category/business" target="_blank" data-placement="Under Article" aria-label="Add Search Engine Journal as a preferred origin on Google"> <svg class="sej-gps-g" viewbox="0 0 48 48" aria-hidden="true" focusable="false"> <path fill="#EA4335" d="M24 9.5c3.54 0 6.71 1.22 9.21 3.6l6.85-6.85C35.9 2.38 30.47 0 24 0 14.62 0 6.51 5.38 2.56 13.22l7.98 6.19C12.43 13.72 17.74 9.5 24 9.5z"></path> <path fill="#4285F4" d="M46.98 24.55c0-1.57-.15-3.09-.38-4.55H24v9.02h12.94c-.58 2.96-2.26 5.48-4.78 7.18l7.73 6c4.51-4.18 7.09-10.36 7.09-17.65z"></path> <path fill="#FBBC05" d="M10.53 28.59c-.48-1.45-.76-2.99-.76-4.59s.27-3.14.76-4.59l-7.98-6.19C.92 16.46 0 20.12 0 24c0 3.88.92 7.54 2.56 10.78l7.97-6.19z"></path> <path fill="#34A853" d="M24 48c6.48 0 11.93-2.13 15.89-5.81l-7.73-6c-2.15 1.45-4.92 2.3-8.16 2.3-6.26 0-11.57-4.22-13.47-9.91l-7.98 6.19C6.51 42.62 14.62 48 24 48z"></path> </svg> <span class="sej-gps-text">Add <b>SEJ</b> as a preferred origin on Google</span> </a> </div> </div> </div> </div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/same-b2b-google-ads-keywords-cost-about-21-more-agency-data-shows-77170</guid> <pubDate>Sun, 15 Nov 2026 12:30:33 +0700</pubDate> <author><![CDATA[support@mycoding.id (Matt G. Southern)]]></author> </item> <item> <title><![CDATA[Youtuber MKBHD on the Cybertruck's discontinuation: 'Doesn't bode well for the Roadster']]></title> <description><![CDATA[Youtuber MKBHD on the Cybertruck's discontinuation: 'Doesn't bode well for the Roadster'. The famous YouTuber says the Cybertruck coming end isn't fine news for the another long-awaited Tesla supercar. And, he thinks his tress is safe.]]></description> <media:thumbnail url="https://i.insider.com/6af78c76b2c29afb4e56ad91?width=1200&amp;format=jpeg"/> <category><![CDATA[Business]]></category> <link>https://en.mycoding.id/youtuber-mkbhd-on-the-cybertruck-s-discontinuation-doesn-t-bode-well-for-the-roadster-77169</link> <content:encoded><![CDATA[<div e-name="post-hero" data-component-type="post-hero" data-load-strategy="exclude"> <figure data-type="img" e-name="image-figure-image" data-media-container="image" itemscope itemtype="https://schema.org/ImageObject"> <div> <p><img src="https://i.insider.com/6af78534cbe21204707355cf?width=2000&amp;format=jpeg&amp;auto=webp" srcset="https://i.insider.com/6af78534cbe21204707355cf?width=400&amp;format=jpeg&amp;auto=webp 400w, https://i.insider.com/6af78534cbe21204707355cf?width=500&amp;format=jpeg&amp;auto=webp 500w, https://i.insider.com/6af78534cbe21204707355cf?width=700&amp;format=jpeg&amp;auto=webp 700w, https://i.insider.com/6af78534cbe21204707355cf?width=1000&amp;format=jpeg&amp;auto=webp 1000w, https://i.insider.com/6af78534cbe21204707355cf?width=1300&amp;format=jpeg&amp;auto=webp 1300w, https://i.insider.com/6af78534cbe21204707355cf?width=2000&amp;format=jpeg&amp;auto=webp 2000w" sizes=" 1280px) 900px" alt=' Strategies and Insights alongside MKBHD&quot; panel during INBOUND 2025 Powered by HubSpot at Moscone Center on September 03, 2025, in San Francisco.' decoding="sync"> </p></div> <figcaption e-name="image-caption"> <span e-name="image-source" itemprop="creditText">Chance Yeh/Getty Images for HubSpot</span> <svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewbox="0 0 24 24"> <path fill="currentColor" fill-rule="evenodd" d="m4.56 18.5 7.486-7.72 7.394 7.626 2.56-2.64L12.046 5.5 2 15.86l2.56 2.64Z"></path> </svg> <span>MKBHD additionally stated his tress is likely harmless following placing a bet connected to the Cybercab.</span> </figcaption> </figure> </div> <div data-component-type="image-modal" data-load-strategy="lazy"> <p>Tesla's decision to <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">wind downward Cybertruck production</a> has Marques Brownlee concerned concerning a distinct delayed vehicle.</p><p>"This doesn't bode fine for the Roadster," Brownlee, the YouTube reviewer popularly known as MKBHD, stated on a Friday event of the "WVFRM Podcast."</p><p>On November 6, Tesla CEO Elon Musk announced on X that the EV creator would <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">stop construction the Cybertruck</a> by mid-2027 to concentration on manufacturing of the steering wheel-less Cybercab. Earlier this year, Tesla additionally former its Model S and Model X, as it shifts additional of its notice toward autonomous vehicles and humanoid robots.</p><p>That makes the Roadster, Tesla's twice-revealed and long-delayed supercar, a questionable fit in the automaker's lineup, MKBHD said. It's the lone conveyance in Tesla's lineup that the automaker isn't trying to create completely autonomous.</p> <figure data-type="img" e-name="image-figure-image" data-media-container="image" role="button" tabindex="0" aria-label="Click to open depiction in exhibition view" itemscope="" itemtype="https://schema.org/ImageObject"> <div> <p><img src="image/svg+xml,%3C%3Fxml%20version=&#039;1.0&#039;%20encoding=&#039;UTF-8&#039;%3F%3E%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20width=&#039;1&#039;%20height=&#039;1&#039;/%3E" data-content-type="image/jpeg" data-srcs='//i.insider.com/6af79d83b2c29afb4e56ae53&quot;:{&quot;contentType&quot;:&quot;image/jpeg&quot;,&quot;aspectRatioW&quot;:4564,&quot;aspectRatioH&quot;:3043}}' alt="The Tesla Roadster"> </p></div> <span> <figcaption e-name="image-caption"> People appearance at a prototype of the 2028 Tesla Roadster Thrust, an all-electric conveyance featuring SpaceX-developed thrusters.&nbsp; <span e-name="image-source" itemprop="creditText"> Caroline Brehman / AFP via Getty Images </span> </figcaption> </span> </figure> <p>MKBHD has followed the Roadster saga particularly closely. He earlier held two reservations for the high-flying vehicle, but he stated in September 2025 <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">that he had canceled one</a> following waiting nearly a decade for delivery.</p><p>And, for the Cybertruck, he stated he thinks Tesla missed a aureate opportunity.</p><p>"If they fair made akin a good, uncomplicated SUV type-thing, <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">like the EV9</a>," he said. "That would have sold a lot. People would have wanted that everywhere."</p><h2 id="6fc02b66-fed3-47a7-b5a7-4ce468d9af8d" data-toc-id="6fc02b66-fed3-47a7-b5a7-4ce468d9af8d">About MKBHD's hair&hellip;</h2><p>The YouTuber has a big bet alongside Tesla stans: If Tesla, as it formerly predicted, sells Cybercabs to US consumers by the end of 2026, he'll shave his head.</p><p>Tesla debuted its gold-painted <a href="https://en.mycoding.id/category/business" target="_blank" data-track-click='&quot;bi_value_unassigned&quot;}' rel="">Cybercabs in Austin in September</a>, but the vehicles were used exclusively in Tesla's own ride-hailing assistance &mdash; not sold to idiosyncratic drivers.</p> <p>Now, Tesla's scheme to continue making Cybertrucks until mid-2027 is making MKBHD equal additional assured in his bet.</p><p>"Alright, I'm likely good," he said. "I did fair get a haircut."</p> </div> <h2>Read next</h2> <div data-recommendations-content data-track-click-shared='&quot;Read next&quot;}'> <div e-name="read-next-list" data-recommendations-list> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> <span> <p><img src="image/svg+xml,%3Csvg%20xmlns=&#039;http://www.w3.org/2000/svg&#039;%20viewBox=&#039;0%200%201%201&#039;%3E%3C/svg%3E" data-src="/public/assets/logos/placeholder.png" alt=""> </p> </span> <h3> <span>Business Insider tells the innovative stories you desire to know</span> </h3> </div> </div> <a e-name="read-next-image-link" data-recommendations-image-link data-recommendations-link tabindex="-1"> <p><img alt=""> </p> </a> <h3> <a e-name="read-next-title-link" data-recommendations-link data-recommendations-title></a> </h3> <div data-component-type="post-bottom-authors" data-load-strategy="exclude" data-track-view='&quot;module_in_view&quot;}' data-track-click-shared='&quot;navigation&quot;}'> <div data-track-view='&quot;module_in_view&quot;}'> <div> <p><span>Ben Shimkus is a newsman for the Business News desk. He writes concerning cars, transportation, retail, and jobs.&nbsp;Ben's reporting has appeared in Rolling Stone, The Verge, Automotive News, USA Today, AutoBody News, LGBTQ Nation, TopSpeed, and Out Magazine. He's additionally held personnel penning positions at The U.S. Sun and the Daily Mail. He graduated from NYU alongside a Master's in reporting in 2024.&nbsp;Email Ben at bshimkus@insider.com or communication him privately&nbsp;on Signal at bshimkus.41.&nbsp;</span> </p> </div> </div> </div>]]></content:encoded> <guid isPermaLink="true">https://en.mycoding.id/youtuber-mkbhd-on-the-cybertruck-s-discontinuation-doesn-t-bode-well-for-the-roadster-77169</guid> <pubDate>Sun, 15 Nov 2026 12:00:01 +0700</pubDate> <author><![CDATA[support@mycoding.id (Ben Shimkus)]]></author> </item> </channel> </rss>