Hacking the Xbox 360 Hypervisor Part 3: Bad Update Improvements

Welcome to part 3 of the Hacking the Xbox 360 Hypervisor blog series. In this part I’ll go complete what happened following I released the PoC for Bad Update, and how it went from taking ~20 minutes alongside a ~30% achievement charge to being near instantaneous alongside a near 100% achievement rate. If you haven’t already I extremely propose you peruse (or at smallest skim through) part 1 and part 2 as this article volition citation a lot of the matter discussed there. This part volition be extremely abbreviated but volition capture you up for part 4 which covers the equal force utilize and volition be out soon.

Rock Band Blitz

In the months following the publish of Bad Update v1.0 there were two important improvements made to it by others. The archetypal by “ihatecompvir” and “InvoxiPlayGames” is a new equivalent preserve admission item in the demo type of Rock Band Blitz (RBB) called DaTArrest. The demo type of RBB is a bundle document that’s normally downloaded from the Xbox Live shop for liberated and saved to your console’s HDD or recollection unit. When the Xbox 360 got assistance for using USB sticks as retention devices all of the packages signed to footwear from the HDD or recollection component were grandfathered in to additionally footwear from USB retention devices. This method the RBB demo bundle can be copied to a USB rod as a normal document and launched on any console on a application type that supports it.

By exploiting several recollection misconduct vulnerabilities in the game’s DLC parsing code they were capable to accomplishment a equivalent preserve document that whenever loaded would bootstrap the Bad Update phase 2 ROP sequence and commencement the hypervisor utilize chain. Using RBB as an admission item we no longer needed the Tony Hawk American Wasteland equivalent disc. Running the Bad Update utilize was now as uncomplicated as copying several records to a USB stick, plugging it into your console, and launching Rock Band Blitz from the dashboard.

Hypervisor Race Condition Optimizations

Shortly following the RBB admission item was discovered I received a drag petition from “Máté” who had made several improvements to the hypervisor competition circumstance part of the exploit. These improvements included a hash array for precomputing and storing the malicious ciphertext values, a improved cache flushing primitive, and improved thread scheduling between the XKE payload thread and ciphertext overwrite thread. The blend of these improvements reduced implementation period downward to a few seconds and the competition circumstance had a much higher achievement rate.

Ciphertext Hash Table

In part 2 I talked concerning attacking encrypted memory by using hypervisor APIs to create encrypted recollection allocations at predetermined recollection addresses and used two distinct recollection pathways to grasp the ciphertext for several chosen plaintext data. Using this primitive I was capable to accomplishment ciphertext data to get kernel manner code execution as fine as the ciphertext data used to attack the LZX decompression code in the bootloader update XKE payload.

The whitening value used in the recollection encryption scheme is 12 bits ample which gives 1024 distinctive ciphertext combinations for several stated plaintext data. However, in my first PoC I was lone generating the ciphertext for among the 1024 imaginable whitening values and the logic for that was a two part nonaccomplishment on my end. When I initially prototyped out the ciphertext assault I did really try using a lookup array for ciphertext combinations but I wasn’t capable to get it working.

The archetypal matter was I was using a extremely old ~2011 era hypervisor/kernel build (it was fair what all my old investigation was based on) which was doing a TLB flush for the encrypted VA range in the hypervisor function that setup the recollection encryption. In afterward hypervisor builds that TLB flush was removed since the kernel wrap function for this hypervisor function already does a TLB flush for the VA range. However, I call the hypervisor syscall immediately (skipping the kernel wrap code which does the TLB flush) and since the TLB is lazy-loaded the whitening bits would rotate into old and the resulting ciphertext peruse out of recollection was invalid.

When evaluation the utilize on the 17559 (latest) type hypervisor the lookup array was causing issues and fairly than examine I decided I’d fix it afterward and moved on. By the period I got to the end of the utilize I had rotate into so blinded by tunnel imagination chasing CPU cache issues that I had convinced myself the lookup array fair wasn’t going to work. What I should have likely done (aside from examine the lookup array issues whenever they surfaced) was to put this project downward for a few weeks and arrive rear alongside a apparent mindset. Oh well, instruction learned…

Getting into the implementation particulars of the hash table, we use the archetypal N bits of the ciphertext we anticipate to see in recollection as an indicator into two arrays. The archetypal gathering contains the archetypal 64-bits of the oracle ciphertext which is used for validation in case there’s always a collision in the hash table. The second gathering stores our malicious ciphertext we desire to compose into memory.

In command to create this as productive as imaginable (remember we’re racing the CPU and cache here) we disregard collisions. It’s improved to have a few collisions and have to skip several iterations on the competition circumstance iteration than to execute assistance for buckets in the hash table. Instead we compensate for collisions by choosing a sufficiently ample adequate figure of bits for the hash array key specified that the figure of collisions is on the command of sole digits, if we equal have any (majority of the period there’s 0).

However, additional bits in the hash array key method a ample gathering to clasp the entries (meaning additional recollection is used) which could possibly activity against us as we’ll be performing additional fetches from RAM since CPU cache is being thrashed. Máté’s first PR used a key size of 10 bits but I afterward upped this to 20 bits which seems to be a blessed place anywhere there’s normally no collisions or if there are it’s a low sole numeral number. The total size of recollection needed to clasp the two hash tables is concerning 24MB.

Looking at the snippet complete we can see how the hash array is built. We iteration 1024 times to exhaust all imaginable whitening values for this leaf of memory. Each repeat we encrypt the leaf (which volition increment the whitening value), get the ciphertext for the LZX decoder data we anticipate to be in recollection and the ciphertext for our malicious data, afterward add them to the hash tables. On lines 15-16 we can see the HvxFlushSingleTb call that thwarted my own attempt at using a hash table, and on lines 30-31 we can see how the archetypal N bits of the expected ciphertext is used as the hash array key.

Improved Cache Flushing

In my first PoC I used a distinctive characteristic of the Xbox 360 CPU to lock distant fractional of the accessible L2 cache to try and put as much force on it as I could. The L2 locking method was pivotal in the first PoC but it didn’t put as much force on CPU cache as I would have liked. Putting force on CPU cache is particularly annoying since kernel manner (where the utilize code is running) cannot use certain cache flushing instructions because of them being privileged and lone usable in hypervisor mode.

Additionally, the hypervisor lone runs alongside CPU caching enabled whenever functioning on encrypted recollection which makes finding a hypervisor flat cache flush primitive difficult. I tried to activity about this by finding a hypervisor syscall that would peruse a ample amount of encrypted memory, and I did discover one syscall that fit the description, but it wasn’t nearly adequate for what I needed. That function additionally had locks in it which prevented me from executing it on multiple threads simultaneously.

However, there was a kernel manner cache education I entirely overlooked that Máté had established to activity extremely well, dcbz or data cache obstacle set to zero. Aside from fair zero’ing out a CPU cache row this education has an engaging flank effect:

If the cache obstacle is already in the data cache, all of its bytes are cleared to zero. If it is not in the cache, the obstacle is allocated and initialized immediately in the cache to zero without study the old contents from chief memory.

So if this education is used on a recollection location that’s not currently in CPU cache the CPU volition allocate area for it and possibly evict item alternatively to create area for it. By operating the dcbz education complete ample regions of recollection it can be used as a cache flush primitive and power old data in CPU cache to be evicted. In Máté’s first implementation he allocated an 0x80000 byte area of recollection afterward looped complete it and for all 0x80 bytes (the size of a cache line) called dcbz.

This worked extremely well, however, I established that by expanding the allocation size by 3x and calling dcbz at three distinct locations I was capable to province slightly additional force on CPU cache which would be crucial for another betterment I’ll get to afterward on. Here’s the final implementation:

Improved Thread Scheduling

In part 2 I talked concerning trying to schedule the utilize threads on distinct hardware threads and how I got the finest results by putting them on CPU hardware threads 0 and 1 (which are the two on center 0). I don’t really cognize item concerning how CPUs activity at a hardware flat but I suspected that the load/store recollection transactions from the two hardware threads on the identical center were being queued onto the L2 crossbar in several kind of per-core fashion. Thus by putting the utilize threads on the hardware threads for a sole center the writes from the utilize thread and says from the XKE payload thread would get queued together and decrease latency for loads/stores to RAM, or item akin that.

It’s likely not extremely amazing to anyone study this but CPUs frequently have “hidden” instructions, ones not defined by the education set architecture (ISA) but implemented by the CPU manufacturer. Sometimes these have a particular education encoding and another times you’d use an existing education in a nonsensical way to inform the CPU to do a particular operation.

In the case of PowerPC the ISA really has several fine defined instructions for changing the precedence of a hardware thread by executing a particular OR education alongside all 3 enroll operands set to the identical particular enroll according to what precedence you desire the hardware thread to run at. For example:

The Xbox 360 SDK docs depict a few intrinsic macros that can be used to set the precedence of the hardware thread the code is operating on: __SetHWThreadPriorityLow, __SetHWThreadPriorityMed, and __SetHWThreadPriorityHigh. The genuine CPU instructions used by these macros are slightly distinct than what I listed complete (these macros are xenon CPU specific) but that doesn’t really matter.

The SDK docs depict all center as having 32 dispatch slots which are divided evenly between hardware threads operating at average precedence (which is the default priority). By using the __SetHWThreadPriorityLow and __SetHWThreadPriorityHigh intrinsic macros you can have one hardware thread use 31 of the 32 dispatch slots and the another lone use 1. How the dispatch slots connect to figure of instructions fed to the CPU implementation units I’m not certain (again I have no idea how a CPU pipeline really works), but being capable to go from 16/16 to 1/31 appears beautiful helpful for scheduling the utilize threads.

Using these intrinsic macros Máté was capable to agenda the utilize threads specified that the hardware thread operating the XKE payload was lone getting 1 of the 32 dispatch slots during the ciphertext overwrite thread got the remaining 31 slots. The outcome was that the CPU was now spending the overwhelming bulk of period in the ciphertext finding iteration than it was really operating the XKE payload.

Improved Ciphertext Overwrite Detection

The final betterment was item I came up alongside during operating on the Peer Pressure utilize to try and create the Bad Update utilize as stable as I could get it. In the first PoC I had discovered the compressed bootloader data the XKE payload operates on was compressed into 15 blocks alongside all but the final obstacle having a decompressed size of 0x8000 bytes. If the XKE payload assault was prosperous part of the final section of the hypervisor would be overwritten alongside the decompressed data for one or additional of the compressed blocks. By observing the ciphertext for the final section of the hypervisor in two distinct locations I could decide if the final obstacle of compressed data was written to it by the XKE payload.

The assault was setup specified that if the final obstacle of compressed data was written to the hypervisor at a particular offset it would alter the instructions for one of hypervisor syscalls to a write-what-where primitive I could use to get complete hypervisor code execution. However, there was a flaw in how the ciphertext was being observed which not lone led to false positives that would collision the console but additionally missed iterations anywhere the final obstacle was written to recollection successfully.

Above is the first implementation for the utilize thread that ran the XKE payload and checked for ciphertext changes in the final section of the hypervisor. The lines applicable to ciphertext study have been highlighted. Because the identical thread is the two operating the XKE payload and checking for ciphertext changes it can lone detect if the ciphertext has changed at all and not how many times it changed. Put additional simply: I could lone infer if among the compressed blocks was written to memory, but not how many blocks were written. With that in intellect lets run through two scenarios that could happen whenever operating an repeat of the XKE payload during attempting to get compressed obstacle 14 written to memory.

Missing Successful Writes

While operating the XKE payload the competition assault is prosperous on several obstacle N specified that N < 14 (0x8000 bytes get written) and prosperous on obstacle 14 (less than 0x8000 bytes). Upon returning from HvxKeysExecute we detect the ciphertext for two locations in the mark compose area and see that the ciphertext has changed in the two locations. We can’t infer that obstacle 14 was written following obstacle N since we don’t have the capability to detect how many times the ciphertext changed. Thus we regard this a unsuccessful repeat equal although it was successful.

False Positive Write

Assume we have fair executed an repeat of the XKE payload that exhibits the missed prosperous compose behavior for several obstacle N and obstacle 14. The final section of the hypervisor now contains the data for obstacle 14 and the rear data for several obstacle N. We run another repeat of the XKE payload and this period lone obstacle N is written. When we detect the ciphertext changes for this repeat we’ll see the ciphertext at the archetypal location has alter but the ciphertext at the second location hasn’t since it already contained the ciphertext for the rear data of obstacle N from the former iteration.

This gets detected as a compose for obstacle 14 and the utilize volition try to execute several hypervisor code assuming the education form for the mark syscall has been changed to the write-what-where form established in bock 14. However, the education form is really random instructions established in obstacle N and volition logic the console to crash.

The New Detection Method

Now that the cache flushing primitive was considerably improved it was imaginable to detect the ciphertext changes for all obstacle of compressed data that was successfully written to the mark area of hypervisor memory. The resolution I came up alongside for the detecting if obstacle 14 was written to recollection was fairly simple: I created a new thread that continuously watched the ciphertext for changes and counted how many changes were observed.

If the oppose reached 15 formerly the XKE payload was done executing afterward we cognize all 15 blocks were written to recollection and thus the final obstacle had to have been written. If we observed item small we merely regard it a unsuccessful repeat and try again. Here’s the code for the new ciphertext finding thread:

Too Much Pressure

The new ciphertext study function worked awesome but there was a new issue it introduced. Due to the amount of force I was putting on CPU cache the console would collision concerning 30% of the period whenever executing the XKE payload. When the XKE payload runs it takes the compressed update data buffer and does an in-place relocation to move it from unprotected recollection to protected recollection so it can’t be tampered alongside asynchronously. After the XKE payload runs its chief logic it’ll do another in-place relocation to move the update data rear to unprotected recollection before returning.

After modifying the XKE payload and inserting a bunch of printf calls I was capable to decide that the console would collision in a function I called HvpRelocateProtectedToPhysical whenever trying to move the update data from protected recollection rear to unprotected memory. The function plant by looping complete a area of recollection and for all CPU cache row value of data (128 bytes) it will:

1. Read the complete cache row into CPU registers using the protected origin address.
2. Invalidate the two data and education caches for the origin address, zero the cache row for the destination address.
3. Write the data rear to recollection using the unprotected destination address.

Here’s the disassembly for the inner iteration of the HvpRelocateProtectedToPhysical function:

In the modified type of the XKE payload I added a bunch of printf calls in a re-implemented type of this function so I could detect were it would crash:

After operating a few dozen iterations I was capable to decide that the CPU would collision whenever trying to compose data rear to the unprotected destination location on the row highlighted in red. As I mentioned before in the improved cache flushing division I made several modifications to Máté’s first implementation to put additional force on cache which made the new ciphertext study method additional reliable.

I don’t cognize exactly what’s happening current that causes the CPU to collision whenever trying to compose data rear to the unprotected destination address. Perhaps among the cache invalidation instructions didn’t completely complete before the shop education was executed, or perchance there’s several border case alongside regard to the tradition recollection pathways whenever the CPU is starved for cache. Regardless, I did oversee to discover a way to activity about this issue.

The New Detection Method v2

Since the collision happens following we would have been capable to detect all 15 blocks of data being written to hypervisor recollection the resolution I came up alongside was to immediately halt the cache flushing thread and autumn the L2 locks formerly the ciphertext alter oppose hits 15. This reduces the force on CPU cache before the call to HvpRelocateProtectedToPhysical which prevents it from crashing. Here’s the updated implementation of the ciphertext study thread logic:

I moved the L2 cache locks into the ciphertext study thread which we can see on row 5. Once we detect all 15 blocks of data have been written to recollection we autumn the L2 locks on row 36, and indication to the cache flushing thread to immediately terminate on row 40.

There’s motionless an border case anywhere we don’t detect all 15 blocks being written and thus won’t be capable to decrease force on CPU cache before the call to HvpRelocateProtectedToPhysical. But this didn’t appear to matter. With all the improvements the utilize is normally prosperous in the archetypal run of the XKE payload and at worst it may obtain 2-5 iterations, but I didn’t encounter any crashes whenever testing. It’s motionless imaginable it could happen but I think it’s extremely rare it does.

So there you have it, gratitude to all the improvements from ihatecompvir, InvoxiPlayGames, and Máté the Bad Update utilize is now instantaneous and has nearly 100% achievement rate, and you lone need to copy a few records to a USB rod to hack your console. When I released the first PoC I made several statements akin “I don’t think this can be optimized any further” mostly to evade endless comments from non-technical group spit balling random useless ideas. The fact is I continually idea it could be optimized additional but I didn’t anticipate it would always attain the item it’s at now.

I think the biggest item I learned current is that I should have taken a interrupt following I completed the archetypal repeat of Bad Update and came rear alongside a apparent intellect to try and refine it. Having fair established among the lone two bugs in the Xbox 360 hypervisor I put a lot of force on myself to get this utilize operating at all. It took a lot of attempt and sleepless days to drag it off and in the end I got tunnel imagination chasing issues that didn’t really matter which distracted me from the genuine issues at hand. Had I taken a interrupt following the first PoC and came rear a few weeks afterward I likely could have had item additional refined for the first publish equal if it wasn’t as refined as it is now.

Any ways, that does it for part 3. You can discover the origin code for the Bad Update improvements on my GitHub. Part 4 which covers the Peer Pressure utilize volition be out shortly so remain tuned for that!