4.5B Posts Scraped from TikTok

What this is

Almost each TikTok scraper you will find drives a headless browser aliases hits the nationalist web endpoints. Both are the incorrect layer: slow, fragile, and missing astir of the absorbing fields. The Android app does not usage either. It talks to a private HTTP+JSON API, the aforesaid 1 com.zhiliaoapp.musically hits erstwhile you scroll, and that API is fast, stable, and returns acold more.

Getting into it is the difficult part, and it is difficult successful a circumstantial way. Four wholly unrelated things person to beryllium correct astatine once: a instrumentality credential TikTok issued, a valid petition signature, the correct regional host, and a TLS handshake that looks for illustration a phone.

Get immoderate 1 of them incorrect and you person the identical response: a cleanable HTTP 200 pinch an quiet body. No correction message. No position code. Your HTTP customer reports success, your logs enactment green, and your database fills pinch nothing. There is nary awesome telling you which of the 4 you are opinionated at.

This article walks done each four, past documents the 24 endpoints that travel out the different side. It names the primitives, shows the existent pipeline and gives measured numbers alternatively than claims.

None of the 4 has a feedback loop. A incorrect rotation constant, a incorrect byte order, a incorrect big and a incorrect cipher suite successful the handshake each nutrient the same well-formed petition and the aforesaid quiet response, truthful location is nary correction to bisect on and nary partial credit.

Anatomy of a request

Before thing else, present is what 1 of these requests really looks like. This is simply a existent call, pinch the identifying values shortened:

GET /aweme/v1/aweme/post/ ?# ── what you are asking for ────────────────────────────── source=0 &user_id=6744630345964389381 &count=20 &max_cursor=1751028792000 &sort_type=0 &# ── who is asking: 38 params, bid matters ───────────── ts=1788361402&ac=mobile&ac2=lte &aid=473824 # app id: TikTok Lite &iid=7680617333853718293 # instal id ← from register &device_id=7680616891110524437 # instrumentality id ← from register &cdid=4a1d... # client-generated uuid &openudid=8f2c... # client-generated 16-hex &device_brand=Samsung&device_type=SM-A136U&os_version=12&os_api=30 &resolution=1080*2280&dpi=440&host_abi=arm64-v8a &region=SG&carrier_region=SG&sys_region=SG&mcc_mnc=52506 &language=ja&app_language=ja&locale=ja-SG&timezone_name=Asia%2FSingapore &version_name=32.8.2&version_code=320820&manifest_version_code=320820 &_rticket=1788361402193&channel=googleplay&app_type=normal Headers: user-agent: com.ss.android.ugc.tiktok.lite/320802 (Linux; U; Android 12; ...) x-tt-trace-id: 00-6a9f...-6a9f...-01 x-ss-req-ticket: 1788361402193 x-khronos: 1788361402 # timestamp x-ladon: XKp9... # Speck-128/256 x-argus: cQqbRZm8k1x... # the difficult one x-gorgon: 0404b0d30000... # bequest digest A azygous creator-timeline request. Everything beneath the fold is instrumentality identity.

Three things to notice, because each 1 bites later:

• Two thirds of the URL is instrumentality identity. Thirty-eight common parameters picture the handset, the carrier, the region and the app build. They are not decoration. The signature covers them.
• device_id and iid are issued by TikTok, not chosen by you. cdid and openudid you make and taxable astatine registration. Getting the favoritism incorrect is the first wall.
• Parameter bid is fixed. The signature hashes the query string arsenic a literal, truthful url.Values.Encode(), which sorts keys alphabetically, silently produces an invalid signature. In Go you person to build the query by hand.

The vocabulary, since it recurs throughout:

The first quiet 200

A correctly implemented signer produces output that verifies against captured traffic, pinch parameters matching byte for byte. The consequence is still this:

$ curl -sD- -o /tmp/body "https://api16-normal-c-alisg.tiktokv.com/aweme/v1/user/profile/other/?..." HTTP/1.1 200 OK content-type: application/json content-length: 0 x-tt-logid: 2026090117... server: TLB $ wc -c /tmp/body 0 /tmp/body Two hundred. Zero bytes. No status_code, because location is nary assemblage to put 1 in.

This is TikTok's soft block, and it is the azygous astir important point to understand astir this API. It is not a 403. It is not a 429. It is not a situation page. It is a successful HTTP consequence containing nothing.

Which intends this code, which is what everyone writes first, is silently broken:

res = requests.get(url, headers=signed) if res.ok: # True. Always true. store(res.json()) # {} stored, nary exception # six hours later: 400,000 rows successful the database, each empty, # thing successful the correction log, dashboard green

It is costly to debug because four unrelated failures nutrient it:

1. Your instrumentality was ne'er activated (§ activation)
2. Your signature is incorrect (§ X-Argus)
3. You are talking to the incorrect location big (§ regions)
4. Your TLS handshake looks for illustration a server, not a telephone (§ JA3)

There is thing successful the consequence to show you which. You cannot bisect it by reading errors, because location are none. The only measurement done is to hole each 4 and measure each 1 successful isolation.

Where instrumentality IDs travel from

You cannot invent a device_id. TikTok issues it, from /service/2/device_register/ connected its logging host, successful speech for a plausible handset.

The petition assemblage is simply a JSON archive (app header, instrumentality header, civilization block) encrypted pinch TTEncrypt (TikTok's ain assemblage cipher, a elemental byte-level transform pinch a fixed cardinal schedule) and posted as application/octet-stream;tt-data=a. It goes retired pinch the full signature set, truthful you request moving signing earlier you tin get a device, and the signing needs a device. You bootstrap pinch the client-generated fields and zeros wherever the issued ones go.

The body's shape, pinch the parts that matter:

{ "magic_tag": "ss_app_log", "header": { // app identity: must work together pinch the assistance successful the query string "aid": 473824, "package": "com.ss.android.ugc.tiktok.lite", "app_version": "32.8.2", "version_code": 320820, "sdk_version": "...", "git_hash": "...", "sig_hash": "...", // hardware: each section present has to beryllium internally consistent "device_model": "SM-A136U", "device_brand": "Samsung", "device_manufacturer": "samsung", "cpu_abi": "arm64-v8a", "os_version": "12", "os_api": 30, "resolution": "2280*1080", "density_dpi": 440, "rom": "...", "rom_version": "...", // personality you make and are astir to waste and acquisition in "cdid": "<uuid4>", "openudid": "<16 hex>", "clientudid": "<uuid4>", "google_aid": "<uuid4>", // region: bearer must plausibly beryllium successful this country "region": "SG", "sim_region": "sg", "carrier": "Singtel", "mcc_mnc": "52506", "tz_name": "Asia/Singapore", "tz_offset": 25200, "custom": { "screen_width_dp": 408, "screen_height_dp": 883, "web_ua": "Dalvik/2.1.0 (Linux; U; Android 12; SM-A136U Build/...)", "apk_last_update_time": 1788361409271 }, "apk_first_install_time": 1788360902118 }, "_gen_time": 1788361402240 }

Every section location is checked against the others. A Samsung SM-A136U has a circumstantial screen resolution, a circumstantial DPI, a circumstantial ABI, and shipped pinch a circumstantial scope of Android versions. It is sold connected carriers successful immoderate countries and not others. A flagship handset connected a web that ne'er carried it is not a existent phone, and the registration is refused.

Rather than generating these procedurally, I build them from a catalogue of ~250 existent Android instrumentality profiles crossed pinch a bearer array of MCC/MNC pairs (roughly 2,000 rows, derived from nationalist numbering-plan data). Pick a handset, pick a bearer that really exists successful the target country, capable successful the coherent values.

A successful registration comes backmost pinch the 2 ids you needed:

{ "device_id_str": "7680616891110524437", "install_id_str": "7680617333853718293", "new_user": 1 }

Most implementations extremity here.

The activation call

With registration working, astir endpoints answered. Video listings, search, hashtags, sounds, each fine. But /aweme/v1/user/profile/other/, the afloat floor plan record, returned the quiet 200 every azygous time, connected every instrumentality I made, forever.

The evident fishy is the signature, and it is the incorrect one. The show is that an older excavation of devices, generated months earlier by different code, worked good on that aforesaid endpoint pinch the aforesaid signer and the aforesaid parameters. The only quality was successful really the devices had been created, and it came down to 1 extra HTTP call:

GET /service/2/app_alert_check/?<common params> &cronet_version=...&ttnet_version=... &tt_info=<base64url(TTEncrypt(<60-field key=value blob>))> → {"message":"success"}

That is it. It returns thing you need. It looks for illustration telemetry, and functionally it is telemetry. It is the telephone the existent app makes connected launch, earlier it requests immoderate data.

That is what the telephone is for. A instrumentality that registered and past instantly started querying the API is, from ByteDance's side, an instal that never launched. Registration unsocial does not make you a moving app. The startup telephone does.

The tt_info blob is the absorbing portion of the request: astir sixty key=value pairs (GAID, timezone, instal id, instrumentality id, carrier, screen, ABI, locale, a petition UUID) TTEncrypt-ed and base64url-encoded. It is the app reporting its afloat situation connected startup. My guess, and it is only a guess, is that this is wherever the instrumentality gets marked arsenic a existent instal alternatively than a bare registration; I person not tried to beryllium it, because the empirical consequence is unambiguous.

Proving a instrumentality earlier you usage it

The activation fixed the floor plan endpoint, but it introduced a second-order problem: activation itself sometimes fails silently, and a instrumentality that grounded activation looks precisely for illustration a instrumentality that succeeded until you usage it.

So procreation does not extremity astatine activation. It ends pinch a existent publication against a known creator. If existent contented comes back, the instrumentality joins the pool. If not, it is thrown away. Not retried, not quarantined. Discarded.

func GenerateDevice(client *http.Client, state string) (map[string]any, error) { tmpl, err := NewAndroidTemplate() // handset × carrier ... if err := registerDevice(client, tmpl); err != nil { return nil, fmt.Errorf("register: %w", err) } // Without this TikTok will not service floor plan item to a caller device. if err := appAlertCheck(client, tmpl); err != nil { return nil, fmt.Errorf("activate: %w", err) } // Survivorship filter: only provably-capable devices participate the pool. if !profileCapable(client, tmpl) { return nil, errors.New("profile probe failed: instrumentality not capable") } return tmpl, nil }

The three-stage pipeline. Roughly 60-95% of attempts past it, depending almost wholly connected proxy quality.

Without the select you get a excavation that is simply a substance of moving and softly dead devices, and because dormant devices return the quiet 200, the aforesaid arsenic each different failure, the excavation degrades invisibly. Your occurrence complaint drifts down complete days and location is thing successful the logs to explicate it.

With the filter, the excavation is uniformly tin by construction. Live wellness is visible from the moving server:

$ curl -s localhost:8080/v1/devices | jq { "live": 43, "generated_total": 43, "rejected_total": 2, "evicted_total": 0, "success_total": 177, "failure_total": 74, "generation_survival_rate": 0.9555 }

Inside X-Argus

X-Argus is not a hash of a string. Its plaintext is a protobuf message successful proto3 ligament format, varints and length-delimited fields, which is past tally done a two-stage encryption pipeline.

The connection carries, among different fields:

type Argus struct { Magic int32 // fixed marker Version int32 Rand int64 // per-request random, 0x10000000..0xFFFFFFFF MsAppID drawstring // "1233" / "473824" LicenseID string DeviceID string SdkVersion int32 SdkVersionStr string AppVersion string EnvCode []byte CreateTime int64 // X-Khronos, again, wrong the blob BodyHash []byte // SM3 of the assemblage (16 zero bytes connected GET) QueryHash []byte // SM3 of the literal query string AlgorithmCount struct { SignCount int32 // really galore signatures this instal has made ReportCount int32 SettingCount int32 Timestamp int64 } SecDeviceToken string IsAppLicense int64 PskHash []byte CallType int32 ChannelInfo struct { PhoneInfo, Channel string; ... } }

The subset the signer really populates. Establishing the section numbering is astir of the reverse-engineering work.

AlgorithmCount.SignCount is simply a antagonistic of how galore requests this instal has signed. A existent phone's antagonistic climbs steadily over the life of the install. A scraper that emits a constant, aliases resets to zero connected every request, is producing a statistically evident shape moreover erstwhile each individual signature verifies. I seed it randomly per instrumentality successful a plausible scope and it has ne'er been a problem, but it is the benignant of section that exists specifically truthful that naive replay is detectable successful aggregate alternatively than astatine the individual request.

The pipeline

Once the protobuf is serialised, it goes done this, successful order:

1. pb = proto3_serialize(Argus{...}) 2. padded = pkcs7(pb, 16) // cardinal derivation: the signing cardinal is simply a per-aid 32-byte constant 3. xmKey = SM3( signKey[0:32] || f(rand_lo, rand_hi) || signKey[0:32] ) 4. enc1 = Simon-128/256-ECB( cardinal = xmKey, padded ) 5. enc1 = reverse_bytes(enc1) 6. enc1 = xor_mix(enc1, derived_from(rand)) // bit-level, order-sensitive // framing: a type byte, entropy, and a 3-byte marker built from // the first bytes of 2 abstracted SM3 digests 7. framed = hexFirstByte(aid) || rand_bytes || append_array || enc1 8. enc2 = AES-128-CBC( cardinal = MD5(signKey[0:16]), iv = MD5(signKey[16:32]), framed ) 9. X-Argus = base64( rand_lo || enc2 )

Two encryption layers pinch different primitives and different cardinal derivations, pinch a byte reversal and an XOR operation sandwiched betwixt them. None of the individual steps is hard. The trouble is wholly that there is nary feedback . Get measurement 6 incorrect and you nutrient a perfectly well-formed, correctly-sized, base64-clean header that TikTok answers pinch an quiet 200.

Which is why the implementation ships pinch independent trial vectors for every primitive. You verify Simon, Speck, SM3 and TTEncrypt separately against known input/output pairs, truthful that erstwhile a petition fails you already cognize the crypto is correct and the bug is successful composition.

Simon, Speck and SM3

The prime of primitives is deliberate, and it says thing astir the threat model.

ARX ciphers

Simon and Speck are lightweight artifact ciphers published by the NSA successful 2013. Both are ARX constructions, built entirely from modular Addition, bitwise Rotation and Xor. No S-boxes. No lookup tables. No multiplication.

Speck's information function, successful full, is 2 lines:

x = (ROR(x, α) + y) ⊕ k y = ROL(y, β) ⊕ x // for the 128-bit artifact size: α = 8, β = 3, 64-bit words // 128/256 configuration: 256-bit key, 34 rounds

Simon is the aforesaid thought pinch the summation swapped for AND, which makes it cheaper in hardware and somewhat much costly successful software:

x' = y ⊕ (ROL(x,1) & ROL(x,8)) ⊕ ROL(x,2) ⊕ k y' = x // 128/256 configuration: 256-bit key, 128-bit block, 72 rounds, // information constants from the Z4 series (a 62-bit LFSR period)

Why these and not AES? Three reasons, and they each constituent the aforesaid way:

• They compile to almost nothing. A fewer 100 bytes of ARM, nary tables, nary data-dependent representation access. That matters erstwhile the codification lives wrong an obfuscated autochthonal room that has to beryllium mini and has to avoid cache-timing broadside channels that would make it easy to locate.
• They are not successful your modular library. AES is simply a usability telephone in each language. Simon and Speck you person to implement, and the parameterisation abstraction is ample (block size, cardinal size, information count, rotation constants, cardinal schedule) truthful a incorrect conjecture produces plausible ciphertext and nary error.
• They are easy to get subtly wrong. Speck's cardinal schedule reuses the information usability itself. Get the connection bid aliases the endianness incorrect and you get 32 valid-looking information keys that are each incorrect.

Note that AES-128-CBC is successful the pipeline, arsenic the outer layer. The absorbing creation prime is that the soul layer, the 1 really protecting the protobuf, is the 1 you can't conscionable call.

SM3

SM3 is the Chinese nationalist cryptographic hash standard (GB/T 32905-2016). 256-bit output, 512-bit blocks, Merkle-Damgård building with a compression usability structurally akin to SHA-256 but pinch 2 parallel message description schedules and a different information function:

// 2 boolean functions, switching astatine information 16 FF(x,y,z) = x ⊕ y ⊕ z // j < 16 = (x&y) | (x&z) | (y&z) // j ≥ 16 GG(x,y,z) = x ⊕ y ⊕ z // j < 16 = (x&y) | (~x&z) // j ≥ 16 // IV 7380166F 4914B2B9 172442D7 DA8A0600 A96F30BC 163138AA E38DEE4D B0FB0E4E

SM3 shows up successful ByteDance's stack for the evident reason. It is also, usefully for them, absent from each Western modular library. And the 2 connection description arrays (W and W') are trivially transposable, truthful a large fraction of the reference implementations floating astir are incorrect successful ways that only show up connected definite inputs.

TTEncrypt

The assemblage cipher, utilized for the registration payload and the activation blob. Not a modular construction, conscionable a fixed-key byte toggle shape pinch a mini table. It is not cryptographically superior and is not meant to be; it exists to extremity casual traffic inspection, and it is the easiest of the 4 to reimplement.

X-Ladon

Much simpler than Argus, and worthy showing successful afloat because it is simply a bully illustration of really these schemes are layered: a inexpensive gross successful beforehand of an costly one.

plaintext = "<khronos>-<license_id>-<aid>" key = ascii_hex( MD5( rand_bytes(4) || assistance ) ) // 32 bytes cipher = Speck-128/256-ECB( key, pkcs7(plaintext) ) X-Ladon = base64( rand_bytes || cipher )

Four random bytes, an MD5, and a Speck encryption of a dash-joined string. The random bytes are prepended to the output truthful the server tin rederive the key. That is the full construction.

It filters retired anyone who hasn't looked astatine the app astatine all, and costs approximately thing to verify astatine scale. Argus is the costly cheque that runs after.

Version gating

A correct signature is basal and not sufficient. Some endpoints are gated connected the customer build, and the gross is server-side.

The clearest lawsuit is comments. Same device, aforesaid signer, aforesaid second, same everything. Only version_code differs:

The type bump besides unlocked remark replies and follower listing. It is not that the older build's signature is rejected, because it verifies fine. It is that the endpoint is simply not served to that customer version.

Practically this intends the app type is simply a per-endpoint property, not a global setting. In my catalogue each endpoint records the build it needs and the server swaps the 4 type fields transparently earlier signing:

func WithAppVersion(dev *DevInfo, version, codification string) *DevInfo { c := *dev c.App.AppVersion = version c.App.AppVersionCode = code c.App.ManifestVersionCode = code c.App.UpdateVersionCode = code return &c }

Pinning the newest build everyplace is not the answer, because newer builds tighten different checks. The catalogue exists truthful that each endpoint sits connected the build that useful for it.

Region partitioning

The 3rd gate, and the 1 pinch thing to spell on: nary error, nary redirect, nary hint successful the response.

TikTok does not tally 1 API. It runs respective location information centres: alisg (Singapore), useast1a, useast5 and others. And they do not service the aforesaid endpoints to the aforesaid devices.

With activation fixed, floor plan item still grounded connected freshly generated devices while moving connected an older pool. Same code, aforesaid signer. The quality turns retired to beryllium the host:

Same second, aforesaid credential, aforesaid signed request, 3 hosts, 1 answer. And music/detail is the reverse: it answers connected useast1a and returns thing connected the Singapore big that serves almost everything else.

So the big is portion of the endpoint definition. Not a world guidelines URL but a per-route property, established by measurement, because location is nary archiving to consult:

{ ID: "user.info", Route: "/v1/user/info", Host: tiktok.HostUSEast5, // the ONLY big that serves this to caller devices Path: "/aweme/v1/user/profile/other/", ... }, { ID: "music.info", Route: "/v1/music/info", Host: tiktok.HostUSEast1A, // and this 1 is the only big for THIS Path: "/aweme/v1/music/detail/", ... },

There is simply a useful second-order effect here. The device's registered region also influences content connected the region-scoped endpoints: trending sounds and trending class shelves. Running 1 excavation registered successful US and different successful BR gives you genuinely different charts from the identical call, which is really you get per-country information without immoderate per-country code.

The TLS fingerprint

The 4th gate, and the 1 that is invisible astatine each furniture an application developer usually inspects.

Before immoderate of your bytes arrive, your TLS customer sends a ClientHello. Everything in it, and crucially the order of everything successful it, is simply a fingerprint. JA3, the modular measurement of capturing this, is an MD5 of 5 comma-joined fields:

TLSVersion , Ciphers , Extensions , EllipticCurves , ECPointFormats 771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53, 0-23-65281-10-11-35-16-5-13-18-51-45-43-27-21,29-23-24,0 ↓ MD5 cd08e31494f9531f560d64c695473da9

A JA3 drawstring and its hash. The cipher database and the hold database are ordered, and libraries bid them differently.

That fingerprint identifies your TLS library, and often its version, with precocious precision. OpenSSL, BoringSSL, NSS, Go's crypto/tls, Java's JSSE are each distinguishable, earlier a azygous byte of HTTP is exchanged.

Go's crypto/tls has a very unique one. And nary Android app has ever emitted it, because Android apps usage BoringSSL done OkHttp. TikTok's useast5 separator checks.

The research that isolated it

The aforesaid petition useful from Python and fails from Go. Signature byte-identical, parameters byte-identical, cookies irrelevant (it useful pinch and without). Dump the nonstop headers Python conscionable used, replay them from Go, and clasp everything else constant. Same URL, aforesaid signature, aforesaid device, aforesaid second:

// identical request, 3 clients, backmost to back python urllib3 / OpenSSL → 9,517 bytes curl OpenSSL → 9,519 bytes go crypto/tls → 0 bytes ← HTTP 200

Nothing astir the petition was different. The handshake was.

The hole is uTLS, which lets you specify the nonstop ClientHello to emit alternatively of accepting the 1 Go builds for you:

cfg := &utls.Config{ServerName: host, NextProtos: []string{"http/1.1"}} conn := utls.UClient(raw, cfg, utls.HelloAndroid_11_OkHttp) if err := conn.HandshakeContext(ctx); err != nil { return nil, fmt.Errorf("utls handshake: %w", err) }

One statement of floor plan selection. Profile item connected caller devices went from 0% to 100%.

NextProtos is pinned to http/1.1 connected purpose. The Android floor plan advertises h2, but the carrier underneath this is HTTP/1.1 only. Negotiate h2 and you get a relationship thing tin speak on.

The Go proxy trap

Short, and circumstantial to Go.

Wire up uTLS, trial it directly, corroborate the fingerprint has changed, past put it down the rotating proxy. The failures travel consecutive back.

The logic is that http.Transport ignores DialTLSContext erstwhile Proxy is set. It dials the proxy, issues CONNECT itself, and past runs its ain standard-library handshake complete the resulting tunnel. Your civilization dialer is silently discarded. No error, nary warning, nary log line.

You person to do the passageway by hand:

dialTLS := func(ctx context.Context, network, addr string) (net.Conn, error) { // 1. plain TCP to the proxy raw, err := d.DialContext(ctx, "tcp", proxyURL.Host) ... // 2. CONNECT by hand: this is the portion Transport would person done req := &http.Request{Method: "CONNECT", URL: &url.URL{Opaque: addr}, Host: addr, ...} req.Write(raw) resp, _ := http.ReadResponse(bufio.NewReader(raw), req) if resp.StatusCode != 200 { return nil, fmt.Errorf("CONNECT: %s", resp.Status) } // 3. NOW tally the uTLS handshake complete the tunnel u := utls.UClient(raw, cfg, utls.HelloAndroid_11_OkHttp) return u, u.HandshakeContext(ctx) } tr := &http.Transport{ DialTLSContext: dialTLS, DisableKeepAlives: true, // spot the adjacent section // note: NO Proxy field. Setting it would bypass each of the above. }

The Proxy section is deliberately absent from the transport. Setting it is what silently discards the dialer.

Detecting the quiet 200

With each 4 gates passed you still request to know, per response, whether you really sewage data. Status codes will not show you. The cheque has to beryllium connected content:

if resp.StatusCode != http.StatusOK { return body, fmt.Errorf("upstream HTTP %d", resp.StatusCode) } if len(body) < minBodyBytes { // 64 // The soft block: 200 pinch (almost) thing successful it. return body, fmt.Errorf("empty upstream assemblage (%d bytes)", len(body)) } var probe map[string]json.RawMessage if err := json.Unmarshal(body, &probe); err != nil { // HTML, usually a proxy correction page alternatively than TikTok return body, errors.New("upstream assemblage is not a JSON object") } if raw, good := probe["status_code"]; good { var n int if json.Unmarshal(raw, &n) == nil && n != 0 { return body, fmt.Errorf("upstream status_code %d", n) } }

Four conditions, successful order: HTTP status, magnitude floor, parseable JSON object, cleanable soul status_code. Anything that fails 1 is retried against a different instrumentality from a different IP.

Except erstwhile retrying is pointless

Some non-zero status_code values are TikTok answering rather than refusing. Retrying those 4 times is simply a discarded of 4 devices and 4 IPs:

Which surfaces to the caller arsenic a existent reply alternatively of a gateway failure:

$ curl -s localhost:8080/v1/user/following?user_id=6744630345964389381 | jq { "error": { "code": "hidden_by_user", "message": "This creator has hidden their pursuing list. Most accounts do; location is nary measurement astir it.", "upstream_status_code": 3002060, "upstream_status_msg": "Profile personification is hiding pursuing list", "retried": false, "retry_would_not_help": true } }

Everything other keeps its afloat retry budget, and erstwhile it exhausts it you get the per-attempt breakdown alternatively than a generic failure, which is what makes this debuggable successful production:

{ "error": { "code": "upstream_failed", "attempts": 4, "attempt_failures": [ {"attempt": 1, "reason": "empty upstream assemblage (0 bytes)"}, {"attempt": 2, "reason": "empty upstream assemblage (0 bytes)"}, {"attempt": 3, "reason": "transport: ... EOF"}, {"attempt": 4, "reason": "upstream HTTP 429"} ] } }

Real output from the weakest endpoint successful the catalogue. Two soft blocks, a dropped connection, and an honorable complaint limit.

Keep-alive pins the exit IP

With each 4 gates passed, the highest-volume endpoint ran at 88.2% complete 174 cardinal attempts. Good, and astatine that measurement the missing 12% is 20 cardinal mislaid records.

The evident move is much retries. It does nothing, because of really complaint limiting and relationship reuse interact.

Rate limiting present is per exit IP. A rotating proxy gateway assigns an exit IP per TCP connection. HTTP keep-alive, which each customer does by default and which is usually precisely what you want, pins you to 1 exit IP for the life of that connection.

So the retry went retired from the reside that had conscionable been refused. And the adjacent one. And the next:

// keep-alive connected a rotating proxy attempt 1 → exit 203.0.113.44 → empty 200 attempt 2 → exit 203.0.113.44 → empty 200 ← aforesaid IP attempt 3 → exit 203.0.113.44 → empty 200 ← aforesaid IP attempt 4 → exit 203.0.113.44 → empty 200 ← aforesaid IP // caller relationship per attempt attempt 1 → exit 203.0.113.44 → empty 200 attempt 2 → exit 198.51.100.7 → ok

Four attempts, 1 IP, 4 identical failures. The retry fund bought thing at all. It was structurally incapable of helping.

Why the naive hole stalls astatine 96%

Setting DisableKeepAlives: existent everyplace took it to 96.2% and then stopped. The origin is that astatine full concurrency you are now paying a TLS handshake for every attempt, including the ~88% that were going to win first time. The proxy gateway, not TikTok, became the bottleneck and started refusing tunnels:

{"attempt": 1, "reason": "transport: proxy CONNECT: 466 Too Many Requests"}

The nonaccomplishment had moved, not gone. The accumulation style is simply a hybrid of the two, which comes down to 2 pools and 1 argumentation switch:

// First-attempt pool: keep-alive, truthful the communal lawsuit costs nary handshake. r.proxyPool, _ = httpclient.New(httpclient.Config{ProxyURL: cfg.ProxyURL}) // Retry pool: DisableKeepAlives => caller TCP => NEW exit IP. r.proxyPoolFresh, _ = httpclient.New(httpclient.Config{ ProxyURL: cfg.ProxyURL, DisableKeepAlives: true, }) // ...and successful the petition path: pool := r.proxyPool if effort > 0 && r.proxyPoolFresh != nil { excavation = r.proxyPoolFresh // rotation precisely wherever it matters }

Measured complete hundreds of millions of calls crossed 4 shards. The self-hosted server described beneath keeps the simpler always-fresh form, because a single lawsuit is obscurity adjacent the load wherever the 2nd bottleneck appears.

Proxies: the 1 moving cost

Everything up to present is simply a package problem you lick once. The proxy is the single outer dependency and the only recurring cost, and the request for 1 is structural alternatively than incidental.

What a proxy is, briefly

A proxy is simply a instrumentality that makes the petition connected your behalf. You link to it, it connects to TikTok, and TikTok sees the proxy's IP reside alternatively of yours. A rotating gateway is 1 wherever each caller relationship comes retired of a different reside successful a ample pool, which is the spot that matters here.

Why it is mandatory alternatively than recommended

Two reasons, and the 2nd is the 1 group underestimate.

Rate limiting is per exit IP. One reside gets a fund and it is not a ample one. Without a proxy each petition successful your strategy shares a single address, and you exhaust it successful minutes.

Retries are structurally useless without rotation. This is the constituent from the previous section. When a petition is soft blocked, the retry has to time off from a different reside aliases it fails identically. Rotation per relationship is the full system down the jump from 88% to 99.3%. A fixed proxy gives you 1 IP and truthful gives you nothing.

So the request is simply a rotating gateway, and you tin verify yours really rotates successful 1 statement earlier you perpetrate to anything:

$ for one successful 1 2 3; do curl -s --proxy "$PROXY_URL" https://api.ipify.org; echo; done 203.0.113.44 198.51.100.7 # different IP each clip = rotating, good 192.0.2.19

If the aforesaid reside comes backmost 3 times, your retry fund is ornamental and your occurrence complaint will beryllium adjacent the first-try complaint nary matter what you set MAX_ATTEMPTS to.

What it really costs

The first point to cognize is that you should not beryllium paying by the gigabyte. Metered plans are the default proposal successful this abstraction and they are the incorrect style for this workload, because the endpoints that return the astir useful information are the ones measured successful megabytes. A page of videos is 1.1 MB. A page of recommended creators is 2.9 MB. Metered billing turns each 1 of those into a statement item.

Flat monthly subscriptions beryllium for some proxy types, and they are what you want. Two tiers screen fundamentally everyone:

For enriching a fewer 100 1000 creators, search sounds daily, aliases mapping a niche, the $150 tier is capable alternatively than a compromise: a rotating datacenter excavation registers devices, passes activation and sustains the occurrence rates successful the array further down.

The residential tier is what you escalate to erstwhile you are saturating the datacenter one.

What the $150 tier buys

On a level scheme the 2 limits are threads (how galore requests tin be successful flight) and line speed (how galore bytes per second). Which one binds depends wholly connected consequence size, and the endpoints present disagree by 2 orders of magnitude: a floor plan is 9.5 KB, a page of videos is 1.1 MB.

The figures beneath are arithmetic from the measured consequence sizes and latencies in the benchmark, astatine 100 threads and 200 Mbit/s. They are ceilings astatine afloat saturation, truthful dainty them arsenic an precocious bound alternatively than a promise.

On the mini endpoints you tally retired of threads agelong earlier bandwidth, truthful the hole is simply a higher thread count. On the video endpoints you saturate the line astatine astir 22 requests a second, and much threads bargain you thing astatine all. That is the number MAX_CONCURRENT exists to control, and mounting it supra what your scheme tin transportation produces proxy CONNECT: 466 Too Many Requests successful the effort failures rather than much throughput.

What to look for erstwhile buying one

• Rotating, pinch a azygous gateway endpoint. Verify rotation pinch the loop supra earlier you salary for a month.
• A published thread limit. If it is not stated, presume it is low. This is the number you really scheme around.
• Flat complaint complete metered, unless you cognize your measurement is mini and stays small. Metered plans punish precisely the endpoints that return the astir useful data.
• Country targeting, if you want location charts. The instrumentality region and the exit region should agree.
• A proceedings aliases 1 period first. Registration occurrence complaint is the existent trial and it varies betwixt providers advertizing the aforesaid product. Generate 30 devices and publication generation_survival_rate earlier committing.

Measured occurrence rates

Every endpoint ships pinch a existent occurrence complaint alternatively than a claim: 100 calls each, astatine astir 4 attempts, against a freshly generated excavation complete a rotating proxy gateway. Seeds are discovered unrecorded by stepping the API alternatively than hardcoded, which changes the numbers. The statement beneath explains why.

Average attempts is the much informative column. A 100% endpoint astatine 1.00 attempts succeeds first time, each time. A 92% endpoint astatine 2.12 attempts is being soft-blocked connected astir half its first tries and recovering connected retry, which intends a wider budget moves it, whereas thing moves a first-try-clean endpoint because location is nothing to move.

feed.recommended is genuinely weak, astatine 10-25% crossed runs, and it is dominated by honorable 429s alternatively than soft blocks. An anonymous device pinch nary watch history asking for a personalised provender is precisely the postulation shape TikTok astir wants to throttle. It ships documented arsenic anemic pinch the 2 100% alternatives named successful its place.

The 24 endpoints

All of the supra is packaged arsenic a self-hosted Go service. One binary, nary database, nary queue, nary emulator, nary autochthonal library. Reference information is compiled in.

$ cp .env.example .env # group PROXY_URL $ docker constitute up -d $ docker constitute logs -f TikTok Open API 1.0.0 starting config: port=8080 country=SG pool=15/30 attempts=4 concurrency=32 proxy=http://***@gw:9000 auth=true pool: 0 device(s) live, filling to 30 ... listening connected http://0.0.0.0:8080 (GET /healthz, GET /v1/endpoints) pool: first capable complete, 43 device(s) live

Real startup output. Cold commencement is 15 to 60 seconds; the excavation persists to disk truthful restarts aft that are instant.

All 24 routes are GET, each return query parameters, each return TikTok's JSON unmodified.

Creators

Videos

Sounds

Hashtags and search

Discovery

A existent response

Trimmed to the absorbing fields. The earthy entity has respective 100 keys:

$ curl -s "localhost:8080/v1/user/posts?user_id=6744630345964389381&count=20" \ | jq '{has_more, max_cursor, first: (.aweme_list[0] | {aweme_id, desc, statistics, music, author})}' { "has_more": 1, "max_cursor": 1751028792000, "first": { "aweme_id": "7678101694902832397", "desc": "Who Remembers 2022? #fortnite #piececontrolkyle #dogwater", "statistics": { "play_count": 19438, "digg_count": 2461, "comment_count": 39, "share_count": 176 }, "music": { "id_str": "7245172246876227585", "title": "Need 2 (Instrumental)" }, "author": { "uid": "6744630345964389381", "unique_id": "freakynaughty", "nickname": "freaky", "follower_count": 1277258 } } }

Note that the writer entity is embedded successful each video. One petition gives you twenty videos and the afloat creator record. On the web that is twenty-one requests.

Request metadata comes backmost successful headers alternatively than polluting the body:

HTTP/1.1 200 OK X-Endpoint-Id: user.posts X-Attempts: 2 ← first effort was soft-blocked X-Elapsed-Ms: 1874 X-Upstream-Region: sg X-Device-Region: SG

Fields the web does not springiness you

The past statement was measured crossed 579 creators connected some endpoints that could plausibly transportation it. The outbound floor plan nexus is simply a web-surface section only.

Tutorial: a sound-trend detector

Concrete worked example, because the endpoint database connected its ain does not show you what the information is bully for. The goal: find sounds that are taking disconnected right now, earlier they are evidently trending.

The awesome is music.user_count, really galore videos person been made pinch a sound. The absolute number tells you a sound is big. The rate of change tells you it is moving, which is the portion you want.

Step 1. Snapshot the chart

curl -s "$API/v1/music/trending?count=50" \ | jq -r '.music_list[] | [.id_str, .user_count, .title] | @tsv' \ > "sounds-$(date +%s).tsv"

Run it hourly from cron. Each statement is id, uses, title.

Step 2. Diff consecutive snapshots

import glob, csv, collections snaps = sorted(glob.glob("sounds-*.tsv"))[-2:] prev, curr = [{r[0]: (int(r[1]), r[2]) for r in csv.reader(open(f), delimiter="\t")} for f in snaps] movers = [] for mid, (n, title) in curr.items(): was = prev.get(mid, (0, title))[0] if was > 0: movers.append((n / was - 1, n - was, title, mid)) for growth, delta, title, mid in sorted(movers, reverse=True)[:10]: print(f"{growth:6.1%} +{delta:>8,} {title[:40]:<40} {mid}")

Step 3. Confirm it is really accelerating

Growth successful the floor plan is simply a candidate, not a confirmation. The cheque that separates a existent acceleration from a chart-placement artefact is the time dispersed of recent videos. Pull the newest videos utilizing that sound and look astatine really tightly their upload times cluster:

curl -s "$API/v1/music/posts/fresh?music_id=$MID&count=30" \ | jq '[.aweme_list[].create_time] | (max - min) / 3600' 2.4

Thirty videos successful a 2.4-hour model intends 30 group picked up that sound this afternoon. Compare pinch the celebrated ordering, which tells you whether it has already landed:

curl -s "$API/v1/music/posts?music_id=$MID&count=30" \ | jq '[.aweme_list[].statistics.play_count] | add'

High fresh-clustering positive debased cumulative plays is the interesting quadrant. Lots of group utilizing it, not overmuch accumulated scope yet. That is simply a sound connected the measurement up alternatively than 1 connected the measurement down.

Step 4. Find who is driving it

curl -s "$API/v1/music/posts/fresh?music_id=$MID&count=30" \ | jq -r '.aweme_list[].author | [.follower_count, .unique_id] | @tsv' \ | benignant -rn | head

Because the writer entity is embedded, this costs nary other requests. If 1 large relationship is astatine the apical and everyone other is small, you are looking astatine a sound that one creator kicked off, which is simply a different (and usually shorter-lived) arena than integrated uptake crossed galore mid-sized accounts.

Step 5. Widen it

music.trending is region-scoped to the device. Run a 2nd instance pinch POOL_COUNTRY=US and a 3rd pinch POOL_COUNTRY=BR and you get 3 independent charts from identical code. Sounds often break successful one marketplace days earlier another.

Getting the code

Everything described present is simply a backstage Go repository. One-time payment, permanent access, complete source.

Before you tally it

The 1 difficult request is simply a rotating proxy gateway. Rate limiting is per exit IP and the retry creation assumes a caller relationship gets a new address, truthful a fixed proxy is nary amended than none. It is the azygous external dependency and it is not optional astatine volume.

Maintenance is yours erstwhile you self-host. The repository is system truthful that when thing moves it is usually 1 struct literal successful 1 table, but it is your struct literal.

If you would alternatively skip the setup entirely, location is a done-for-you tier wherever I build it connected your server and manus it complete running.

Done for you

The repository is 1 constituent of a postulation system, and connected its ain it answers 1 petition astatine a time. Getting from location to billions of records is simply a different portion of work: deciding what to fetch next, keeping the queue moving through failures, landing the results location that is still queryable astatine that size, and moving the full point crossed capable shards and proxy capacity to prolong the rate.

This tier is that system, built connected your infrastructure and handed complete running. Not a demo pointed astatine a fewer creators. The aforesaid style arsenic the 1 that produced the figures astatine the apical of this page, sized to what you are collecting.

Storage, if you are keeping the data

Collecting the information is what the repository solves. Keeping it queryable erstwhile there are billions of rows is simply a abstracted problem pinch its ain nonaccomplishment modes, and it is the half that decides whether the postulation was worthy doing. If you are building a shop alternatively than moving a one-off pull, that creation is portion of the handover: array engines, partition and benignant keys, the update way for records that change, and the denormalisation that keeps a creator-to-video-to-sound mobility answerable without a subordinate crossed billions of rows.

Four decisions that find whether it holds, each of them measured connected a accumulation ClickHouse shop astatine billion-row scale:

• Partition keys. A bare modulus of an id looks moreover and is not, because level ids are not uniformly distributed. One array walled on author_id % 8 ended up pinch a 43x dispersed betwixt its largest and smallest partition, the largest heading for the size wherever merges extremity keeping up. Hashing the id earlier the modulus flattens it.
• The update path. ReplacingMergeTree keeps the newest full row, not the newest worth per column. Write a partial update and each section you left retired is silently blanked. Nothing errors, and you find retired overmuch later.
• Duplicate control. Re-collecting the aforesaid creator is normal and the retention costs of that compounds quietly. One array was carrying 3 times the rows it needed earlier thing made it visible, and rebuilding it returned respective terabytes.
• The database's ain logs. ClickHouse writes text_log and trace_log by default pinch nary retention. They reached 243 GB connected 1 lawsuit and took a 7 TB measurement to full, which stops writes for everything sharing it. A TTL connected the strategy tables is not optional astatine this scale.

How it goes

1. You show maine what you are collecting. Creators successful a niche, sounds connected a schedule, comments connected a group of videos. This determines the proxy tier and the excavation size, truthful it is worthy being concrete.
2. You supply a server and a proxy subscription. The collection furniture is ray and runs comfortably connected 2 cores. If you are storing what you pull, the database is the portion that needs existent disk and existent memory, and sizing it is portion of measurement 1 alternatively than a astonishment later.
3. I deploy and tune it. Usually the aforesaid week. You get the running instance, the repository access, and the reasoning down each mounting alternatively than conscionable the settings.
4. We tally the integration trial together. You watch 24 endpoints come backmost pinch unrecorded information connected your ain hardware earlier you see it delivered.

Questions

Do I request TikTok accounts?

No. There is nary login anywhere. Every instrumentality is an anonymous app instal TikTok issued credentials to. Nothing to get banned, nary credentials to rotate, nary 2FA.

Why won't it activity without a proxy? Is it legal?

It is against TikTok's position of service. It is sold for investigation and acquisition use.