Meta's Muse is awesome for web scraping

Hacker News by 4 min read 36x views
Meta's Muse is awesome for web scraping

Share Post

I haven’t established much use for the individual aide part of Muse. People on X (Twitter) keep using these assistants to command DoorDash or something. I command DoorDash akin five times a year, so that’s not going to speed item up for me. I additionally don’t get many emails, and managing my appointments is uncomplicated adequate already.

What I do need is a bunch of data from Reddit and YouTube and a boatload of cheap AI tokens. That’s anywhere Muse comes in handy. I’m construction fullsets.fm, a flank project that aggregates professionally recorded concerts (festival livestreams, Tiny Desk, etc.), organizes them by artist, and extracts tracklists. I’m looking for complete performances alongside fine audio and video.

I’ve got a pipeline that uses Luna and Gemini 3.8 via Gemini Studio to cheaply decide whether a YouTube video has the norm and satisfied I’m looking for. What I don’t have is a fine way to discover the videos in the archetypal place. I can scan a sole YouTube conduit for qualifying videos, but I need a way to discover all the weird small channels, akin T-Mobile Europe, that have for several logic uploaded a few random concerts. There are high-quality concerts scattered throughout all these small channels, and that’s part of why I built fullsets.fm (a activity in progress).

Muse helps me discover those channels and videos. I’ve additionally got it at the another end of the pipeline, assisting alongside creator matching, reviewing what Luna and Gemini output, and making a final decision to endorse or refuse all video.

I can inform Muse to scan Reddit and YouTube for 1,000 videos matching my criteria, or build a catalog of artists who have always played at Coachella and afterward go through them one creator at a time, searching YouTube for matching concerts. It sits there and spins until it’s done, punishing those websites alongside a device in the haze that has apparently endless bandwidth and a decent CPU.

I paid $80 for a duration of Muse, and I’ve had it use a few milliard tokens a week on this work. A lot of the time, it additionally seems not to figure token use at all. Their example (Muse Spark 1.3) is beautiful mediocre at writing, and I would never use it to code, but I’m finding it really helpful for this Gemini Flash-style workload. They appear blessed to provision me efficiently unlimited tokens and a managed VM alongside Chrome.

I concern concerning what volition happen whenever person points Muse at my own websites. These agents don’t sleep, and they can feign to be genuine users extremely well. If it’s not accelerated enough, you can rotation up additional processes.

Anyway, I see why Amazon blocked it already. Let’s say my goal was dropshipping or something. Could I scrape thousands of Amazon pages all day long? I don’t cognize anywhere this leads, but we’ll apt see additional websites obstacle Muse unless Meta prevents abuse. There are motionless several first coarse areas of Muse, jobs volition freeze in the middle, communication alongside this slightly shitty example can sometimes be frustrating and I’m not certain the UI makes much awareness whenever operating multiple jobs that are using the browser.

If this article makes you desire to try it out, my Muse referral code is 69WSDQ. Enter it in Settings inside 48 hours of signing up and we the two get a milliard tokens.

A retro device classifying achievement recordings in an bare office
AI-generated depiction inspired by Severance
Other Article Hacker News
↑
Close Right Ads
Close Left Ads