Meta’s Muse product is Meta’s consumer AI solution. It’s an agent that takes the form of a virtual device assigned only to you in the cloud that works on the tasks you assign it whether you are there or not. You can reach it in the app on desktop, mobile, and web, and in other ways, like texting it through WhatsApp. It has its own computer running Linux and its own browser. It can install software, addons, and plugins, and get on with its work while you are doing other things.
I have been using Muse for about a month now and wanted to share my experience as it saves me real time. it saves me time because I spent the month training it, correcting it, and setting limits it is not allowed to cross.
It doesn’t do things instantly but does them in the background. It is a stream of interactions like you would have with a person and not a series of projects, folders, chats or anything else. You can just text to it that way, asking it to do multiple things at once and it hands that off to another task so you can continue chatting with it and when it finishes a task it comes back with a response. I will admit that I named mine Jarvis.
So lets dig into how I’ve been using Muse so far:
Reading I No Longer Do
The biggest win is high-volume reading. Listservs, forums, group feeds: I do not read them any more, Muse does, and hands me the substance with the useless posts [such as off topic posts, back and forth arguing, posts promoting another group] filtered out. In 1 week that was 294 listserv messages in 75 threads, plus 14 Facebook assistive technology groups checked the same way. Getting summaries of conversations, being able to dig deeper, getting links and information quickly. Getting lists of apps discussed and people’s impressions, getting to the point without the fluff. Responding to posts but having Muse post them. Never having to navigate the UI of those forums, groups or chains of emails. Anyone who has waded through groups.io or Facebook with a screen reader knows exactly how much time that is.
Projects I Rely On
Muse keeps up with the open source projects I depend on, and contributes to them when I tell it to. My watched repositories notify on the web only, so there is no email flood. When I ask what has changed, it sweeps notifications, commits, issues, and pull requests since the last check. Contributing runs the same way: it drafts from my point form notes, I approve the exact wording, it posts. To give 1 example, I filed an issue and feature request against BackTalk, the screen reader I use on Android, asking it to prompt the user if battery optimization is on, because without this, on some devices backtalk would be turned off by aggressive battery optimization management. It took my bullets and drafted, posted on github and made sure it went through. The developer shipped it in a build the same day. I have it keep track of comments and updates on the issues I submit and summarise updates that come out. It is extremely efficient when looking for a download link, a question on github or item I am interested in.
Converting Content
Converting has turned out to be 1 of the most useful things it does. I bought an audiobook from a site, had it downloaded, converted to MP3, and made available to me. Documents, and images of documents, come back as readable text through OCR, and files get converted into accessible formats my tools actually handle. Content locked in a format I cannot use comes back in a format I can.
content presented the way I want it. Before Muse, I used Google for a morning summary and took what it gave me. This 1 lands in my email at 7:15 AM, built to my spec: my calendar, and today’s weather only, in a few short lines, and news in it as well, ordered Ottawa 1st, as plain text and headings with no formatting junk.
Groceries and Sales
I’ve used it to figure out recipes based on family food preferences. I’ve used it to build grocery lists. When I am figuring out groceries, , Muse reads the weekly grocery flyers, which Superstore, FreshCo, and Costco publish as images, matches the deals against my grocery list, and tells me which store wins item by item.
Research
I’ve been researching travel destinations, hotels, flights, all inclusive places. I have a regular check going against my criteria.
One challenge has been getting information that isn’t written down anywhere but that sighted people would figure out by looking at photos.
As an example, now when I ask Muse about a resort pool and the answer starts with the depth, then a generalisation of the size, then the rest. Pool depths are almost never published anywhere. When the official figure does not exist, Muse estimates the depth from general information and images of the pool, and labels it as an estimate. That sounds small. It is not. A sighted traveller glances at a photo and judges the pool for themselves. I cannot, and the number is not posted anywhere for me to find another way. Anything that lives only in an image works like this: if the information is not in text, a blind person would not have traditionally had access to it.
Now I have all kinds of rules that when I ask for something result in outputs like this, without me having to say it explicitly over and over. I just ask how deep is the pool? Describe it and Muse figures it out as best it can.
Home Automation by Text
Some smart home apps are not accessible, slow to navigate, awkward or a combination of all of these when using a screen reader. Managing multiple apps for home automation is time consuming. So, I run them through Muse instead, as a text conversation, in a format I set. The thermostat is a great example of this: The app is quasi-usable with a screen reader, changes often, doesn’t have labelled buttons in some areas and you have to navigate this to do anything in the app. I read and set my Ecobee through Muse , I told it how I want things formatted, such as when I ask for temperature to give me a list of sensors, named by the floor they are on and the readings come back as a list of floor/name and temperature, then a list of which rooms are occupied. I can tell it to change the temperature or do other things without leaving the chat. I’ve extended this to other smart home functions like my various different brand smart lights. Want the lights in the basement on, the temperature a specific way for a period of time and it to check periodically for the next few hours to see if the upper floor has reached the desired temperature, that can all be done in 1 statement.
The Rules Make It Work
None of the above worked out of the box. The output rules came 1st, because an agent you cannot read is useless. Every answer starts with a level 1 heading and descends in order, so I can navigate through the chat, the sub topics and the results by heading instead of listening to a wall of text. It knows No tables, no widgets, nothing that requires sight. No filler, in anything. Suggestions come with at least 5 options, 10 when that many exist, and comparisons give the bottom line 1st. Sources appear only when I ask, or when I need the link to act on the answer.
2 rules protect my time and my accounts. Nothing gets sent, posted, or bought without the exact text in front of me and my approval on it. And there is no unprompted work: scheduled checks stay silent unless something is genuinely new, because background work spends my usage allowance.
The last rule covers failure. When a lookup fails, or comes back partial, Muse tries another route before reporting blocked. I set most of these rules after something annoyed me, including the household preferences and food ones [no mushrooms for anyone, no cucumbers, beans, or fish for me, who in the family dislikes what, 1 restaurant chain that never gets recommended]. Stated once, held without being re-asked. That is the difference between a tool and a toy.
How It Improves
It did not improve on its own. I gave it feedback throughout over a long period of time. I spent a lot of initial time correcting it after different tasks. Not that it got things wrong, more about how to present things better. The corrections stick. My forum summaries included useless posts until I said so once, on October 16th, and no summary has included them since. The weekly assistive technology digest kept including categories I do not care about until I narrowed it in October: products and research only, 5 to 10 items, and a product showing up at a new retailer does not count as news. Those rules now govern every edition.
However; none of this makes it smart. It makes it trained, by me, on my rules. The improvement is real, but I am the one doing the improving.
What It Cannot Do
So far, all the core items work great. Given it’s always through text chat and the activities occur on a virtual device in the cloud it’s been a really good experience. On rare occasions muse hits a bump and wants you to access the remote device. A sighted person would access a virtual connection to the device and manually do the task in the virtual window on the virtual meta device. This comes in the form of an image in a browser window However; in my case If it needs me to take over the browser myself (a sign-in, a challenge), the live browser window has no screen reader access. No Orca, no speech. This has rarely been needed and when it has, it’s a 1 and done thing for an initial step in giving it access to something. Taking over the browser is not a core feature, and I rarely need it. However; it is the 1 place where I cannot be independent with it, and I filed that feedback with the Muse team on October 5th. Funny enough, some sites do block Muse and that has been more problematic than anything else. For example, Amazon wants their pound of flesh and have blocked muse. So far though, that has been the only site it has happened. Muse has even gotten around captchas when doing tasks for me.
What I Will Not Hand Over
Access stays deliberate. I have not given it my main Google account, and I am still not sure I am comfortable doing that. The Gmail account it can read is 1 I use only for subscribing to listservs and forums, and there it is extremely useful. It sends my briefings to my main email address without having any access to that inbox. No bank information. No financial accounts. It is capable of paying for things, and I do not trust it enough to let it. I have not had to directly give it my Passwords as they live in a secure vault [e.g. Ecobee, Audible, dlvr.it]. I never type a password into chat, and Muse never sees the password itself; the browser fills it in at the moment of use, with my approval. Accounts that sign in another way stay that way: my GitHub account signs in through Google, so there is no GitHub password to hand over in the 1st place. When a saved password does not work, Muse tries once and stops. No retries, and no reset started without me saying so.
I have been cautious about how much I give it from the start, and that line has not moved in a month.
A Starting Prompt for Other Blind Users
If you try an agent like this, do not start with tasks. Start with rules. Something like this:
“I am blind and I use a screen reader. Structure every answer with headings, starting with a level 1 heading. Use lists, never tables. Never give me information that only works visually. Be concise and direct, with no filler. Remember every preference I state and apply it without being asked again. Before you send a message, post, or buy anything for me, show me the exact text or order and wait for my approval. If a lookup fails or only partly works, try another way before you tell me it cannot be done, and tell me plainly what you verified and what you did not. Do not do research or background work I did not ask for.”
You will still have to correct it. However; corrections that stick are the whole point.
Verdict
For a blind user willing to spend the 1st month setting rules and correcting output, this is a genuine time saver. It does linear reading and comparison work at machine speed, and it reports back in a form a screen reader handles well.
Now it’s your turn to share, what has your experience been, what prompts have been useful, what have you been using Muse for or other agents for and where have things succeeded and failed?