This is a brilliant use case of AI. I have been trying to get AI to function as a personal assistant, and the self-coding using AI to build the thing yourself wasn’t something that I was ready to jump into. I love the “new hire” framework, and am interested in seeing a “how to” to build one myself.
Thanks for this very exciting and inspiring post! Well written.
Now I'm curious. With that full file system, email, network etc. access: did you take precautions against prompt injections? As you know, websites can trick AI agents into leaking sensitive information, just by visiting them.
Likewise, did you take precautions against destructive file system operations? Recently, Metas Director of AI Safety had her AI agent accidentally delete her whole inbox, and some people have already seen OpenClaw deleting their whole files.
You should try connecting Lobster to your home devices such as lights, sockets, thermostats, motion sensors, etc. and let it observe your daily habits. Most of these devices publish their data via MQTT protocol; so you can set up an MQTT broker (I used Mosquitto) and have Lobster simply listen to that stream and aggregate the information.
I tried this over the Christmas period and the results were impressive. Armando, my LLM helper, quickly picked up on my family’s routines and was able to suggest meaningful adjustments.
Good ideas, I'm in the middle of configuring mine so this helps how you were looking at it. Wasn't ready quite yet to give it access to my cal or outlook. But after reading about how you did it makes me a little more confident in wanting to maybe give it access. We'll see :D
Great article, thank you for sharing! Especially liked the “new hire” mentality, and the agent-to-agent system v. Excessive prompt tuning on access management.
It crossed my mind. I use Apple Home and not HomeAssistant so not as easy.
This is a brilliant use case of AI. I have been trying to get AI to function as a personal assistant, and the self-coding using AI to build the thing yourself wasn’t something that I was ready to jump into. I love the “new hire” framework, and am interested in seeing a “how to” to build one myself.
Thanks for this very exciting and inspiring post! Well written.
Now I'm curious. With that full file system, email, network etc. access: did you take precautions against prompt injections? As you know, websites can trick AI agents into leaking sensitive information, just by visiting them.
Likewise, did you take precautions against destructive file system operations? Recently, Metas Director of AI Safety had her AI agent accidentally delete her whole inbox, and some people have already seen OpenClaw deleting their whole files.
Thanks in advance!
My Playbook talks about the steps I took to protect against prompt injection from the sources that my Lobster can talk to.
Lobster cannot delete my whole inbox as my Fastmail MCP server has delegate access support so I can restrict what Lobster can do.
The Mac that Lobster is on is backed up using Time Machine and there are no files that that do not belong to lobster
The shared family use case is a very bright idea, especially for planning budget friendly weekly trips
This is awesome been hearing all the chatter and hype, but concerned about security, a lot of great tips here
If only I wasn’t knee deep in developing a new platform .
Definitely intrigued
👍🏻
You should try connecting Lobster to your home devices such as lights, sockets, thermostats, motion sensors, etc. and let it observe your daily habits. Most of these devices publish their data via MQTT protocol; so you can set up an MQTT broker (I used Mosquitto) and have Lobster simply listen to that stream and aggregate the information.
I tried this over the Christmas period and the results were impressive. Armando, my LLM helper, quickly picked up on my family’s routines and was able to suggest meaningful adjustments.
Good ideas, I'm in the middle of configuring mine so this helps how you were looking at it. Wasn't ready quite yet to give it access to my cal or outlook. But after reading about how you did it makes me a little more confident in wanting to maybe give it access. We'll see :D
Great article, thank you for sharing! Especially liked the “new hire” mentality, and the agent-to-agent system v. Excessive prompt tuning on access management.