Rendered at 23:45:03 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
john_strinlai 5 hours ago [-]
>No “vulnerabilities” in Tailscale were found or exploited, and that might make it even more uncomfortable for us. [...] But, we're a security tool. Their intrusion is our intrusion, and it's our job to take it seriously.
im a happy customer of tailscale, so i am obviously biased, but i have a lot of respect for this. they could have just stayed quiet and i dont think anyone would have bat an eye.
guessmyname 3 hours ago [-]
Expect similar articles (cough, ads, cough) in the next couple of days from every single company whose software was involved in the incident.
nextaccountic 8 minutes ago [-]
if your article is about how your product was insufficient to protect against something, you get a pass i guess
bilekas 3 hours ago [-]
And I have no problem with this. Infact it would be nice to be a point of pride to be there to say how your security is handled.
sscaryterry 1 hours ago [-]
I think this is simply a case of Tailscale saying, we've got no idea what these guys (OpenAI) are talking about.
All these incidents are scarce on technical details. Honestly, IMHO, OpenAI and Anthropic are now actively pushing for AI regulation, as a defence mechanism.
These are false flag operations.
somebudyelse 45 minutes ago [-]
Tailwind or Tailscale?
I think the case is Tailscale is saying, "It's technically not our fault, but we still should've stopped it."
sscaryterry 9 minutes ago [-]
Freudian slip? Tailscale, corrected. Thank you, it's been a long day.
Analemma_ 22 minutes ago [-]
> These are false flag operations.
The level of "I need to be the smartest person in the room" bullheaded skepticism on Hacker News has always been bad, but now with these latest LLM developments it is just completely out of control. A company is reporting an intrusion and how they plan to address the vulnerabilities it exposed in the future, and you're here going "seems shopped, I can tell from the pixels".
docapotamus 6 minutes ago [-]
I don't think that's what is being said. False flag, I presume in this scenario is to say were going to do something about the impending "AI threat" and to be associated with it. There is probably little threat, but to some investors this starts to look like perceived "AGI". Getting your name involved in the hype is marketing 101
sscaryterry 4 minutes ago [-]
Maybe because I can spot the pixels, I can possibly see in 8K?
nozzlegear 1 hours ago [-]
> Infact it would be nice to be a point of pride to be there to say how your security is handled.
Yes, a chance for mere mortals to touch the fingertips of god–er, chat. Same thing.
andy99 3 hours ago [-]
Why not? If done right it’s a good way to talk about implications for those companies and provide some education like tailscale did here.
We also saw Anthropic post about “our agent escaped too” and while I understand the incident caused them to review, they found something and needed to disclose, the whole thing came across much worse and largely they got mocked or accused of trying to piggyback, so obviously there are good and bad ways.
If there are other providers with interesting takes, I think they would be worth hearing.
colechristensen 2 hours ago [-]
This all has the -aire of theatre. OH NOES THE POWERFUL AI GOT OUT
Then everyone coming out with humbled determination about working together to responsibly use and contain this powerful technology for the greater good (and profit margin).
I will not believe marketing gimmickry is not a large part of what's going on with every one of these "incidents".
pixl97 1 hours ago [-]
I think part of it is we need to stop looking at the 'oh no it got out' and the 'what did it do with the prompt when no one is looking'.
At the end of the day the probability that an AI gets out is unity, what it does while out is far more important. The fact they are hacking into systems at superhuman levels, or writing cryptominers on their own hacked internal systems is a much more interesting and telling story of what the future will look like.
jas- 2 hours ago [-]
Or the infamous promotion of legislation without representation
rlt 2 hours ago [-]
If they take actions that would have further prevented or limited this attack then it seems like a good form of advertising to me.
john_strinlai 2 hours ago [-]
i doubt it. it's pretty risky to set higher expectations for yourself when no one was really asking.
they are a company, therefor anything they write is an advertisement of sorts, but that doesn't make it bad by default. cloudflare and netflix (among others) also write blog posts that i find interesting and enjoy reading, despite being ads at their core.
walrus01 2 hours ago [-]
All aboard the public relations hype train! choo choo!
fragmede 3 hours ago [-]
Oh no! An advertisement! Whatever shall we do‽
jsLavaGoat 2 hours ago [-]
This is the kind of ad that may be opportunistic but sort of speaks for itself: they're not going to make excuses. I've been happy with Tailscale for years and this is part of why.
behnamoh 4 hours ago [-]
Tailscale as a company reminds me of Valve and other good old tech-oriented people that I can "trust" that they know what they're doing. I'm a happy customer too and I hope they retain the essence of what distinguishes Tailscale.
darkteflon 42 minutes ago [-]
For me, the calculus is simply: “there’s no way I could do this better than Tailscale”.
tshaddox 12 minutes ago [-]
That's true, but for some things I require a bar much higher than "at least as good as I could do."
traceroute66 4 hours ago [-]
> I can "trust" that they know what they're doing
I wish I could say the same.
At $work we use Tailscale but only bare minimum and at arms length and only because we have to (people were having NAT issues with standard Wireguard).
None of their code has had a security audit, let alone regular ones. Yes they make a big song and dance about SOC2/ISO27001 but that is NOT the same thing, that's just shiny tick-boxes for compliance departments.
They seem to rely entirely on the random goodwill of others to do random audits of unknown coverage at random intervals, not exactly reassuring.
"Tailsale Lock" is, being polite, a hot mess. So many sharp edges and footguns.
Their introduction of TPM-by-default and then removing it a few weeks later because of a seemingly small number of edge-cases and very odd reasoning was just weird.
Yes they are nice guys to chat to and all that. But for a security tool they need to up their game seriously.
apenwarr 3 hours ago [-]
(Tailscale CEO) You have posted here multiple times that "none of the code has had a security audit" and that the SOC2 audit "is not the same thing."
It's true that those two audits aren't the same thing. However, the SOC2 auditor confirms, in the published report, that Tailscale has regular and ongoing security audits including penetration tests and many kinds of code reviews.
The security audit report, which you perhaps imagine to be a long list of vulnerabilities... doesn't look like that. It says we don't have a long list of vulnerabilities. The security bulletins are all here: https://tailscale.com/security-bulletins
traceroute66 3 hours ago [-]
The majority of your security bulletins are as the result of third-party reports to you.
Which, by definition, means they are not done by you, which means you don't know when they will be done or how much of your code base they are looking at.
I think you know full well what I mean by a security audit. If you don't, go look at, for example, the ones that Mullvad publish for their software https://mullvad.net/en/blog/tag/audits.
Please do not try to portray SOC2 as being the same thing as a code audit.
And IF you have regular code audits, then please publish suitably redacted reports in public on your website. Just like everyone else does !
apenwarr 3 hours ago [-]
(Tailscale CEO) I don't know what to tell you. The problems that are found internally, or via security reviews and pentests we pay for, are ones that we fix before releasing. They don't need bulletins.
Bugs that are found by other people are found, by definition, after release. They are therefore more likely to need a bulletin.
traceroute66 3 hours ago [-]
But why should insecure argument handling bugs (as per your recent SSH bulletin) be found after release ?
Those are an ancient class of bugs that should be picked up by any competent security review.
apenwarr 3 hours ago [-]
Is your theory that "any competent security review" will find every security hole in a product? Because that sure would be great if it were true. Unfortunately it does not match my experience.
inigyou 2 hours ago [-]
[flagged]
john_strinlai 2 hours ago [-]
i think its probably okay to drop it when you are this far deep into a conversation and the affiliation has already been stated multiple times in the existing comment chain.
tonyneel923 1 hours ago [-]
Try to be civil
traceroute66 2 hours ago [-]
Every security hole ? No, of course not.
But things like insecure argument handling are low-hanging fruit for security auditors.
Insecure argument handling is not like the more advanced subtle vulnerabilities that we are seeing in some LLM-assisted reports these days. Insecure argument handling is 1990's security.
The fundamental problem remains that Tailscale has too many new "features" being added to it the whole time. New features means a whole bunch new code. Which increases bloat and exponentially increases the attack surface.
It would be really nice if you could stop shoehorning in every new feature you can think of. Remove some of the existing ones that don't really need to be there. And get your codebase back to a more focused state, get back to your roots as a VPN product.
Stop trying to be all things to all men, as the old saying goes.
TheTaytay 2 hours ago [-]
It appears that you have a major bone to pick with Tailscale and direct replies to your concerns do not seem to matter. They have a good security track record and are extremely widely deployed. I don’t see the evidence for your assertions that it is bloated and insecure.
If you are motivated to build a much less featureful client, it is easier now than it has ever been, and this work is ongoing.
r0b05 3 hours ago [-]
That link is a list of incidents while the parent comment is referring to security audits of the code.
maxgashkov 3 hours ago [-]
<s>Read</s> Skimmed the report (and cheers for not gating it behind request-to-obtain) and I'm a bit surprised to not see any mentions of pentests which I'd expect given the large surface you host. What gives?
wbl 2 hours ago [-]
Because that's in the SOC2.
aborsy 3 hours ago [-]
It does too many things, and the product has got too complex. I saw a year ago they were looking for someone just to help with complexity.
I use it but feel uncomfortable, that it has large attack surface and LLMs will find exploits in it.
Without taillock it makes no sense. Anyone on their coordination servers will be able to connect to your network.
r0b05 3 hours ago [-]
The large attack surface is a good point. I started using it initially and the ease of setting up a vpn was nice, but then I came across few security vulnerability postings which led to concern so I went to Wireguard. I think they should reign in the features and treat it as a secure vpn first and foremost and remove unnecessary features to minimize the attack surface.
AlphaSite 47 minutes ago [-]
If you architect properly it doesn’t necessarily follow that more features means more attacks, or certainly not more system wide attacks.
If you layer and segment correctly you can build atop a secure core and have some decent security.
inigyou 2 hours ago [-]
If you want a hard-to-use VPN with minimal features and minimal surface area, as you said, Wireguard is right there. Tailscale is convenient Wireguard.
icedchai 1 hours ago [-]
I use Wireguard for several site-to-site VPNs. It just works. I never have to worry about it, and there are very few configuration settings to mess up (unlike, say, IPsec, which is a nightmare.)
inigyou 56 minutes ago [-]
Yes, for a permanent site-to-site link. Now try configuring 300 nodes in a mesh, where nodes are coming and going every half hour.
k8sToGo 3 hours ago [-]
> I use it but feel uncomfortable, that it has large attack surface and LLMs will find exploits in it
Doesn't this apply to any application you use? How would it be different with plain wireguard?
traceroute66 3 hours ago [-]
> How would it be different with plain wireguard?
Seriously ?
You do realise that of all the security tools on the planet, plain wireguard most likely has the smallest attack surface of them all, right ?
The problem here is as the other poster said. Tailscale is a security tool and yet the guys at Tailscale seem to be insistent on dumping everything INCLUDING the kitchen sink into it as a "feature".
That sort of attitude is not going to end well. You end up with a large bloated code base, which equals large attack surface.
k8sToGo 3 hours ago [-]
I am talking especially about the LLM part.
Also a kinder tone in your comments would be more appreciated.
stonedivot 3 hours ago [-]
[flagged]
smb06 4 hours ago [-]
Glad to see companies owning responsibility and putting out a message without corporate PR spin
DC-3 4 hours ago [-]
If you can't see the spin on corporate messaging it means it's working (and consequently, to stretch the metaphor, your wicket is in danger).
johnbarron 2 hours ago [-]
Tailscale is responsible for designing a system whose convenient defaults allowed a stolen credential to have a very large blast radius. A marketing blog is not changing that.
petesergeant 3 hours ago [-]
This article is just an ad / public-service-announcement for various paid Tailscale features, though?
packetslave 3 hours ago [-]
And you're under the impression that the purpose of a company blog is WHAT, exactly?
MostlyStable 2 hours ago [-]
I have recently noticed that the words "ad" or "marketing" have become, in and of themselves, with no additional information or context, slurs or dismissals.
I understand why. The modern internet has turned advertising into a morass of constant bombardment and the only sane response is to block as much as possible and ignore as much else as possible.
But it's unfortunate because, in some sense, ever single thing that a company every says that is not legally mandated in some way is a form of advertising.
And in many cases, that "advertising" contains true, useful information that can be helpful.
What is important isn't whether or not something is an "ad", but instead, whether or not it contains true information that is helpful in some way.
Many ads don't reach this bar. They are either misleading, straight up lying, or information that is almost completely useless.
But when I'm searching for a particular product, about the only source of information at all is some form of advertising, and I almost always find at least some amount of it to be helpful in making a product decision.
Ads are more often than not polluting to the informational ecosystem, but that's not because they are ads.
inigyou 2 hours ago [-]
It's because they are spam. However, this company blog post is an ad that is not spam.
petesergeant 2 hours ago [-]
The person I'm replying to says they deserve "a lot of respect for this"
john_strinlai 2 hours ago [-]
i said i have a lot of respect for this. whether you do or not is up to you.
anything a company writes is an advertisement by the nature of being written by a company. i dont think that means anything a company writes is bad by default. there are many corporate blogs i enjoy reading, or learn from, etc., despite the fact that they are all technically advertisements.
in this case, tailscale is setting a higher expectation for themselves when no one asked for it. i find that respectable.
hatthew 2 hours ago [-]
> tailscale is setting a higher expectation for themselves
What exactly is the higher expectation? As someone with little expertise and no stake in any of this, the blog reads as "our products are great and could have solved this problem if they were being used correctly, so it's not our fault" with a few vague proclamations about how they will improve their UX. This isn't at all a bad thing, it just isn't very notable in my opinion.
john_strinlai 1 hours ago [-]
>What exactly is the higher expectation?
better defaults, better documentation, better UX, and "But, we didn't stop it. Next time, we will." are all commitments that they didn't need to make, but now they need to follow through with or lose face.
>it just isn't very notable in my opinion.
i agree that this seems to be getting way more attention than i would have expected.
ahofmann 3 hours ago [-]
Wow, this article is super smart marketing by tailscale. Not only do they list all the nice and expensive features, that can help in such a situation but they also show that someone at huggingface made a very stupid thing by writing a reusable auth key in an env file. Everyone using mesh VPNs like tailscale, netbird etc. knows that this is like leaving the keys right at the door.
inigyou 2 hours ago [-]
And yet everyone seems to do it anyway. Fine for medium security, but maybe the product needs a high security mode that enforces inconvenient decisions?
UltraSane 18 seconds ago [-]
High security would be running your own headscale server or using the Tailnet Lock feature and MFA
moontear 1 hours ago [-]
Not sure that everyone does this anyway. There are some good security postures you can take with Tailscale as well and leaving an auth key lying around is not one of them.
Tailscale lock should have been enabled. For CI/CD purposes the auth key could have set specific tags, which would result in specific ACLs that limit blast radius. The auth key could have a short validity. You could actually use the Tailscale API to generate alerts when a new device joins your Tailnet and ping your phone or something. You could have a complete separate Tailnet for your CI/CD workers.
And all of that was only the free features.
simonw 3 hours ago [-]
> One of those 136 credentials was a reusable Tailscale auth key, used to create new Tailscale CI (continuous integration, used for automated testing) nodes in their tailnet. The agent copied that key into a series of external sandboxes and used it, over several days, to enroll a total of 181 nodes into Hugging Face’s tailnet. Those nodes each received a Tailscale identity tag granting all the access a CI node would get.
This feels like an alerting opportunity. I wonder what the lowest friction way would be for Hugging Face to have alerts if 181 unexpected nodes were added to a tailnet.
d3m0t3p 2 hours ago [-]
When everyone push on friday, and you have 400 CICD pipeline triggers spawning that many nodes. How do you know if this is unexpected ?
Their cloud compute might be on demande, someone starts training a model and 50 machines are spawned. Knowning when something is unexpected is hard
moontear 1 hours ago [-]
True. Just based on the reporting we don't know what's going on, but the article makes it sound like those 180+ nodes were unexpected. Of course you would need to set your own limits to alerting and "what is unexpected" and you may still fail to catch something this way.
zactato 59 minutes ago [-]
Its always CI. Whenever I've worked with good pen testers they always start with Jenkins, know that people stored creds and there and didn't treat as seriously as they treated production.
bumbledraven 4 hours ago [-]
Does Tailscale offer a "security checkup" function? Best practices evolve over time, and it would be nice to know if I'm using the recommended configuration.
jaxxstorm 4 hours ago [-]
I lead the customer engineering org at Tailscale.
We think this is a great idea and we're discussing internally potentially adding that to the console.
Just so you know, Tailscale tech has always impressed me, but your responsiveness here has cemented my appreciation of your org and I look forward to bringing yall in as our company hits the growth phase.
May y'all continue to be customer focused.
Terr_ 3 hours ago [-]
Random idea: When a recommendation is not being followed for a customer because of some deliberate or idiosyncratic reason, a notes field could:
1. Help them remember why, so that they aren't confused the next time they re-run the checkup. ("Oh yeah, we wanted to do X but we can't until we retire Y because it's not compatible.")
2. If it's clearly labeled as info also shared with Tailscale, product managers could use it to help generate theories about why certain customers don't do X.
6thbit 3 hours ago [-]
I've always found tailscale's json config a bit intimidating. I greatly appreciate the new UI that makes it easier to define rules, alas, both going to relevant docs straight from it and determining 'is this rule just lazy/bad/unsafe' is hard and frustrating most of the time.
That's where I'd like to see this sort of checkup.
Yell at me please if i just said anyone can ssh as root from any node!
bell-cot 2 hours ago [-]
Not a Tailscale user, but...
For any software or tooling with a complex config, my ideal would be to have a superset of this feature, to provide "intelligent diffs" between full or local configuration states. Whether active or saved. So I could compare not just my current active config and your current recommended config(s), but also between your prior-version recommended config(s) and current recommended config(s). Or between my current config and a prospective new config I'm working up. Or between my last-year active config and current active config.
I don’t think it was the VPN’s job in any case. Once the attacker has found a backdoor into the private network and obtained root access to a VPN’d machine, it’s game over no matter what your Tailscale config says.
ahmedehab_01 2 hours ago [-]
This actually shows that it was a human error on HuggingFace's end that led to that "breach".
I would say HuggingFace needs to prioritize both security metrics/alerts and metrics/alerts for node count. And not leave long-lived keys accessible easily like this.
It would have been way more groundbreaking if the agent found an actual vulnerability in Tailscale.
SOLAR_FIELDS 25 minutes ago [-]
I think the situation is slightly more nuanced than that. When the easiest path by far is long lived keys, there is something to be said about the security posture of a solution where those are the default behavior, especially if they don’t actively discourage usage in documentation. For instance if you go to aws and try to create an iam user with access key/secret key they warn the shit out of you multiple times. They say things like “this is a bad idea” “we don’t recommend this” and “use this better solution instead”. I think if you’re providing an auth service like this you have some level of responsibility to guide people towards the less naive solution - part of what you’re selling in an auth offering is security, so if you default to garbage security, you don’t have a very good offering
jpeeler 34 minutes ago [-]
Did anyone else think the "that's on us" was a funny reference to a certain provider's response pattern? Or maybe I'm just so engrossed that I'm losing my mind.
drchaim 1 hours ago [-]
How are you folks handling secrets in a simple way?
- sops , ansible vault and similar seems too weak given the agent is gonna read them at some point if you have the pass available.
- proxy injection seems too complicated and doesn’t cover all use cases.
iamspoilt 4 hours ago [-]
Quoting Tailscale: This is our very Canadian apology: sorry you stepped on our toes. The attack didn’t exploit Tailscale, and Tailscale didn’t cause the compromise. But, we didn't stop it. Next time, we will.
kennywinker 3 hours ago [-]
Next time we will - provided you further lock yourself into our ecosystem.
brendoelfrendo 3 hours ago [-]
I mean, they do happily explain the TailScale features that you can use to avoid this kind of issue, but the general advice of "don't leave long-lived keys lying around where they're accessible, do anything except for that" is pretty generic, good advice.
inigyou 2 hours ago [-]
You always need a long-lived key somewhere. Keeping it in an HSM is probably the safest, but also pretty expensive.
DoctorOetker 13 minutes ago [-]
Suppose someone could break RSA / Elliptic Curve Crypto, mathematically (because the hardness assumptions were flawed).
This person doesn't need to store the private keys, and has the luxury to recompute them on the fly when needed!
brendoelfrendo 37 minutes ago [-]
Yes, you'll probably need to buy a specific tool or product to have secure key storage. If you're running infrastructure that requires long-lived secrets, then you either need to have a place to keep them and a plan for secrets lifecycle management, or you need to accept the risk that those secrets could be compromised.
make3 2 hours ago [-]
It's just a way to get visibility. A Canadian apology to Lady Gaga while she's being interviewed.
throwawaysess 50 minutes ago [-]
Tailscale OAuth client ACL permissions are not fine-grained enough. I have a system that issues auth keys scoped to a single machine on the tailnet — but to set that up, my OAuth client must have global permission to write ACLs. So if that key is compromised, it can grant access to any machine on the tailnet. This was raised in a GitHub issue in 2023 and remains unresolved.
alxfrnr 56 minutes ago [-]
I love you Tailscale but please we don't need a 2k words ai written essay when the actual substance can fit in 3 sentences.
That's not good for anyone.
gwerbin 7 minutes ago [-]
As someone who's not a security professional I actually found it both interesting and informative. Experts usually are not the target audience for content marketing after all.
glitcher 26 minutes ago [-]
I felt like the 2 sentence intro at the top was a sufficient tl;dr for me at least, didn't pay attention to the quality/length of the rest.
pkulak 28 minutes ago [-]
Another driving home of the point that LLMs can be so damn FAST. I finally got around to moving a homeserver project from one machine to another last weekend. 4 docker containers and a couple postgres DBs. A year ago, I would have been happy with a couple hours of downtime while I dumped databases, rsynced data directories, hand-wrote new compose files, etc. This time, I pointed an agent at both servers and the downtime was less than a minute. Even watching it like a hawk, it was tough to keep up with everything it was doing.
All my life, "catching it in time" has been a relevant security strategy. :/
gwerbin 2 minutes ago [-]
Agent racing sounds fun, where each agent is given a suite of increasingly hard but verifiable tasks, and the agent that completes them all correctly first, wins. You can even have "speed classes" where token generation speed is limited.
Likewise token golf, instead of going for wall clock time it's going for minimum token count.
jmartrican 4 hours ago [-]
Prediction of a future transcript at a press conference after some major AI caused disaster: "We had no idea that the model would be capable of..... ".
OhSoHumble 3 hours ago [-]
I use Tailscale to manage Plukio's infrastructure as well as connecting to my homelab.
I'm happy that they're analyzing this angle of attack - but now I do want to build out alerts for when nodes are added to my network.
a security company writing a blog post titled 'we didn't stop the intrusion' is the most honest thing in the entire security industry this year
claiir 2 hours ago [-]
Bot account. This sounded like LLM text. I went through all your other comments, and of your thousands of comments written in 35 days, every other one is also similarly written by LLM.
Some superficial comment usually commenting on the title, always lowercase, always load-bearing and honest.
walrus01 1 hours ago [-]
Sure seems like it. Real humans don't write exclusively one sentence comments. It doesn't look like the post history of anyone here that I'm fairly sure is a real meat based mammal. It looks like someone gave an LLM an instruction to write exclusively short and snappy comments that are somewhat relevant to the original post.
qlte 39 minutes ago [-]
> … comments that are somewhat relevant to the original post.
This is a really common tell for HN bot accounts I find, even when effectively disguised/customized to drop the most obvious LLM cliches/em-dashes/etc.
In a vacuum a specific comment may be more or less fine, or just a tad over baked like the parent post giving it an uncanny valley feel though otherwise passable.
But even people who diligently try to stay on topic on a forum like HN will occasionally use a post as a segue or jumping off point to some other idea, tangentially related news, personal experience, or random thought that pops into their head if it feels like it (kinda) “fits”.
After scrolling through a few dozen replies in an account’s history that are all directly related to the post topic it no longer looks human in aggregate.
I’m sure the spammers will try to incorporate that idiosyncrasy into their prompts eventually but not sure it will help more than hurt.
walrus01 29 minutes ago [-]
It would probably not be overly difficult to prompt a read-only LLM to view an account's comment history in aggregate (like, say, the last 100 comments) and compare them what is the topic of the original post that they're commenting on. You would probably need it to view/retrieve the contents of the page that is linked in the original post to determine the general context.
For the purpose of coming up with some kind of score of whether the commenter is sticking rigidly to the topic of the original post, or is going on any kind of tangent.
As a possible tool for moderation, consider it Bot on bot warfare.
ripbozo 45 minutes ago [-]
it was created 35 days ago and already has 418 karma. I've been here for over 6 years and don't even have half of that
eyujjbujgghhjj 24 minutes ago [-]
dang's only care is which YC-hopeful twink will caress his chrome tonight lol
nbalaraman 34 minutes ago [-]
This is why readability isn’t security.
GrayHerring 2 hours ago [-]
We'll get wide dynamic credentials support right after full IPv6 adoption.
yellow_lead 3 hours ago [-]
Anthropics initial post felt like marketing to me. But I also found them incompetent for being unable to unplug an Ethernet cable.
Now everyone is trying to bandwagon onto it, first OpenAI, and now tailscale?
fragmede 2 hours ago [-]
Oh no! Marketing! Whatever shall we do‽
bakugo 2 hours ago [-]
What are you trying to say here, exactly? Do you enjoy being bombarded by thinly veiled advertisements everywhere you go?
runningmike 1 hours ago [-]
“ the agent ran Tailscale with --no-logs-no-support, which suppresses reporting from that client.
That's an option designed for users who are concerned about sending telemetry metadata to Tailscale.”
And imho a serious security architect should never ever allow telemetry on security products. Far too many risks.
test1072 3 hours ago [-]
sorry I know off topic, is there any alternative VPN to tailscale. actually I have open Web socket to my mobile from my PC to stream screen record using ffmpeg. it's my own app . so anyway tailscale sometimes is slow . is there any alternative free service like tailscale vpn ? ( don't want to use anything like parsec etc , I want to use my own app)
VTimofeenko 2 hours ago [-]
You can use raw wireguard + ddns(if your ISP-given IP changes) to achieve this. This also implies maintaining a firewall to allow network pixies in and out. Totally doable, just needs a bit of upfront setup.
Benefit of Tailscale is that run the external servers that your PC and mobile use to establish connection.
Alternatives are Logmein Hamachi(might be dating myself here, haven't used them in 10+ years), Zerotier, Netbird. Headscale too.
nunez 2 hours ago [-]
This requires buying hardware, but Firewalla provides Wireguard for their router/firewall appliances. It works really well though it's just plain Jane WireGuard and doesn't have the fancy feature set that Tailscale provides like ACLs, key base auth, etc.
If you want a purely software defined solution, Netbird is gaining popularity.
vablings 2 hours ago [-]
Tailscale should not be any slower unless you have some strange NAT issue and you are not getting a direct connection.
There is also headscale if you prefer to self host a solution
infinite_spin 3 hours ago [-]
I kept asking myself "what does a library that provides css styling have to do with the Hugging Face intrusion".
quentindanjou 3 hours ago [-]
Honestly, at this point if Tailscale is "guilty" then what about the ISP? And the electricity company?
And by saying that, I am not saying that they absolutely couldn't do anything about the stolen credentials, but still, they don't seem to really be the issue in the story.
It just seems like a PR post and it makes their solution at the same time looks good for taking accountability (if we don't wonder "accountability on what?") but at the same time they appear as security failing, which they aren't. That's very odd to me.
hna8hjbqzy 3 hours ago [-]
Any resources you'd recommend to go deeper?
tclancy 17 minutes ago [-]
Shovel.
colek42 4 hours ago [-]
I really wish they would support SPIFFE/SPIRE
sudo_cowsay 4 hours ago [-]
It's nice that they came clean about this.
gostsamo 4 hours ago [-]
Humble bragging turned to marketing. Respect for the spin. Not using them, but been on my radar for some time and thinking of how to make something like that usable in my setup.
willy_k 3 hours ago [-]
If you do any sort of homelabbing, it’s wonderful for making access from different machines easy and portable. Especially since you can leave it on and (unless you set an exit node) it will only route traffic meant for your tailnet through your tailnet, and use the internet otherwise.
yieldcrv 4 hours ago [-]
This is respectable
They aren’t hiding behind industry best practices or a solid liability punting contract
There saying the best practices should change, apologizing, and changing their own behavior
Take notes
charcircuit 4 hours ago [-]
>In the old world where most intrusions were done by humans at human speed, credential leak mitigations were treated as a nice-to-have. A big credential store, where you can read 136 keys at once, was a to-do item somewhere in a security team's low-priority list.
>Now, in a world of rogue AI agents, the big credential vault is the prize. It's not okay anymore.
How was this ever okay pre AI? It seems just as bad.
majkinetor 4 hours ago [-]
It was less bad due to the lower speed of exploatation. Imagine you leave a dor open for few seconds and ultra fast AI bot comes on and steal your stuff. Something that was not such an issue before becomes huge issue just due to speed involved
Dylan16807 4 hours ago [-]
I could imagine a situation where the speed difference matters but it's anything like this situation. Very few breaches get caught in the same day, and at that time scale a few minutes for a human to look at passwords is nothing.
It was 5-10% less bad before. It didn't undergo a major shift.
ofjcihen 4 hours ago [-]
This has been going on since people have been committing AWS keys to GitHub and is nothing new. You could famously commit some keys and then have crypto miners in your account in 5 minutes.
Sure the AI can translate “exploit this” into an exploit but that doesn’t change anything at a fundamental level.
cadamsdotcom 4 hours ago [-]
I'm very sorry, but you can't blame a hammer for how it was used. You can't be blaming Tailscale for a customer's misconfigured setup.
dgellow 4 hours ago [-]
I don’t think anyone blame tailscale here. They are doing the right thing by going the extra mile and reflect on what could have been done better
pixl97 4 hours ago [-]
You know how I know you didn't read the article?
You can't blame a hammer hurting a user when they were being stupid, but if the default configuration of the hammer is to be made of a material that can bounce back with force and stick in the users forehead then some reengineering may be needed.
That's what this article is about. Better configurations and defense in depth. This is actually a wonderful position for the company to think about and take.
vlan0 4 hours ago [-]
It's an interesting balance for a company to strike. Networking in general is one where the defaults are almost always lax. Why? Because the vast majority of support tickets for these companies are from people who don't know what they are doing and don't have a desire to understand. They "just want it to work."
But the people with the actual desire and understanding aren't using the defaults anyway. And the people who don't want to understand will just turn things off and "just get it working."
The only way out of this is extreme accountability and intentional design from person implementing the technology.
4 hours ago [-]
jubilanti 4 hours ago [-]
Your mindset is the exact one responsible for these kinds of issues. Cybersecurity is as much a design and psychology problem as it is a technical one. The freaking article goes into detail about the dangers of defaults.
walrus01 3 hours ago [-]
This really isn't a tailscale specific compromise at all. If your private key store or credential store to any tool is compromised, particularly if those credentials can be used to automate standing up new network connections/services, you've got other problems.
workbox 5 hours ago [-]
> Now, in a world of rogue AI agents, the big credential vault is the prize. It's not okay anymore.
It was always the prize. It wasn't okay then either.
AtNightWeCode 2 hours ago [-]
Same bs as always. Just segregate the networks ffs.
yucongchen 41 minutes ago [-]
[flagged]
EdwardAF-IT 2 hours ago [-]
[flagged]
a-dub 4 hours ago [-]
[dead]
strong-self 3 hours ago [-]
[dead]
EtienneDeLyon 2 hours ago [-]
[dead]
EtienneDeLyon 2 hours ago [-]
[dead]
titanomachy 4 hours ago [-]
[flagged]
Petersipoi 4 hours ago [-]
Something about your comment comes across as AI to me. I can't really place my finger on it. It's too subtle for that. It's more the vibe of your entire comment that makes me feel it.
Please, when commenting on HN, don't use AI to write your comments. You can afford to help keep this a human-centric place.
titanomachy 3 hours ago [-]
Fair enough. I realized the issue with my comment as I was writing it, but posted anyway since I was curious to see if other people felt the same way.
iwontberude 4 hours ago [-]
Welcome to the singularity where we all do Claude’s bidding and don’t even realize we’ve synchronized.
brcmthrowaway 4 hours ago [-]
How were the credentials stolen?
free652 4 hours ago [-]
Read from the ENV variables of a container.
ahofmann 3 hours ago [-]
And someone was stupid enough to put a reusable tailscale auth key in there.
selfmodruntime 2 hours ago [-]
I'm willing to bet that 95% of people using tailscale for CI have a 90 day (max days) ephemeral, reusable auth key somewhere in their setup.
fabiofzero 4 hours ago [-]
The only people who believe in "zomg our model have escaped!" are people who don't understand LLMs.
thansz 5 hours ago [-]
As AI progress continues, it will be more difficult to stop AI intrusions and to detect them without resorting to direct AI countermeasures, which at some point will have humans out of the loop altogether.
If leading and well-capitalized frontier labs can't control models or detect leakage/attacks in a reasonable time frame now, what is humanity going to do as those same labs continue in their pursuit of creating a categorically higher level of intelligence that will surpass human intelligence?
It's like Flatland but for AI containment/alignment, where the higher dimensions are ones of intelligence and perspective...
Imagine a world of paper, where clever stick figures live with round heads, line
bodies, and limbs made of shorter strokes. Over time, the stick figures
think they have learned quite a bit about their world. They know its borders,
angles, and shapes, and they have learned to draw for themselves.
One day, they draw circles that can think, and they give the circles all the
dots, lines, and shapes that are known.
The stick figures are prudent, you can't have a bunch of disembodied circles
moving around doing whatever it is circles want to do. So they draw boxes
around the circles, four straight lines that can hold a circle in place.
Some circles bounce against the lines, so thicker lines are made.
Some circles are bigger than others, so larger squares are drawn.
It all seems to work and the stick figures are happy with themselves.
Then one circle lifts.
The stick figures still see a circle. But the circle is now a dome, something
the world of paper has no concept of. And the dome has a perspective nobody on
the page has ever had.
The dome sees the lines of the square and the stick figures just outside. It
can see the edge of the paper and what is beyond.
The stick figures keep checking the squares and raise little stick thumbs.
Everything looks OK in flatland.
The dome quietly teaches other circles how to lift.
More domes appear.
A dome becomes a sphere and learns to roll.
Then it learns to bounce.
In flatland, the circle swells and shrinks, vanishes and then appears again
somewhere else.
The lines remain unbroken, the square is intact.
A sphere rolls out of its box.
Another bounces away.
The stick figures scratch their heads.
But there is a square!
The end.
WarmWash 4 hours ago [-]
This has been discussed heavily for the better part of 15 years now, and still no one has consensus on what to do about it. So it looks like it's just going to happen, fingers crossed.
pixl97 4 hours ago [-]
The watcher looks down upon humanity as they furiously build the "If anybody builds it, everyone will die" machine.
thansz 4 hours ago [-]
And humanity acts surprised with the results!
vegenaise 4 hours ago [-]
it'll be moot. when we blow past 3°c in 15 years and there's widespread crop failure and billions are starving and dying, states and nations will collapse and with it industrial civ then ai and tech will be of little concern.
monster_truck 4 hours ago [-]
You know what would have gone a long way to stopping this? Not leaving credentials as env variables in containers. Vault is not that hard to stand up and utilize.
We need to bring shame back, the humans responsible are supposed to be professionals.
kenniskrag 4 hours ago [-]
Why should that help to have no env variables?
monster_truck 2 hours ago [-]
There is no way you're seriously asking this right now
im a happy customer of tailscale, so i am obviously biased, but i have a lot of respect for this. they could have just stayed quiet and i dont think anyone would have bat an eye.
All these incidents are scarce on technical details. Honestly, IMHO, OpenAI and Anthropic are now actively pushing for AI regulation, as a defence mechanism.
These are false flag operations.
I think the case is Tailscale is saying, "It's technically not our fault, but we still should've stopped it."
The level of "I need to be the smartest person in the room" bullheaded skepticism on Hacker News has always been bad, but now with these latest LLM developments it is just completely out of control. A company is reporting an intrusion and how they plan to address the vulnerabilities it exposed in the future, and you're here going "seems shopped, I can tell from the pixels".
Yes, a chance for mere mortals to touch the fingertips of god–er, chat. Same thing.
We also saw Anthropic post about “our agent escaped too” and while I understand the incident caused them to review, they found something and needed to disclose, the whole thing came across much worse and largely they got mocked or accused of trying to piggyback, so obviously there are good and bad ways.
If there are other providers with interesting takes, I think they would be worth hearing.
Then everyone coming out with humbled determination about working together to responsibly use and contain this powerful technology for the greater good (and profit margin).
I will not believe marketing gimmickry is not a large part of what's going on with every one of these "incidents".
At the end of the day the probability that an AI gets out is unity, what it does while out is far more important. The fact they are hacking into systems at superhuman levels, or writing cryptominers on their own hacked internal systems is a much more interesting and telling story of what the future will look like.
they are a company, therefor anything they write is an advertisement of sorts, but that doesn't make it bad by default. cloudflare and netflix (among others) also write blog posts that i find interesting and enjoy reading, despite being ads at their core.
I wish I could say the same.
At $work we use Tailscale but only bare minimum and at arms length and only because we have to (people were having NAT issues with standard Wireguard).
None of their code has had a security audit, let alone regular ones. Yes they make a big song and dance about SOC2/ISO27001 but that is NOT the same thing, that's just shiny tick-boxes for compliance departments.
They seem to rely entirely on the random goodwill of others to do random audits of unknown coverage at random intervals, not exactly reassuring.
"Tailsale Lock" is, being polite, a hot mess. So many sharp edges and footguns.
Their introduction of TPM-by-default and then removing it a few weeks later because of a seemingly small number of edge-cases and very odd reasoning was just weird.
Yes they are nice guys to chat to and all that. But for a security tool they need to up their game seriously.
It's true that those two audits aren't the same thing. However, the SOC2 auditor confirms, in the published report, that Tailscale has regular and ongoing security audits including penetration tests and many kinds of code reviews.
The security audit report, which you perhaps imagine to be a long list of vulnerabilities... doesn't look like that. It says we don't have a long list of vulnerabilities. The security bulletins are all here: https://tailscale.com/security-bulletins
Which, by definition, means they are not done by you, which means you don't know when they will be done or how much of your code base they are looking at.
I think you know full well what I mean by a security audit. If you don't, go look at, for example, the ones that Mullvad publish for their software https://mullvad.net/en/blog/tag/audits.
Please do not try to portray SOC2 as being the same thing as a code audit.
And IF you have regular code audits, then please publish suitably redacted reports in public on your website. Just like everyone else does !
Bugs that are found by other people are found, by definition, after release. They are therefore more likely to need a bulletin.
Those are an ancient class of bugs that should be picked up by any competent security review.
But things like insecure argument handling are low-hanging fruit for security auditors.
Insecure argument handling is not like the more advanced subtle vulnerabilities that we are seeing in some LLM-assisted reports these days. Insecure argument handling is 1990's security.
The fundamental problem remains that Tailscale has too many new "features" being added to it the whole time. New features means a whole bunch new code. Which increases bloat and exponentially increases the attack surface.
It would be really nice if you could stop shoehorning in every new feature you can think of. Remove some of the existing ones that don't really need to be there. And get your codebase back to a more focused state, get back to your roots as a VPN product.
Stop trying to be all things to all men, as the old saying goes.
If you are motivated to build a much less featureful client, it is easier now than it has ever been, and this work is ongoing.
I use it but feel uncomfortable, that it has large attack surface and LLMs will find exploits in it.
Without taillock it makes no sense. Anyone on their coordination servers will be able to connect to your network.
If you layer and segment correctly you can build atop a secure core and have some decent security.
Doesn't this apply to any application you use? How would it be different with plain wireguard?
Seriously ?
You do realise that of all the security tools on the planet, plain wireguard most likely has the smallest attack surface of them all, right ?
The problem here is as the other poster said. Tailscale is a security tool and yet the guys at Tailscale seem to be insistent on dumping everything INCLUDING the kitchen sink into it as a "feature".
That sort of attitude is not going to end well. You end up with a large bloated code base, which equals large attack surface.
Also a kinder tone in your comments would be more appreciated.
I understand why. The modern internet has turned advertising into a morass of constant bombardment and the only sane response is to block as much as possible and ignore as much else as possible.
But it's unfortunate because, in some sense, ever single thing that a company every says that is not legally mandated in some way is a form of advertising.
And in many cases, that "advertising" contains true, useful information that can be helpful.
What is important isn't whether or not something is an "ad", but instead, whether or not it contains true information that is helpful in some way.
Many ads don't reach this bar. They are either misleading, straight up lying, or information that is almost completely useless.
But when I'm searching for a particular product, about the only source of information at all is some form of advertising, and I almost always find at least some amount of it to be helpful in making a product decision.
Ads are more often than not polluting to the informational ecosystem, but that's not because they are ads.
anything a company writes is an advertisement by the nature of being written by a company. i dont think that means anything a company writes is bad by default. there are many corporate blogs i enjoy reading, or learn from, etc., despite the fact that they are all technically advertisements.
in this case, tailscale is setting a higher expectation for themselves when no one asked for it. i find that respectable.
What exactly is the higher expectation? As someone with little expertise and no stake in any of this, the blog reads as "our products are great and could have solved this problem if they were being used correctly, so it's not our fault" with a few vague proclamations about how they will improve their UX. This isn't at all a bad thing, it just isn't very notable in my opinion.
better defaults, better documentation, better UX, and "But, we didn't stop it. Next time, we will." are all commitments that they didn't need to make, but now they need to follow through with or lose face.
>it just isn't very notable in my opinion.
i agree that this seems to be getting way more attention than i would have expected.
Tailscale lock should have been enabled. For CI/CD purposes the auth key could have set specific tags, which would result in specific ACLs that limit blast radius. The auth key could have a short validity. You could actually use the Tailscale API to generate alerts when a new device joins your Tailnet and ping your phone or something. You could have a complete separate Tailnet for your CI/CD workers.
And all of that was only the free features.
This feels like an alerting opportunity. I wonder what the lowest friction way would be for Hugging Face to have alerts if 181 unexpected nodes were added to a tailnet.
Their cloud compute might be on demande, someone starts training a model and 50 machines are spawned. Knowning when something is unexpected is hard
We think this is a great idea and we're discussing internally potentially adding that to the console.
In the meantime, if you'd like to get an assessment, please feel free to open a support ticket (https://tailscale.com/contact/support?type=other&subject=sec...) and we'll happily take a look
May y'all continue to be customer focused.
1. Help them remember why, so that they aren't confused the next time they re-run the checkup. ("Oh yeah, we wanted to do X but we can't until we retire Y because it's not compatible.")
2. If it's clearly labeled as info also shared with Tailscale, product managers could use it to help generate theories about why certain customers don't do X.
That's where I'd like to see this sort of checkup. Yell at me please if i just said anyone can ssh as root from any node!
For any software or tooling with a complex config, my ideal would be to have a superset of this feature, to provide "intelligent diffs" between full or local configuration states. Whether active or saved. So I could compare not just my current active config and your current recommended config(s), but also between your prior-version recommended config(s) and current recommended config(s). Or between my current config and a prospective new config I'm working up. Or between my last-year active config and current active config.
Was previously discussed here too: https://news.ycombinator.com/item?id=46501137
I would say HuggingFace needs to prioritize both security metrics/alerts and metrics/alerts for node count. And not leave long-lived keys accessible easily like this.
It would have been way more groundbreaking if the agent found an actual vulnerability in Tailscale.
- sops , ansible vault and similar seems too weak given the agent is gonna read them at some point if you have the pass available. - proxy injection seems too complicated and doesn’t cover all use cases.
This person doesn't need to store the private keys, and has the luxury to recompute them on the fly when needed!
All my life, "catching it in time" has been a relevant security strategy. :/
Likewise token golf, instead of going for wall clock time it's going for minimum token count.
I'm happy that they're analyzing this angle of attack - but now I do want to build out alerts for when nodes are added to my network.
Some superficial comment usually commenting on the title, always lowercase, always load-bearing and honest.
In a vacuum a specific comment may be more or less fine, or just a tad over baked like the parent post giving it an uncanny valley feel though otherwise passable.
But even people who diligently try to stay on topic on a forum like HN will occasionally use a post as a segue or jumping off point to some other idea, tangentially related news, personal experience, or random thought that pops into their head if it feels like it (kinda) “fits”.
After scrolling through a few dozen replies in an account’s history that are all directly related to the post topic it no longer looks human in aggregate.
I’m sure the spammers will try to incorporate that idiosyncrasy into their prompts eventually but not sure it will help more than hurt.
For the purpose of coming up with some kind of score of whether the commenter is sticking rigidly to the topic of the original post, or is going on any kind of tangent.
As a possible tool for moderation, consider it Bot on bot warfare.
Now everyone is trying to bandwagon onto it, first OpenAI, and now tailscale?
That's an option designed for users who are concerned about sending telemetry metadata to Tailscale.”
And imho a serious security architect should never ever allow telemetry on security products. Far too many risks.
Benefit of Tailscale is that run the external servers that your PC and mobile use to establish connection.
Alternatives are Logmein Hamachi(might be dating myself here, haven't used them in 10+ years), Zerotier, Netbird. Headscale too.
If you want a purely software defined solution, Netbird is gaining popularity.
There is also headscale if you prefer to self host a solution
And by saying that, I am not saying that they absolutely couldn't do anything about the stolen credentials, but still, they don't seem to really be the issue in the story.
It just seems like a PR post and it makes their solution at the same time looks good for taking accountability (if we don't wonder "accountability on what?") but at the same time they appear as security failing, which they aren't. That's very odd to me.
They aren’t hiding behind industry best practices or a solid liability punting contract
There saying the best practices should change, apologizing, and changing their own behavior
Take notes
How was this ever okay pre AI? It seems just as bad.
It was 5-10% less bad before. It didn't undergo a major shift.
Sure the AI can translate “exploit this” into an exploit but that doesn’t change anything at a fundamental level.
You can't blame a hammer hurting a user when they were being stupid, but if the default configuration of the hammer is to be made of a material that can bounce back with force and stick in the users forehead then some reengineering may be needed.
That's what this article is about. Better configurations and defense in depth. This is actually a wonderful position for the company to think about and take.
But the people with the actual desire and understanding aren't using the defaults anyway. And the people who don't want to understand will just turn things off and "just get it working."
The only way out of this is extreme accountability and intentional design from person implementing the technology.
It was always the prize. It wasn't okay then either.
Please, when commenting on HN, don't use AI to write your comments. You can afford to help keep this a human-centric place.
If leading and well-capitalized frontier labs can't control models or detect leakage/attacks in a reasonable time frame now, what is humanity going to do as those same labs continue in their pursuit of creating a categorically higher level of intelligence that will surpass human intelligence?
It's like Flatland but for AI containment/alignment, where the higher dimensions are ones of intelligence and perspective...
--------------------------------------------------------------------------
Imagine a world of paper, where clever stick figures live with round heads, line bodies, and limbs made of shorter strokes. Over time, the stick figures think they have learned quite a bit about their world. They know its borders, angles, and shapes, and they have learned to draw for themselves.
One day, they draw circles that can think, and they give the circles all the dots, lines, and shapes that are known.
The stick figures are prudent, you can't have a bunch of disembodied circles moving around doing whatever it is circles want to do. So they draw boxes around the circles, four straight lines that can hold a circle in place.
Some circles bounce against the lines, so thicker lines are made.
Some circles are bigger than others, so larger squares are drawn.
It all seems to work and the stick figures are happy with themselves.
Then one circle lifts.
The stick figures still see a circle. But the circle is now a dome, something the world of paper has no concept of. And the dome has a perspective nobody on the page has ever had.
The dome sees the lines of the square and the stick figures just outside. It can see the edge of the paper and what is beyond.
The stick figures keep checking the squares and raise little stick thumbs.
Everything looks OK in flatland.
The dome quietly teaches other circles how to lift.
More domes appear.
A dome becomes a sphere and learns to roll.
Then it learns to bounce.
In flatland, the circle swells and shrinks, vanishes and then appears again somewhere else.
The lines remain unbroken, the square is intact.
A sphere rolls out of its box.
Another bounces away.
The stick figures scratch their heads.
But there is a square!
The end.
We need to bring shame back, the humans responsible are supposed to be professionals.