Closed Bug 1134506 Opened 9 years ago Closed 9 years ago

Mark "Superfish, Inc." root certificate as untrusted in NSS

Categories

(NSS :: CA Certificates Code, task)

Product:
NSS
Component:
CA Certificates Code
Type:
task
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: reed, Assigned: rbarnes)

References

(
URL
)

Details

(Keywords: csectype-other, sec-high)

Attachments

(4 files)

superfish.pem
3.27 KB, text/plain
Details
Highly compromised private key for "superfish" rogue CA.
891 bytes, application/x-x509-ca-cert
Details
Revocation for "superfish" rogue CA.
1.56 KB, application/x-x509-ca-cert
Details
BrowserimpactofSuperfish.pdf
6.57 MB, application/pdf
Details 
https://twitter.com/shaver/status/568216937181749248

"Lenovo installs a MITM cert and proxy called Superfish, on new laptops, so it can inject ads? Someone tell me that's not the world I'm in."

https://forums.lenovo.com/t5/Security-Malware/Potentially-Unwanted-Program-Superfish-VisualDiscovery/td-p/1794457
Is there anything to blacklist? Is this root and key the same for all users? Does this even affect Firefox's root store (the images are of Windows' root store)

Is there any reason to believe the blacklisting will not be trivially circumvented, giving that this PUP runs at same-or-higher privilege as NSS using apps?
(In reply to Ryan Sleevi from comment #3)
> Is there anything to blacklist? Is this root and key the same for all users?
> Does this even affect Firefox's root store (the images are of Windows' root
> store)

I would be interested in the answer to the this.  If it's not hitting the NSS root store, it's not an issue for us.


> Is there any reason to believe the blacklisting will not be trivially
> circumvented, giving that this PUP runs at same-or-higher privilege as NSS
> using apps?

At the very least, blacklisting can increase the work that the attackers have to do, and make them take action specifically against NSS/Firefox, vs. against OS.

OneCRL gives us a mechanism to block this above the NSS layer, so even if they're already flipping bits in NSS, we can block them a layer up (and make them meet us there).
(In reply to Richard Barnes [:rbarnes] from comment #4)
> At the very least, blacklisting can increase the work that the attackers
> have to do, and make them take action specifically against NSS/Firefox, vs.
> against OS.
> 
> OneCRL gives us a mechanism to block this above the NSS layer, so even if
> they're already flipping bits in NSS, we can block them a layer up (and make
> them meet us there).

As you know, we have CRLSets. We also have a product designed to prevent or remove potentially unwanted programs.

While Mozilla can and should do what it feels is appropriate for Firefox, I can only counsel you that, from our experience, this is like getting in a land war in Asia. Except you get to pay $400/hour for the privilege.

The work factor for this is trivially low for motivated parties. We're talking on the order of hours for most mitigations, a day or two for the clever ones that take days to weeks to write, and even the most farfetched fall quickly.

If it uses a single key, there is reason to block this on security grounds, regardless of the PUP. If it uses a key per device, there isn't much to do.
(In reply to Ryan Sleevi from comment #5)
> (In reply to Richard Barnes [:rbarnes] from comment #4)
> > At the very least, blacklisting can increase the work that the attackers
> > have to do, and make them take action specifically against NSS/Firefox, vs.
> > against OS.
> > 
> > OneCRL gives us a mechanism to block this above the NSS layer, so even if
> > they're already flipping bits in NSS, we can block them a layer up (and make
> > them meet us there).
> 
> As you know, we have CRLSets. We also have a product designed to prevent or
> remove potentially unwanted programs.
> 
> While Mozilla can and should do what it feels is appropriate for Firefox, I
> can only counsel you that, from our experience, this is like getting in a
> land war in Asia. Except you get to pay $400/hour for the privilege.
> 
> The work factor for this is trivially low for motivated parties. We're
> talking on the order of hours for most mitigations, a day or two for the
> clever ones that take days to weeks to write, and even the most farfetched
> fall quickly.
> 
> If it uses a single key, there is reason to block this on security grounds,
> regardless of the PUP. If it uses a key per device, there isn't much to do.

I agree that there's a limit to how much it makes sense to escalate here.  As you say, if there's a single key (or even a small number), then it makes sense to block it.  If they're willing to adapt, we're SOL anyway.
Can we make this bug public?
Considering this started in public forums and came to us via Twitter keeping it hidden isn't protecting anyone from an unknown flaw.
Group: core-security
I think that just marking it (assuming it is a single cert or a small set of certs) as explicitly blacklisted (perhaps even prompting the user that this cert exists and how to get rid of it) should be enough.

Users who update the browser* will at least not be affected anymore. Of course, Superfish can update itself and become "smarter", but currently there's a lot of scrutiny there so I doubt they will. In the meantime, consumers won't have the dangerous certificate with them, and that's something. With this certificate on the system we run the risk of an insider selling the privkey and/or using it for malicious purposes even if Superfish gets shut down.


*can we force a security update notification for Windows? I'm not sure how Firefox's update mechanism works on Windows currently
Attached file superfish.pem — 
Can't verify this, but here's a dump of the cert that is circling around on Twitter.

https://twitter.com/fugueish/status/568264203020079105
https://twitter.com/fugueish/status/568258997578371072
(In reply to Manish Goregaokar [:manishearth] from comment #9)
> I think that just marking it (assuming it is a single cert or a small set of
> certs) as explicitly blacklisted (perhaps even prompting the user that this
> cert exists and how to get rid of it) should be enough.

I believe we should be able to blacklist using a hotfix add-on.
It seems to use a single key, and the CA root don't seem to get removed on uninstall.
(In reply to Richard Barnes [:rbarnes] from comment #11)
> (In reply to Manish Goregaokar [:manishearth] from comment #9)
> > I think that just marking it (assuming it is a single cert or a small set of
> > certs) as explicitly blacklisted (perhaps even prompting the user that this
> > cert exists and how to get rid of it) should be enough.
> 
> I believe we should be able to blacklist using a hotfix add-on.

Who is taking point on this? Can we also blacklist it using OneCRL?
Is there evidence this root gets installed into the NSS database? The one installed in the windows cert store won't be used by Firefox and can't be blacklisted by Firefox (using current features).
https://twitter.com/supersat/status/568343079268327424

Also, if you google "Superfish Chrome" or "Superfish Firefox", there have been cases in the past[1] where a superfish addon has turned up unwanted. There's a good change it happens on the "infected" laptops too.

Of course, the usual blacklist may not work for an addon. Not sure what we can do for that.


 [1]: https://support.mozilla.org/en-US/questions/878411
Actually, maybe block that plug-in is required.
(Sorry, I don't know that option will cast emails, sorry for disturb.)
From https://twitter.com/supersat/status/568343079268327424: "btw the code that I'm looking at suggests it may try to install itself into Firefox and Opera stores" and the pictured disassembly clearly mentions "thunderbird.exe" and "firefox.exe".
Yeah, but we don't know what plugin it installs or how.

Perhaps WindowShopper[1]?



 [1]: https://addons.mozilla.org/en-us/firefox/addon/windowshopper-automatic-price-/?src=cb-dl-mostpopular
Confirmed: It installs WindowShopper/VisualDiscovery on Firefox and puts the cert in root.


https://twitter.com/ETFovac/status/56836125300297728
(In reply to Reed Loden [:reed] from comment #13)
> Who is taking point on this? Can we also blacklist it using OneCRL?

I am point on this.  Thanks to all for the information.  I'm trying to get my hands on an affected device today to get started on mitigations.
Assignee: nobody → rlb
(In reply to Manish Goregaokar [:manishearth] from comment #19)
> Confirmed: It installs WindowShopper/VisualDiscovery on Firefox and puts the
> cert in root.
> 
> 
> https://twitter.com/ETFovac/status/56836125300297728

Does the AddOn do other bad stuff? Should we also block that?
Flags: needinfo?(dveditz)
(In reply to Eric Rescorla (:ekr) from comment #21)
> Does the AddOn do other bad stuff? Should we also block that?

At least the current public AMO version of the add-on (https://addons.mozilla.org/en-US/firefox/addon/windowshopper-automatic-price-/versions/1.2.0.22) does not modify the cert store.

Is there a non-amo version? If so, the add-ons team would need to have an example.
Flags: needinfo?(dveditz)
This discussion has some links to the binaries:
https://twitter.com/supersat/status/568343079268327424
If we want to a fix/blacklist for 36, this will have to arrive very soon.
https://www.eff.org/deeplinks/2015/02/further-evidence-lenovo-breaking-https-security-its-laptops

EFF has gotten 44k reports of Superfish from Firefox instances.
If we fix https://bugzilla.mozilla.org/show_bug.cgi?id=1059392 we can fix this for pinned sites, including Google/Facebook. However that does not help non-pinned sites.
Highly compromised private key for "superfish" rogue CA.
Note that self-signed CA certificate has previously been attached.
Note that revocation will be attached in a moment.
Previous attachments include the self-signed certificate 
and the private key for "superfish" rogue CA.

It appears that the CA certificate has been revoked.
Here is the CRL.  This ought to be distriubted widely,
the sooner the better.
Just to be clear: Anyone can produce a revocation because the private key is available, right? Also the rogue CA cert doesn't have OCSP or CRL fields so browser won't fetch revocation status from anywhere.

Also, it sounds like there may be a difference between OEM Superfish installs and end-user installs with regards to whether they put the rogue CA in Firefox's trust store: https://twitter.com/ericlaw/status/568494324155076608. It makes sense: If the OEM build doesn't include Firefox, it would be more difficult for Snapfish to insert its rogue CA later. What I don't yet understand is that it sounds like OEM installs don't get the MITM CA cert, but also appear to browse fine without running into cert errors.
(In reply to Richard Barnes [:rbarnes] from comment #4)
> OneCRL gives us a mechanism to block this above the NSS layer, so even if
> they're already flipping bits in NSS, we can block them a layer up (and make
> them meet us there).

No, because of bug 1130757. And also, because of the problem identified by bug 1128607, the effectiveness of a OneCRL-based block would be limited, even if bug 1130757 were fixed.
(In reply to John Denker from comment #29)
> Created attachment 8566634 [details]
> Revocation for "superfish" rogue CA.

Let's remember that root trust anchor CA certificates cannot be revoked with CRLs or OCSP because certificate verification libraries don't check them for revocation because the design of CRL/OCSP revocation mechanisms is such that doing so doesn't make sense.

It is possible for Microsoft to remove this cert with its certificate trust list mechanism. For Gecko-based and NSS-based software, an update to NSS is required, currently.
(In reply to Brian Smith (:briansmith, :bsmith, use NEEDINFO?) from comment #32)

I don't understand the following remark:

> the design of CRL/OCSP revocation mechanisms is such that
> doing so doesn't make sense.

Could somebody give a few words of explanation?  What's the
problem.  In particular, could we discuss changing the design 
so that checking the CRL does make sense?

I can understand that "sometimes" checking wouldn't make sense,
but superfish is the poster child illustrating that sometimes
it does make sense.  Checking the list to see if the CA cert 
has been revoked is an easy computation.  It is an expedient 
way to deal with compromised CAs.  Why not just do the check?

If anybody cares about the theory involved:  This may look like
the Epimenides paradox, but it isn't quite.  If you trust the
issuing CA even for a moment, then you trust the revocation,
so from now on it's revoked.  If you don't trust the issuing 
CA cert (because it's been revoked!), then you don't even need 
to look at the revocation;  the CA is already untrusted, QED.
It's a roach motel:  certs check in, but they don't check out.
This document describes my findings from testing a Superfish-infected computer at Best Buy today.  The upshot is that Firefox HTTPS connections appear to be unaffected by Superfish.  The observed behavior has been replicated by Chris Palmer on another device.

Given this above, it's not inconceivable that we could close this WONTFIX.  The only impact of blocking the Superfish cert would be to prevent users clicking through certificate warnings if this cert is used for an attack.
According to this thread: https://twitter.com/ETFovac/status/568425380513771520

A restart is required before Firefox is affected. Can anybody confirm that?
I think the certificate should be blacklisted even if current OEM installations don't appear to affect Firefox.

The private key has been published, which means exploits _will_ start to circulate.  These exploits could potentially affect (at least) anyone who clicks through sec_error_untrusted_issuer.  There are no sites legitimately signed with this certificate, so blacklisting it has no compat downside.

Also, the EFF's data indicate that the certificate _does_ get installed into the NSS cert store under some conditions.  We don't know what those are, but I can think of several plausible scenarios.  For instance, maybe this happens when the Superfish software is bundle-installed onto a machine that already has Firefox.
According to this tweet,[0] Superfish's installer will add its cert to Fx if Fx is installed:

https://twitter.com/FiloSottile/status/568584150988537856

Thus, I say blacklist.
(In reply to Zack Weinberg (:zwol) from comment #36)
> The private key has been published, which means exploits _will_ start to
> circulate.  These exploits could potentially affect (at least) anyone who
> clicks through sec_error_untrusted_issuer.  There are no sites legitimately
> signed with this certificate, so blacklisting it has no compat downside.

Unfortunately it will always be possible to generate a certificate that will result in sec_error_untrusted_issuer for non-pinned sites. Blacklisting this certificate will not prevent those attacks. The compatibility downside of doing so is that if a user's traffic is being proxied through Superfish, blocking the certificate would prevent them from visiting any https site. It would then be difficult for the user to figure out what's wrong and how to fix it.

> Also, the EFF's data indicate that the certificate _does_ get installed into
> the NSS cert store under some conditions.  We don't know what those are, but
> I can think of several plausible scenarios.  For instance, maybe this
> happens when the Superfish software is bundle-installed onto a machine that
> already has Firefox.

As I understand it, the EFF's data indicates that some Firefox users have seen certificates issued by the Superfish certificate. It doesn't indicate that it has been added to the root store as a trust anchor.
(In reply to David Keeler [:keeler] (use needinfo?) from comment #38)
> (In reply to Zack Weinberg (:zwol) from comment #36)
> > The private key has been published, which means exploits _will_ start to
> > circulate.  These exploits could potentially affect (at least) anyone who
> > clicks through sec_error_untrusted_issuer.  There are no sites legitimately
> > signed with this certificate, so blacklisting it has no compat downside.
> 
> Unfortunately it will always be possible to generate a certificate that will
> result in sec_error_untrusted_issuer for non-pinned sites. Blacklisting this
> certificate will not prevent those attacks. The compatibility downside of
> doing so is that if a user's traffic is being proxied through Superfish,
> blocking the certificate would prevent them from visiting any https site. It
> would then be difficult for the user to figure out what's wrong and how to
> fix it.

I think this is hitting the news right?
Are they removing the certs too?
(In reply to David Keeler [:keeler] (use needinfo?) from comment #38)
> if a user's traffic is being proxied through Superfish,
> blocking the certificate would prevent them from visiting any https site. It
> would then be difficult for the user to figure out what's wrong and how to
> fix it.

Do we have the ability to display a custom certerror screen for a particular blacklisted root?  (If not, maybe we should!)

> As I understand it, the EFF's data indicates that some Firefox users have
> seen certificates issued by the Superfish certificate. It doesn't indicate
> that it has been added to the root store as a trust anchor.

I could be wrong about how the SSL Observatory works, but I am under the impression that A implies B.
It's been reported at https://news.ycombinator.com/item?id=9078247 that the Superfish proxy allows even invalid or self-signed certificates to be accepted by the browser; it tries to make the browser reject them by prepending "verify_fail." to the hostname, but it neglects to modify or remove the SubjectAltName.

So if the Superfish MITM proxy is being used, and its root certificate is trusted, any certificate with a SAN matching the hostname will be accepted.
(In reply to Cesar Eduardo Barros from comment #43)
> It's been reported at https://news.ycombinator.com/item?id=9078247 that the
> Superfish proxy allows even invalid or self-signed certificates to be
> accepted by the browser; it tries to make the browser reject them by
> prepending "verify_fail." to the hostname, but it neglects to modify or
> remove the SubjectAltName.
> 
> So if the Superfish MITM proxy is being used, and its root certificate is
> trusted, any certificate with a SAN matching the hostname will be accepted.

As far as Firefox goes the cert doesn't actually go into our store, see https://bugzilla.mozilla.org/attachment.cgi?id=8566794
>As far as Firefox goes the cert doesn't actually go into our store, see https://bugzilla.mozilla.org/attachment.cgi?id=8566794

There are multiple reports it actually *does* if you reboot the machine.
Blocks: 1134989
Any cert which is being added to root stores by widely-deployed software, and for which the private key is known, is a risk. We can examine the behaviour of software and installers we can see, but we don't know what those installers might have done last week, last month or last year. Without extensive research, we also don't know their full function, and under what circumstances running software may continue to edit root lists, and which root lists.

Given that, I think we need to block all the Superfish/Komodia root certs we can get our hands on. (Bug 1134989 seems to point to several.)

zwol raises the reasonable point that this might leave people who are using Superfish/Komodia software unable to access the Internet at all in our browser, even to find uninstall instructions. This is a problem, and ideally we would be able to provide custom error messages of some sort. We don't have that capability now, though. I wonder how hard it would be to hack up a forced redirect to a URL on SuMO, where we can provide support information which can be enhanced as time goes on, even in different languages. This minimises the required changes to Firefox and doesn't give us an l10n problem.

In the medium term, I think "never do SSL MITM under any circumstances" is not a realistic position to take (although we can argue about which use cases are legitimate), and so we need to develop a plan whereby this sort of thing can be done with mandatory transparency. More on that later.

Gerv
(In reply to Gervase Markham [:gerv] from comment #47)
> Any cert which is being added to root stores by widely-deployed software,
> and for which the private key is known, is a risk. We can examine the
> behaviour of software and installers we can see, but we don't know what
> those installers might have done last week, last month or last year. Without
> extensive research, we also don't know their full function, and under what
> circumstances running software may continue to edit root lists, and which
> root lists.
> 
> Given that, I think we need to block all the Superfish/Komodia root certs we
> can get our hands on. (Bug 1134989 seems to point to several.)

I concur.

> zwol raises the reasonable point that this might leave people who are using
> Superfish/Komodia software unable to access the Internet at all in our
> browser, even to find uninstall instructions. 

Correction: David Keeler raised this issue, not me (comment 38).

> This is a problem, and ideally
> we would be able to provide custom error messages of some sort. We don't
> have that capability now, though. I wonder how hard it would be to hack up a
> forced redirect to a URL on SuMO, where we can provide support information
> which can be enhanced as time goes on, even in different languages. This
> minimises the required changes to Firefox and doesn't give us an l10n
> problem.

Isn't there a chicken-and-egg problem with that?  SuMO is HTTPS-only.  I suppose there could be a separate, unencrypted "how to troubleshoot your network" site.

This winds up feeding into the general problem of connectivity diagnostics; I do think we should be trying to do more there.

> In the medium term, I think "never do SSL MITM under any circumstances" is
> not a realistic position to take (although we can argue about which use
> cases are legitimate), and so we need to develop a plan whereby this sort of
> thing can be done with mandatory transparency. More on that later.

I recall you had a proposal for something like that a few years ago.  Unfortunately, I do not see _any_ way of accomplishing it which avoids warning fatigue.  Specifically, if you work in an environment (a law firm, say) that does SSL MITM for data exfiltration monitoring, you will have the "mandatory transparency" notification, whatever form it takes, visible all day at work, and you will learn not to notice it.  Now you take your personal laptop to a coffee shop to write a novel. The bar's not supposed to appear -- but it does, because the coffee shop's WiFi router is compromised -- and you don't notice, because it's become normal.
It appears that MSFT has issued a Windows Defender update that removes the Superfish root from the Windows cert store and also removes the proxy.  (And also the bogus NSS libraries that they were apparently shipping!)

https://twitter.com/FiloSottile/status/568804580743614465

If this can be confirmed, it should leave us clear to revoke the certificate, since if the proxy is removed, then revoking the certificate will not cause breakage (it will just complete the disinfection).
Another idea to consider in parallel to this: use the hotfix addon to remove the certificate from Firefox installs. This would allow the cert to get removed in all versions quickly, without a full update needed. A full update to untrust would still be needed, but the bulk of the infection could be dealt with in a hotfix addon first.
(In reply to Dave Garrett from comment #50)
> Another idea to consider in parallel to this: use the hotfix addon to remove
> the certificate from Firefox installs. This would allow the cert to get
> removed in all versions quickly, without a full update needed. A full update
> to untrust would still be needed, but the bulk of the infection could be
> dealt with in a hotfix addon first.

Yep, that is among the options being considered.  By "revoke", I meant, "somehow mark Superfish connections as sec_error_revoked_certificate, or at least sec_error_unknown_issuer".  We'll have to figure out how exactly we can achieve that effect.
One solution to fix all this mess would be to just use the Windows store for certificates, as God intended.

Only half joking. For me having to maintain multiple certificate stores is extremely annoying.
(In reply to Vladimir-Corneliu Nicolici from comment #52)
> One solution to fix all this mess would be to just use the Windows store for
> certificates, as God intended.
> 
> Only half joking. For me having to maintain multiple certificate stores is
> extremely annoying.

So you would prefer Microsoft and its policies to decide what is accepted for Firefox? 

Having our own root store (as part of NSS work) gives us flexibility and the ability to influence policy and push people in the direction of good decisions about certificates, trust, and security.
Man in the middle attack - marking as critical
Severity: major → critical
Keywords: csectype-other, sec-high
(In reply to Richard Barnes [:rbarnes] from comment #51)
> Yep, that is among the options being considered.  By "revoke", I meant,
> "somehow mark Superfish connections as sec_error_revoked_certificate, or at
> least sec_error_unknown_issuer".  We'll have to figure out how exactly we
> can achieve that effect.

Richard: has a technical course of action been chosen yet?

Do we yet have a full list of the certificates we want to disable in this way (c.f. bug 1134989 and the information therein)?

Gerv
See Also: → 1136150
See bug 1136150
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Bug 1136150 is not a fix to NSS.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I think we should be able to do this pretty soon with clear UI indications given the widespread press.
(In reply to Gervase Markham [:gerv] from comment #55)
> (In reply to Richard Barnes [:rbarnes] from comment #51)
> > Yep, that is among the options being considered.  By "revoke", I meant,
> > "somehow mark Superfish connections as sec_error_revoked_certificate, or at
> > least sec_error_unknown_issuer".  We'll have to figure out how exactly we
> > can achieve that effect.
> 
> Richard: has a technical course of action been chosen yet?
> 
> Do we yet have a full list of the certificates we want to disable in this
> way (c.f. bug 1134989 and the information therein)?

I've actually come around to the view that it's not that useful to address the Komodia issues at the level of NSS.  Unless you can verify that the interception software is gone (as bug 1136150 does), you risk blacking out an infected user.

For non-infected users, the value is minimal (changing SEC_ERROR_UNTRUSTED_ISSUER to SEC_ERROR_REVOKED_CERT).  And it seems quixotic to try to enumerate every possible bogus root out there and include it in NSS as untrusted.  If we did so, the root store would quickly grow unmanageably large.

In light of that, resolving this as WONTFIX, and renaming bug 1134989 more appropriately.
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Flags: needinfo?(rlb)
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: