Intercepting HTTPS Traffic Using the Squid Proxy Service in pfSense

Updated on January 15, 2018
skear profile image

Sam works as a Network Analyst for an algorithmic trading firm. He obtained his Bachelors Degree in Information Technology from UMKC.

Source

A Brief Introduction to Squid

Squid has become one of the most popular packages for pfSense firewalls and it's not hard to see why.

Squid proxy servers can improve network performance by keeping a local cache of commonly accessed web pages, images and other files. Additionally Squid can monitor traffic and keep a log of which web pages users on your network are viewing.

In order to fully take advantage of the benefits of Squid I recommend taking the time to properly enable HTTPS interception.

This guide assumes you already have a pfSense firewall up and running. If you don't have one yet you can easily install pfSense on an old computer you probably already have on hand.

Why You Should Enable HTTPS Interception in Squid

By default Squid proxy servers cannot monitor encrypted HTTPS traffic. Squid simply establishes a TCP connection to the destination server and responds to the client with an HTTP 200 response to indicate the connection was established.

Once this encrypted tunnel has been established Squid passes the packets between the client and the server but has no longer has any visibility to the traffic since it is protected by SSL encryption.

Over the last few years many popular web sites including Google, Youtube, Reddit and Facebook have started enabling HTTPS encryption by default. This means that without configuring HTTPS interception Squid proxies have limited filtering, monitoring and logging capabilities.

Fortunately Squid supports man in the middle SSL filtering which will allow you to more effectively monitor the traffic passing through the proxy server.

Encrypted HTTPS traffic can be inspected using the SSL interception feature in Squid.
Encrypted HTTPS traffic can be inspected using the SSL interception feature in Squid. | Source

Step 1: Install the Squid3 Package

To get started install the Squid3 package using the pfSense package manager (System \ Packages).

After locating Squid3 from the package list click the plus button on the right side of the package to start the package installation.

The package manager will automatically download and install the Squid3 PBI.

Installing the Squid3 package from the pfSense package manager.
Installing the Squid3 package from the pfSense package manager.
Squid3 package installation progress.
Squid3 package installation progress.

Step 2: Configure the Squid General Settings

After the Squid package has been installed the general settings must be configured. The settings page can be found in Services \ Squid Proxy Server.

Configure the following options:

  1. Enable Squid Proxy - Checked
  2. Keep Settings/Data - Checked
  3. Proxy Interface(s) - Select LAN and Loopback

The rest of the settings in the general settings area can be left on the default settings.

Squid3 General Settings Page in pfSense
Squid3 General Settings Page in pfSense

Step 3: Configure Transparent Proxy Settings

Just below general settings you'll find the transparent proxy settings. In most cases you will probably want to enable transparent proxy mode. When this mode is enabled the firewall will automatically redirect all incoming web traffic to the Squid proxy server.

With transparent mode enabled it is not necessary to configure the clients web browser to use the proxy. In most cases the client will not even notice their traffic is passing through the proxy.

To enable the transparent proxy configure the following settings:

  1. Transparent HTTP Proxy - Checked
  2. Transparent Proxy Interface(s) - LAN

The rest of the settings in this section can be left on their default settings unless you wish to configure specific addresses to bypass the proxy.

Scroll down to the bottom of the page and click save to apply the settings.

If you decide not to enable transparent mode you will need to configure the web browser settings for each client you wish to specifically use the proxy server.

Squid3 transparent proxy settings in pfSense.
Squid3 transparent proxy settings in pfSense.

At this point you have configured a basic Squid proxy running in transparent mode. Before proceeding further I recommend testing web browsing functionality from a client computer to make sure everything is working normally.

Troubleshoot and resolve any problems with the basic proxy functionality before proceeding to enable SSL interception.

Step 4: Configure a Certificate Authority

A certificate authority must be configured in pfSense before HTTPS interception can be enabled in Squid. The CA will be used to generate new client SSL certificates on the fly for automatically encrypting and decrypting web traffic.

  1. Access the cert manager in the System \ Cert Manager menu.
  2. Click the plus button to create a new certificate authority.

Configure the following settings for the new certificate authority.

  1. Descriptive Name - Choose a name for your CA. Keep in mind that this will be displayed on the client certificate visible to the clients.
  2. Method - Select 'Create an Internal Certificate Authority' from the dropdown menu.
  3. Key length - I recommend 2048 for maximum compatibility but you can use 4096 bits for maximum security.
  4. Digest Algorithm - Use SHA256 or higher. Like the previous setting you need to balance security with device compatibility.
  5. Lifetime - Set this to 3650 days (10 years)
  6. Distinguished name - Fill out all of the fields in the section (Country, State, etc). These will all be visible in the certificates seen by the clients.

Click the save button to finish creating the CA.

Creating a new certificate authority in pfSense.
Creating a new certificate authority in pfSense.

Step 5: Export the CA Certificate

After creating the new CA the CA certificate must be exported. This certificate will need to be installed on any client machine that will be using the proxy server.

From the CA manager page click the export CA cert button to download the certificate. This will download a copy of the CA certificate in .crt file format.

Exporting the CA certificate from the pfSense certificate authority manager.
Exporting the CA certificate from the pfSense certificate authority manager.

Step 6: Install the CA Cert to the Client Computers

In order to prevent the web browsers on client computers from showing certificate errors the CA certificate from the pfSense CA must be installed on all client computers that will be using the proxy server.

Skipping this step will result in clients receiving browser security errors and can cause various HTTPS connection problems.

The certificate must be placed in the Trusted Root Certificate Authorities store to prevent browser errors.

If you only have a small number of computers on your network then it will probably be easiest to manually import the certificate on each computer.

For larger networks you should consider setting up Microsoft Active Directory Certificate Services. AD integrated certificate authorities can automatically push out a root certificate to hosts which are members of the domain.

Importing the Certificate on Windows 7

To import the certificate to a computer running Windows 7 double click on the .crt file to open the certificate dialog box.

  1. Click the install certificate button to start the import wizard.
  2. Click next on the first page of the certificate import wizard.
  3. Select the option place all certificates in the following store.
  4. Click on the browse button and select Trusted Root Certification Authorities.
  5. Click next, then click finish on the import confirmation page.
  6. When prompted click yes to acknowledge the security warning.

You should see a message indicating the certificate import was successfully completed.

Click thumbnail to view full-size
Windows 7 certificate properties dialogWindows 7 certificate installation wizardImporting a certificate into the Trusted CA store on Windows 7Successfully importing a certificate in Windows 7
Windows 7 certificate properties dialog
Windows 7 certificate properties dialog
Windows 7 certificate installation wizard
Windows 7 certificate installation wizard
Importing a certificate into the Trusted CA store on Windows 7
Importing a certificate into the Trusted CA store on Windows 7
Successfully importing a certificate in Windows 7
Successfully importing a certificate in Windows 7

Importing the Certificate on Mac OS X

Follow the steps below to manually import the certificate on Mac OS X.

  1. Open the Keychain Access application - Use the spotlight search to easily find this app.
  2. Click the lock symbol to unlock the key chain for changes.
  3. Open the File menu and select Import Items.
  4. Select the CA certificate exported from pfSense. (At this point you should see the certificate in the keychain with the message "This root certificate is not trusted")
  5. Double click the certificate and expand the trust section of the dialog box. In the first dropdown box called "When using this certificate" select always trust.
  6. Close the dialog boxes and exit the keychain access application.

Click thumbnail to view full-size
Imported a CA certificate in Mac OS XMarking a certificate as "Always Trusted" in Mac OS X
Imported a CA certificate in Mac OS X
Imported a CA certificate in Mac OS X
Marking a certificate as "Always Trusted" in Mac OS X
Marking a certificate as "Always Trusted" in Mac OS X

Step 7: Enable SSL Man in the Middle Filtering

After loading the certificate to the client computers you are ready to enable SSL filtering in Squid. Access the squid settings page (Services \ Squid Proxy Server) and configure the settings below.

  1. HTTPS/SSL Interception - Checked
  2. SSL Intercept Interface(s) - Select LAN
  3. CA - Select the certificate authority created in step 4

Click save at the bottom of the page to apply the settings.

Squid3 SSL man in the middle settings in pfSense
Squid3 SSL man in the middle settings in pfSense

Step 8: Testing SSL Interception

The best practice after enabling SSL interception is to confirm that it is working as intended. Follow these steps to verify HTTPS connections are being decrypted by the proxy.

  1. Visit a site which uses HTTPS such as Reddit from a client computer behind the proxy.
  2. View the certificate information presented by the web browser. In Chrome this can be done by clicking the lock symbol on the address bar.
  3. Confirm the certificate issuer information matches the information you entered when creating the certificate authority in step 4.


Known Issues in Squid 3.4

The pfSense package manager currently contains Squid version 3.4 which has a known issue where it incorrectly generates SHA1 certificates instead of SHA256.

Since SHA1 is a weak algorithm many browsers will show errors when they encounter certificates signed using this algorithm.

To fix this I recommend manually upgrading to Squid version 3.5.3 which does not have this problem.

Upgrading to Squid 3.5.3

The upgrade instructions are slightly different depending on whether you are running the 32-bit or 64-bit version of pfSense.

To determine which version you have open the pfSense dashboard and check the version section of the system information dashboard widget. If you see AMD64 then follow the 64-bit instructions. If you see i386, then use the 32-bit instructions.

The commands can be run through an SSH terminal, or the web based terminal (Diagnostics \ Command Prompt)

64-Bit (AMD64) Instructions

  1. Download the PBI by running the command: fetch https://files.pfsense.org/packages/10/All/squid-3.5.3-amd64.pbi
  2. Install the package by running: pbi_add --no-checksig -f squid-3.5.3-amd64.pbi
  3. Run the commands below to create the correct directory structure
cd /usr/pbi/squid-amd64/
rm -rf /usr/pbi/squid-amd64/etc
ln -s /usr/pbi/squid-amd64/local/etc .
ln -s /usr/pbi/squid-amd64/local/lib .
ln -s /usr/pbi/squid-amd64/local/libexec .
ln -s /usr/pbi/squid-amd64/local/share .
ln -s /usr/pbi/squid-amd64/bin sbin


Reboot pfSense after running the above commands (Diagnostics \ Reboot).

32-Bit (i386) Instructions

  1. Download the PBI by running the command: fetch https://files.pfsense.org/packages/10/All/squid-3.5.3-i386.pbi
  2. Install the package by running: pbi_add --no-checksig -f squid-3.5.3-i386.pbi
  3. Run the commands below to create the correct directory structure
cd /usr/pbi/squid-i386/
rm -rf /usr/pbi/squid-i386/etc
ln -s /usr/pbi/squid-i386/local/etc .
ln -s /usr/pbi/squid-i386/local/lib .
ln -s /usr/pbi/squid-i386/local/libexec .
ln -s /usr/pbi/squid-i386/local/share .
ln -s /usr/pbi/squid-i386/bin sbin


Reboot pfSense after running the above commands (Diagnostics \ Reboot).

Verifying the Installation of Squid 3.5.3

After rebooting pfSense start a new SSH session (or use the web terminal) to verify the updated package was correctly installed.

When you run the command below you should see version 3.5.3 listed in the output.

/usr/local/sbin/squid -v

Verify Squid 3.5.3 has been correctly installed.
Verify Squid 3.5.3 has been correctly installed.

Completion

Congratulations, if you completed all of the steps above you have successfully configured Squid to intercept encrypted HTTPS traffic. You should immediately notice HTTPS requests being logged in the Squid access logs.

To take full advantage of the new access logs being collected I recommend installing a Squid log analyzer such as Lightsquid.

Questions & Answers

    © 2016 Sam Kear

    Comments

      0 of 8192 characters used
      Post Comment

      • profile image

        Doug 5 weeks ago

        Is there any way to get Suricata to inspect the decrypted traffic?

      • profile image

        zeadtariqhammoody 2 months ago

        what if we are using cellphone in this case what we have to do ?

        and if we want to make the cert instillation automatically

      • profile image

        shang 3 months ago

        Hi, I would like to know if there is a security issue enabling "man in the middle SSL filtering" in Pfsense configuration

      • profile image

        bilal94021 4 months ago

        Dear Sir,

        I Install SSL certificate on client and Browser, When i browse something they shows ERR_SSL_PROTOCOL_ERROR in browser.

      • profile image

        Avithus 6 months ago

        amazing guide!! help me a lot on the ssl issues ive been having, i just have 1 tiny problem, i my office theres arround 150 or so pc's, is there a way to install the certificates to all of them at once ? or remotely?

      • profile image

        sachin19 6 months ago

        Hello Sir,

        how can i block facebook/youtube pages ( Dynamic content filtering) in my entire Network ( AS Internet service provider).

      • profile image

        Franco 7 months ago

        I have had similar issues that is stated below, and while playing around a bit, i found the solution that worked for me.

        With bank sites you visit and get errors: go to squid proxy server - SSL Man in the middle. Set the SSL/MITM Mode to "splice all". Then go to remote cert checks and high-lite "Accept remote server certificate with errors". This worked and the banks and other ssl sites can be visited without any errors.

        To clarify, i had issues with the blacklist option. I re-installed squidgaurd, and did not do the blacklist option again. I manually went to target categories and did my blacklisting there.

        I know some might have better solutions, so, if you don't like what i have done that works for me, please do not take this as the best and final solution.

      • profile image

        Wild Wings 7 months ago

        For David,

        The reason this exists is because users are now bypassing security settings and policies by using https sites for common social media or other blocked/ policy prohibited sites. Most companies and government agencies have network acceptable use policies that clearly indicate when using their assets you are subject to monitoring and their acceptable use policies. If it is not work related then don't use company assets plain and simple. Go home use your personal computer if you don't like it. The short of it is by bypassing company security measures that filter and block you from visiting sites that could put the company in a position where they would be subject to litigation you can and should be fired. This is is one more step to ensure that said filtering can still be accomplished and protect the company.

      • profile image

        Aubrey-mw 10 months ago

        Would you please write the same with reference to pfsense 2.3.4-RELEASE (amd64)

      • profile image

        Kirby 10 months ago

        Hi, I am using the latest version of pfsense squid and squidguard. I enabled the SSL Man In the Middle Filtering in Squid proxy server to block https sites. However, when enabled I can't connect to certain messenger apps (ie. slack, bbm, skype). Any ideas why connection to these apps is always retrying?

      • profile image

        Sergs 11 months ago

        Hi. Is there any way that I can cache https without installing certificates on each clients on my network?? i have my latest version of pfsense 2.3.4-RELEASE (amd64)

        built on Wed May 03 15:13:29 CDT 2017

        FreeBSD 10.3-RELEASE-p19 installed on my pfsense box.

        Thank you

      • profile image

        Sergs 11 months ago

        Hi. Is there any way that I can cache https without installing certificates on each clients on my network?? i have my latest version of pfsense 2.3.4-RELEASE (amd64)

        built on Wed May 03 15:13:29 CDT 2017

        FreeBSD 10.3-RELEASE-p19 installed on my pfsense box.

        Thank you

      • profile image

        Deepak 12 months ago

        Hello Sam,

        This guide worked for me in pf sense 2.3.2 thank you.

        I want to access from home network to office network using vpn please suggest.

      • profile image

        Sheffy 13 months ago

        Can I use Let's encrypt' CA to intercept HTTPS traffic?

      • profile image

        Claudio 13 months ago

        Hi, I want to grant access only to some site, this is what I've done:

        I put in the white list a list of site to be allowed, for example

        google.com

        utorrent.com

        and in the black list I put . to block all others.

        Normal HTTP site are allowed or blocked correctly as they in the WL or not, while HTTPS sites are all permitted, why?

        How can I block HTTPS sites that are not in the WL?

        I also tried to put ^. in the BL but nothing changed.

        Thanks,

        Claudio.

      • profile image

        Waldo Pulanco 13 months ago

        Hi! It works like a charm, but the problem is very slow to load a page when the protocol is https. how do I overcome this problem?

        thanks in advance!!

      • profile image

        Patricio 14 months ago

        Hi, I implemented step by step and it works perfect, except when I want to enter a web of banks, they reject the certificate and do not let me log in. Could you help me ?

      • profile image

        Pfsense Authentication with OpenLDAP 15 months ago

        Sir

        I am new with PFSense, i have installed and configured pfsense and it is working fine.

        i have setup openLDAP server to create users and want to authenticate those users too with PFsense, so that i can provide internet to the user on the basis of userid authentication.

        But i am unable to achieve my goal please guide me how to do the same.

        my PFsense firewall ip is 192.168.1.1

        my OpenLDAP server ip is 192.168.1.4

        Thanks in advance

      • profile image

        David 15 months ago

        ** Installing the certificate on the client machines isn't necessary for deception to take place but the client browsers will display certificate warnings on any https websites. Because of these warnings it's really only feasible if you deploy the certificates to the client systems.

        Set up a forged cert server and spoof the trust settings in client machines to eavesdrop on private traffic. That is the very _essence_ of deception. I find it very unlikely that someone doing this would inform users ahead of time that their encrypted https communications have been exposed and are being monitored.

        ** Fortunately Squid supports man in the middle SSL filtering

        More commonly referred to as a "man-in-the-middle attack," and with good reason.

        Certificate Authorities exist for a purpose. Once that purpose has been subverted, trust is lost and everyone loses. If it's OK for you to eavesdrop on your users, it's OK for your upstream provider to eavesdrop on you, right?

        Offhand, I cannot think of any honest or ethical application for this. You are purposefully crippling https, and then modifying client trust mechanisms in such a way that it will not be immediately apparent to the end user. If you want to block access to sites like reddit and youtube, simply do so. Otherwise, you are violating your users' reasonable expectations of privacy. Depending on the venue, you are also violating law.

        One more reason that everyone should be using a good, reliable VPN service all the time.

        ** Sam works as a Network Analyst for an algorithmic trading firm

        Wow. I'm glad I don't work there.

        ** I’ve visited many of the major financial datacenters in North America to install new equipment and solve problems

        Oh, shit. Welp. We're screwed.

      • profile image

        umair975 16 months ago

        On PFSense 2.3.2... enabled SSL Interception with transparent proxy and Internal CA.. getting certificate error..

        My aim is to block social networking sites https via SquidGuard.

        any suggestions?

      • profile image

        Boyom Rodrigue 16 months ago

        Hello,

        I followed your guide carefully and I succeeded successfully.

        Nevertheless I have a problem. When I activate the https filtering in squid, the skype application of the stations on my network no longer connects to skype. Have you encountered this problem? If so, how did you resolve it?

      • skear profile image
        Author

        Sam Kear 17 months ago from Kansas City

        @Carlos

        HTTPS decryption will still work without the certificate installed on the end devices but they will see a certificate error on encrypted pages they visit. Best practice is to install the certificate to the devices.

        This can be a challenge if you're dealing with a public or guest network. In order to install it on the cell phones you could implement a mobile device management system such as XenMobile or possibly push the certificate using a captive portal.

        Another possibility might be to direct users to a link to download and install the certificate through a splash page.

      • profile image

        Carlos 17 months ago

        Is there any way to intercept https with transparent proxy with out to use a CA?. How I do to install a CA in each cell phone conected to my wifi lan?

      • profile image

        ERNEST 17 months ago

        hi sam

        what version of PF you mount this...i has been problems with your config... hhtp is ok but https doesn open and CA is ok but still not open when i enable SSL _tick_

        i have another question how filter by groups af acl

        the black list into squid is working but for all into the LAN

        if i want to by pass that restriction i puti on in SQUID

        im tried to use squidguard but doesnt work

        could explain me how to do that

        eaxample .- if i deny lego.com into squid = ok but for all....how by pass when a ip add (pc or user ) wants to pass trow this site

        ok i think into squid doesnt possible because apply for all lan ( value given into squid )

        2.- if i dont put it nothing into blacklist into squid

        i have the idea squidguard get into work , right ?? in that supouse case, i generate ACL commun or ACL particular ...obviosly i generate Targets

        but this is not working at all...

        could you explainme how to do that

      • DelfinDelfin profile image

        DelfinDelfin 17 months ago

        Hi Sam

        How can I install the certificate on Ubuntu?. I tried this: http://superuser.com/questions/437330/how-do-you-a...

        But it didn't work. Thanks in advance. Great post by the way

      • profile image

        konti 18 months ago

        use mail.google.com :)

      • profile image

        Nel 19 months ago

        gmail.com can't get through......

      • skear profile image
        Author

        Sam Kear 19 months ago from Kansas City

        Mohan,

        Installing the certificate on the client machines isn't necessary for deception to take place but the client browsers will display certificate warnings on any https web sites. Because of these warnings it's really only feasible if you deploy the certificates to the client systems.

      • profile image

        Mohan 19 months ago

        Hi,

        is there any possibility to without import CA certificate at client machine.

        Thanks

        mohanrao83@gmail.com

      • profile image

        Agi 19 months ago

        Hi !

        i have your solution fully working, but i'm not able to integrate with squidguard url filtering.

        When i enable squidguard the http site are working, but all httpS sites are not displayed and the browser and the url bar report a thing like : https://http.* and ovviously no site displayed.

        Any suggestion ?

      • profile image

        moivg79 21 months ago

        I have pfsense 2.3 and squid 3.5.19 and I have completed all of the steps above but ssl webs show certificate error. Can you help me?

        Thank you.

      • profile image

        Alonso 21 months ago

        On PfSense 2.3.1 Rp5 the last Squid versión available is 3.5.19. Even you suggest to install squid-3.5.3-amd64.pbi, but I can't install it because in PfSense 2.3.1-RELEASE the command pbi_add has been deprecated. Now seems that pkg command must to be used but "pkg add squid-3.5.3-amd64.pbi" is not enought. Do you know some trick to have installed the fixed Squid versión on PfSense 2.3.1 R p5?

      • profile image

        Mike 22 months ago

        Thanks a lot. This guide worked great!

      • profile image

        shr_kaza 23 months ago

        bagaimana cara untuk

        --enable-zph-qos di squid 3.5.3

      working