Intercepting HTTPS Traffic Using the Squid Proxy Service in pfSense - TurboFuture - Technology
Updated date:

Intercepting HTTPS Traffic Using the Squid Proxy Service in pfSense

Author:

Sam works as a Network Analyst for an algorithmic trading firm. He obtained his Bachelors Degree in Information Technology from UMKC.

intercepting-https-traffic-using-the-squid-proxy-in-pfsense

A Brief Introduction to Squid

Squid has become one of the most popular packages for pfSense firewalls and it's not hard to see why.

Squid proxy servers can improve network performance by keeping a local cache of commonly accessed web pages, images and other files. Additionally Squid can monitor traffic and keep a log of which web pages users on your network are viewing.

In order to fully take advantage of the benefits of Squid I recommend taking the time to properly enable HTTPS interception.

This guide assumes you already have a pfSense firewall up and running. If you don't have one yet you can easily install pfSense on an old computer you probably already have on hand.

Why You Should Enable HTTPS Interception in Squid

By default Squid proxy servers cannot monitor encrypted HTTPS traffic. Squid simply establishes a TCP connection to the destination server and responds to the client with an HTTP 200 response to indicate the connection was established.

Once this encrypted tunnel has been established Squid passes the packets between the client and the server but has no longer has any visibility to the traffic since it is protected by SSL encryption.

Over the last few years many popular web sites including Google, Youtube, Reddit and Facebook have started enabling HTTPS encryption by default. This means that without configuring HTTPS interception Squid proxies have limited filtering, monitoring and logging capabilities.

Fortunately Squid supports man in the middle SSL filtering which will allow you to more effectively monitor the traffic passing through the proxy server.

Encrypted HTTPS traffic can be inspected using the SSL interception feature in Squid.

Encrypted HTTPS traffic can be inspected using the SSL interception feature in Squid.

Step 1: Install the Squid3 Package

To get started install the Squid3 package using the pfSense package manager (System \ Packages).

After locating Squid3 from the package list click the plus button on the right side of the package to start the package installation.

The package manager will automatically download and install the Squid3 PBI.

Installing the Squid3 package from the pfSense package manager.

Installing the Squid3 package from the pfSense package manager.

Squid3 package installation progress.

Squid3 package installation progress.

Step 2: Configure the Squid General Settings

After the Squid package has been installed the general settings must be configured. The settings page can be found in Services \ Squid Proxy Server.

Configure the following options:

  1. Enable Squid Proxy - Checked
  2. Keep Settings/Data - Checked
  3. Proxy Interface(s) - Select LAN and Loopback

The rest of the settings in the general settings area can be left on the default settings.

Squid3 General Settings Page in pfSense

Squid3 General Settings Page in pfSense

Step 3: Configure Transparent Proxy Settings

Just below general settings you'll find the transparent proxy settings. In most cases you will probably want to enable transparent proxy mode. When this mode is enabled the firewall will automatically redirect all incoming web traffic to the Squid proxy server.

With transparent mode enabled it is not necessary to configure the clients web browser to use the proxy. In most cases the client will not even notice their traffic is passing through the proxy.

To enable the transparent proxy configure the following settings:

  1. Transparent HTTP Proxy - Checked
  2. Transparent Proxy Interface(s) - LAN

The rest of the settings in this section can be left on their default settings unless you wish to configure specific addresses to bypass the proxy.

Scroll down to the bottom of the page and click save to apply the settings.

If you decide not to enable transparent mode you will need to configure the web browser settings for each client you wish to specifically use the proxy server.

Squid3 transparent proxy settings in pfSense.

Squid3 transparent proxy settings in pfSense.

At this point you have configured a basic Squid proxy running in transparent mode. Before proceeding further I recommend testing web browsing functionality from a client computer to make sure everything is working normally.

Troubleshoot and resolve any problems with the basic proxy functionality before proceeding to enable SSL interception.

Step 4: Configure a Certificate Authority

A certificate authority must be configured in pfSense before HTTPS interception can be enabled in Squid. The CA will be used to generate new client SSL certificates on the fly for automatically encrypting and decrypting web traffic.

  1. Access the cert manager in the System \ Cert Manager menu.
  2. Click the plus button to create a new certificate authority.

Configure the following settings for the new certificate authority.

  1. Descriptive Name - Choose a name for your CA. Keep in mind that this will be displayed on the client certificate visible to the clients.
  2. Method - Select 'Create an Internal Certificate Authority' from the dropdown menu.
  3. Key length - I recommend 2048 for maximum compatibility but you can use 4096 bits for maximum security.
  4. Digest Algorithm - Use SHA256 or higher. Like the previous setting you need to balance security with device compatibility.
  5. Lifetime - Set this to 3650 days (10 years)
  6. Distinguished name - Fill out all of the fields in the section (Country, State, etc). These will all be visible in the certificates seen by the clients.

Click the save button to finish creating the CA.

Creating a new certificate authority in pfSense.

Creating a new certificate authority in pfSense.

Step 5: Export the CA Certificate

After creating the new CA the CA certificate must be exported. This certificate will need to be installed on any client machine that will be using the proxy server.

From the CA manager page click the export CA cert button to download the certificate. This will download a copy of the CA certificate in .crt file format.

Exporting the CA certificate from the pfSense certificate authority manager.

Exporting the CA certificate from the pfSense certificate authority manager.

Step 6: Install the CA Cert to the Client Computers

In order to prevent the web browsers on client computers from showing certificate errors the CA certificate from the pfSense CA must be installed on all client computers that will be using the proxy server.

Skipping this step will result in clients receiving browser security errors and can cause various HTTPS connection problems.

The certificate must be placed in the Trusted Root Certificate Authorities store to prevent browser errors.

If you only have a small number of computers on your network then it will probably be easiest to manually import the certificate on each computer.

For larger networks you should consider setting up Microsoft Active Directory Certificate Services. AD integrated certificate authorities can automatically push out a root certificate to hosts which are members of the domain.

Importing the Certificate on Windows 7

To import the certificate to a computer running Windows 7 double click on the .crt file to open the certificate dialog box.

  1. Click the install certificate button to start the import wizard.
  2. Click next on the first page of the certificate import wizard.
  3. Select the option place all certificates in the following store.
  4. Click on the browse button and select Trusted Root Certification Authorities.
  5. Click next, then click finish on the import confirmation page.
  6. When prompted click yes to acknowledge the security warning.

You should see a message indicating the certificate import was successfully completed.

Importing the Certificate on Mac OS X

Follow the steps below to manually import the certificate on Mac OS X.

  1. Open the Keychain Access application - Use the spotlight search to easily find this app.
  2. Click the lock symbol to unlock the key chain for changes.
  3. Open the File menu and select Import Items.
  4. Select the CA certificate exported from pfSense. (At this point you should see the certificate in the keychain with the message "This root certificate is not trusted")
  5. Double click the certificate and expand the trust section of the dialog box. In the first dropdown box called "When using this certificate" select always trust.
  6. Close the dialog boxes and exit the keychain access application.

Step 7: Enable SSL Man in the Middle Filtering

After loading the certificate to the client computers you are ready to enable SSL filtering in Squid. Access the squid settings page (Services \ Squid Proxy Server) and configure the settings below.

  1. HTTPS/SSL Interception - Checked
  2. SSL Intercept Interface(s) - Select LAN
  3. CA - Select the certificate authority created in step 4

Click save at the bottom of the page to apply the settings.

Squid3 SSL man in the middle settings in pfSense

Squid3 SSL man in the middle settings in pfSense

Step 8: Testing SSL Interception

The best practice after enabling SSL interception is to confirm that it is working as intended. Follow these steps to verify HTTPS connections are being decrypted by the proxy.

  1. Visit a site which uses HTTPS such as Reddit from a client computer behind the proxy.
  2. View the certificate information presented by the web browser. In Chrome this can be done by clicking the lock symbol on the address bar.
  3. Confirm the certificate issuer information matches the information you entered when creating the certificate authority in step 4.


intercepting-https-traffic-using-the-squid-proxy-in-pfsense

Known Issues in Squid 3.4

The pfSense package manager currently contains Squid version 3.4 which has a known issue where it incorrectly generates SHA1 certificates instead of SHA256.

Since SHA1 is a weak algorithm many browsers will show errors when they encounter certificates signed using this algorithm.

To fix this I recommend manually upgrading to Squid version 3.5.3 which does not have this problem.

Upgrading to Squid 3.5.3

The upgrade instructions are slightly different depending on whether you are running the 32-bit or 64-bit version of pfSense.

To determine which version you have open the pfSense dashboard and check the version section of the system information dashboard widget. If you see AMD64 then follow the 64-bit instructions. If you see i386, then use the 32-bit instructions.

The commands can be run through an SSH terminal, or the web based terminal (Diagnostics \ Command Prompt)

64-Bit (AMD64) Instructions

  1. Download the PBI by running the command: fetch https://files.pfsense.org/packages/10/All/squid-3.5.3-amd64.pbi
  2. Install the package by running: pbi_add --no-checksig -f squid-3.5.3-amd64.pbi
  3. Run the commands below to create the correct directory structure

cd /usr/pbi/squid-amd64/
rm -rf /usr/pbi/squid-amd64/etc
ln -s /usr/pbi/squid-amd64/local/etc .
ln -s /usr/pbi/squid-amd64/local/lib .
ln -s /usr/pbi/squid-amd64/local/libexec .
ln -s /usr/pbi/squid-amd64/local/share .
ln -s /usr/pbi/squid-amd64/bin sbin


Reboot pfSense after running the above commands (Diagnostics \ Reboot).

32-Bit (i386) Instructions

  1. Download the PBI by running the command: fetch https://files.pfsense.org/packages/10/All/squid-3.5.3-i386.pbi
  2. Install the package by running: pbi_add --no-checksig -f squid-3.5.3-i386.pbi
  3. Run the commands below to create the correct directory structure

cd /usr/pbi/squid-i386/
rm -rf /usr/pbi/squid-i386/etc
ln -s /usr/pbi/squid-i386/local/etc .
ln -s /usr/pbi/squid-i386/local/lib .
ln -s /usr/pbi/squid-i386/local/libexec .
ln -s /usr/pbi/squid-i386/local/share .
ln -s /usr/pbi/squid-i386/bin sbin


Reboot pfSense after running the above commands (Diagnostics \ Reboot).

Verifying the Installation of Squid 3.5.3

After rebooting pfSense start a new SSH session (or use the web terminal) to verify the updated package was correctly installed.

When you run the command below you should see version 3.5.3 listed in the output.

/usr/local/sbin/squid -v

Verify Squid 3.5.3 has been correctly installed.

Verify Squid 3.5.3 has been correctly installed.

Completion

Congratulations, if you completed all of the steps above you have successfully configured Squid to intercept encrypted HTTPS traffic. You should immediately notice HTTPS requests being logged in the Squid access logs.

To take full advantage of the new access logs being collected I recommend installing a Squid log analyzer such as Lightsquid.

This article is accurate and true to the best of the author’s knowledge. Content is for informational or entertainment purposes only and does not substitute for personal counsel or professional advice in business, financial, legal, or technical matters.

© 2016 Sam Kear

Comments

Muhammad Ahmad Raza on October 29, 2018:

Dear Sir,

We have a problem with SSL interception. In the latest version of pfsense, we have enabled SSL Interception with Splice whitelist bump otherwise. With this option, all goes well. while opening https sites showing internal-ca. But with this option app, mobile apps didn't work.

If we chose Splice ALL option, All https sites start using GlobalSign Cert and nothing get blocked.

Kindly share your expertise.

Regards

Denis on August 28, 2018:

I would also like to know how to use Squid HTTPS interception, with either Suricata or Snort. My interest is not so much about content filtering/logging, but more about malware detection and prevention. Thanks.

Doug on March 19, 2018:

Is there any way to get Suricata to inspect the decrypted traffic?

zeadtariqhammoody on February 04, 2018:

what if we are using cellphone in this case what we have to do ?

and if we want to make the cert instillation automatically

shang on January 08, 2018:

Hi, I would like to know if there is a security issue enabling "man in the middle SSL filtering" in Pfsense configuration

bilal94021 on November 30, 2017:

Dear Sir,

I Install SSL certificate on client and Browser, When i browse something they shows ERR_SSL_PROTOCOL_ERROR in browser.

Avithus on October 20, 2017:

amazing guide!! help me a lot on the ssl issues ive been having, i just have 1 tiny problem, i my office theres arround 150 or so pc's, is there a way to install the certificates to all of them at once ? or remotely?

sachin19 on October 16, 2017:

Hello Sir,

how can i block facebook/youtube pages ( Dynamic content filtering) in my entire Network ( AS Internet service provider).

Franco on September 12, 2017:

I have had similar issues that is stated below, and while playing around a bit, i found the solution that worked for me.

With bank sites you visit and get errors: go to squid proxy server - SSL Man in the middle. Set the SSL/MITM Mode to "splice all". Then go to remote cert checks and high-lite "Accept remote server certificate with errors". This worked and the banks and other ssl sites can be visited without any errors.

To clarify, i had issues with the blacklist option. I re-installed squidgaurd, and did not do the blacklist option again. I manually went to target categories and did my blacklisting there.

I know some might have better solutions, so, if you don't like what i have done that works for me, please do not take this as the best and final solution.

Wild Wings on September 08, 2017:

For David,

The reason this exists is because users are now bypassing security settings and policies by using https sites for common social media or other blocked/ policy prohibited sites. Most companies and government agencies have network acceptable use policies that clearly indicate when using their assets you are subject to monitoring and their acceptable use policies. If it is not work related then don't use company assets plain and simple. Go home use your personal computer if you don't like it. The short of it is by bypassing company security measures that filter and block you from visiting sites that could put the company in a position where they would be subject to litigation you can and should be fired. This is is one more step to ensure that said filtering can still be accomplished and protect the company.

Aubrey-mw on June 20, 2017:

Would you please write the same with reference to pfsense 2.3.4-RELEASE (amd64)

Kirby on June 02, 2017:

Hi, I am using the latest version of pfsense squid and squidguard. I enabled the SSL Man In the Middle Filtering in Squid proxy server to block https sites. However, when enabled I can't connect to certain messenger apps (ie. slack, bbm, skype). Any ideas why connection to these apps is always retrying?

Sergs on May 18, 2017:

Hi. Is there any way that I can cache https without installing certificates on each clients on my network?? i have my latest version of pfsense 2.3.4-RELEASE (amd64)

built on Wed May 03 15:13:29 CDT 2017

FreeBSD 10.3-RELEASE-p19 installed on my pfsense box.

Thank you

Sergs on May 18, 2017:

Hi. Is there any way that I can cache https without installing certificates on each clients on my network?? i have my latest version of pfsense 2.3.4-RELEASE (amd64)

built on Wed May 03 15:13:29 CDT 2017

FreeBSD 10.3-RELEASE-p19 installed on my pfsense box.

Thank you

Deepak on April 12, 2017:

Hello Sam,

This guide worked for me in pf sense 2.3.2 thank you.

I want to access from home network to office network using vpn please suggest.

Sheffy on March 25, 2017:

Can I use Let's encrypt' CA to intercept HTTPS traffic?

Claudio on March 15, 2017:

Hi, I want to grant access only to some site, this is what I've done:

I put in the white list a list of site to be allowed, for example

google.com

utorrent.com

and in the black list I put . to block all others.

Normal HTTP site are allowed or blocked correctly as they in the WL or not, while HTTPS sites are all permitted, why?

How can I block HTTPS sites that are not in the WL?

I also tried to put ^. in the BL but nothing changed.

Thanks,

Claudio.

Waldo Pulanco on March 12, 2017:

Hi! It works like a charm, but the problem is very slow to load a page when the protocol is https. how do I overcome this problem?

thanks in advance!!

Patricio on February 17, 2017:

Hi, I implemented step by step and it works perfect, except when I want to enter a web of banks, they reject the certificate and do not let me log in. Could you help me ?

Pfsense Authentication with OpenLDAP on January 30, 2017:

Sir

I am new with PFSense, i have installed and configured pfsense and it is working fine.

i have setup openLDAP server to create users and want to authenticate those users too with PFsense, so that i can provide internet to the user on the basis of userid authentication.

But i am unable to achieve my goal please guide me how to do the same.

my PFsense firewall ip is 192.168.1.1

my OpenLDAP server ip is 192.168.1.4

Thanks in advance

David on January 18, 2017:

** Installing the certificate on the client machines isn't necessary for deception to take place but the client browsers will display certificate warnings on any https websites. Because of these warnings it's really only feasible if you deploy the certificates to the client systems.

Set up a forged cert server and spoof the trust settings in client machines to eavesdrop on private traffic. That is the very _essence_ of deception. I find it very unlikely that someone doing this would inform users ahead of time that their encrypted https communications have been exposed and are being monitored.

** Fortunately Squid supports man in the middle SSL filtering

More commonly referred to as a "man-in-the-middle attack," and with good reason.

Certificate Authorities exist for a purpose. Once that purpose has been subverted, trust is lost and everyone loses. If it's OK for you to eavesdrop on your users, it's OK for your upstream provider to eavesdrop on you, right?

Offhand, I cannot think of any honest or ethical application for this. You are purposefully crippling https, and then modifying client trust mechanisms in such a way that it will not be immediately apparent to the end user. If you want to block access to sites like reddit and youtube, simply do so. Otherwise, you are violating your users' reasonable expectations of privacy. Depending on the venue, you are also violating law.

One more reason that everyone should be using a good, reliable VPN service all the time.

** Sam works as a Network Analyst for an algorithmic trading firm

Wow. I'm glad I don't work there.

** I’ve visited many of the major financial datacenters in North America to install new equipment and solve problems

Oh, shit. Welp. We're screwed.

umair975 on December 24, 2016:

On PFSense 2.3.2... enabled SSL Interception with transparent proxy and Internal CA.. getting certificate error..

My aim is to block social networking sites https via SquidGuard.

any suggestions?

Boyom Rodrigue on December 07, 2016:

Hello,

I followed your guide carefully and I succeeded successfully.

Nevertheless I have a problem. When I activate the https filtering in squid, the skype application of the stations on my network no longer connects to skype. Have you encountered this problem? If so, how did you resolve it?

Sam Kear (author) from Kansas City on December 01, 2016:

@Carlos

HTTPS decryption will still work without the certificate installed on the end devices but they will see a certificate error on encrypted pages they visit. Best practice is to install the certificate to the devices.

This can be a challenge if you're dealing with a public or guest network. In order to install it on the cell phones you could implement a mobile device management system such as XenMobile or possibly push the certificate using a captive portal.

Another possibility might be to direct users to a link to download and install the certificate through a splash page.

Carlos on November 30, 2016:

Is there any way to intercept https with transparent proxy with out to use a CA?. How I do to install a CA in each cell phone conected to my wifi lan?

ERNEST on November 16, 2016:

hi sam

what version of PF you mount this...i has been problems with your config... hhtp is ok but https doesn open and CA is ok but still not open when i enable SSL _tick_

i have another question how filter by groups af acl

the black list into squid is working but for all into the LAN

if i want to by pass that restriction i puti on in SQUID

im tried to use squidguard but doesnt work

could explain me how to do that

eaxample .- if i deny lego.com into squid = ok but for all....how by pass when a ip add (pc or user ) wants to pass trow this site

ok i think into squid doesnt possible because apply for all lan ( value given into squid )

2.- if i dont put it nothing into blacklist into squid

i have the idea squidguard get into work , right ?? in that supouse case, i generate ACL commun or ACL particular ...obviosly i generate Targets

but this is not working at all...

could you explainme how to do that

DelfinDelfin on November 02, 2016:

Hi Sam

How can I install the certificate on Ubuntu?. I tried this: http://superuser.com/questions/437330/how-do-you-a...

But it didn't work. Thanks in advance. Great post by the way

konti on October 19, 2016:

use mail.google.com :)

Nel on September 29, 2016:

gmail.com can't get through......

Sam Kear (author) from Kansas City on September 28, 2016:

Mohan,

Installing the certificate on the client machines isn't necessary for deception to take place but the client browsers will display certificate warnings on any https web sites. Because of these warnings it's really only feasible if you deploy the certificates to the client systems.

Mohan on September 28, 2016:

Hi,

is there any possibility to without import CA certificate at client machine.

Thanks

mohanrao83@gmail.com

Agi on September 13, 2016:

Hi !

i have your solution fully working, but i'm not able to integrate with squidguard url filtering.

When i enable squidguard the http site are working, but all httpS sites are not displayed and the browser and the url bar report a thing like : https://http.* and ovviously no site displayed.

Any suggestion ?

moivg79 on August 04, 2016:

I have pfsense 2.3 and squid 3.5.19 and I have completed all of the steps above but ssl webs show certificate error. Can you help me?

Thank you.

Alonso on July 07, 2016:

On PfSense 2.3.1 Rp5 the last Squid versión available is 3.5.19. Even you suggest to install squid-3.5.3-amd64.pbi, but I can't install it because in PfSense 2.3.1-RELEASE the command pbi_add has been deprecated. Now seems that pkg command must to be used but "pkg add squid-3.5.3-amd64.pbi" is not enought. Do you know some trick to have installed the fixed Squid versión on PfSense 2.3.1 R p5?

Mike on June 26, 2016:

Thanks a lot. This guide worked great!

shr_kaza on May 23, 2016:

bagaimana cara untuk

--enable-zph-qos di squid 3.5.3