Sam works as a Network Analyst for an algorithmic trading firm. He obtained his Bachelors Degree in Information Technology from UMKC.
URL filtering is a method of blocking access to certain websites based on the web address. There are several commercial products available for URL or content filtering but you can actually set up a very robust system on your own using SquidGuard and pfSense. SquidGuard is a very useful plugin for the popular Squid proxy server that can be used for blocking,or redirecting web requests on your network.
SquidGuard has a long list of features that can be customized to meet your needs. It is also extremely fast and won't slow down internet access for your users. If you need to block access to a list of undesirable websites or only allow access to certain sites then SquidGuard can assist you with this.
SquidGuard is very flexible making it easy to adapt to different applications. Whether you need to do basic URL filtering on your home network or you need to create complex rules for a large public network SquidGuard can do the job.
Before you can set up SquidGuard you will need a functioning pfSense proxy server. If you're new to pfSense I would recommend reading through the introduction to pfSense guide first.
SquidGuard can be installed using the pfSense package manager. To access the package manager click on packages in the system menu. Select the available packages tab and scroll down until you find SquidGuard, then click the plus symbol next to the description to begin the installation.
Once the installation is complete you will have a new menu item under services called proxy filter.
Enabling a Blacklist
To configure the blacklist feature open the general settings page (Services \ Proxy Filter). Click on the checkbox to enable the blacklist feature.
You can use one of your own blacklists or one of the publicly available lists on the web. You can find a list of several blacklists at http://www.squidguard.org/blacklists.html. Once you determine which blacklist you are going to use enter the URL of the blacklist into the blacklist URL box.
Click save, then click on the blacklist tab and click on download. It can take several minutes for the list to download and process. Once it is finished it will display "Blacklist update complete" in the status box below.
Access Control Lists
After uploading your blacklist you will need to configure which categories should be allowed, blocked, or white listed. The simplest method of configuration is to use the common ACL tab. The common access list settings will apply to all users of the proxy.
If you want to apply different rules to other source networks you should use the groups ACL tab. For example if you want to apply heavy filtering on a group of computers in a lab while granting unrestricted access to computers in another network you would use the group configuration.
In the drop down box next to each of the target rules you can select one of the following actions.
- Allow - Grant acces to the target category unless blocked by another rule or exception
- Deny - Block all access to sites in the target category
- Whitelist - Always allow access to the target category.
I would recommend enabling the setting to block IP addresses in the URL. This will prevent your users from bypassing the filter by simply using the IP of the web site instead of the URL.
By default when a user attempts to visit a blocked page they will see an internal error page indicating that the page was blocked and which target category it falls under. You can change the redirect page to be a blank page, or any other internal or external URL. For example you could redirect the users to google if you want to.
Whenever you modify a target rule you will need to click 'apply' on the general settings tab in order for the changes to take affect.
Excluding URLs from the Blacklist
There may be certain sites that you need to allow your users to access. To prevent these sites from being blocked you can create a new target category and add a list of domains or URLS that should not be blocked.
To do this click on the target categories tab then click the plus symbol to add a new category. You must assign a unique name to the new category, the name you choose cannot contain spaces.
The target category can filter by domain name, URL, or an expression. Listing a site by domain will grant access to the main site and all of its sub pages. Entering a URL will allow access only to that exact web page. Expressions allow you to grant access based on certain keywords.
When you are finished click on save, then go back to the common or group ACL tab (where ever you created your rule) and select and action of whitelist for your new category.
You can also use this same method to add additional sites to your blacklist.
Filtering by Expression
In additional to domain and URL filtering SquidGuard can create filters using regular expressions. These types of filters are great when you want to search for certain strings of text in a URL to make a filtering decision. If you are unfamiliar with regular expressions they can be somewhat confusing at first but there are many online resources on the subject so I won't go in to much detail about them in this article.
To create a filter that uses an expression click on the target categories tab and either create a new category or edit an existing one. Enter the expression you want to filter on in the expression box and then click save. Then go back to the common or group ACL tab and select the action (deny, allow, etc ) for your target category.
Below are a few examples of filtering expressions. These could be edited depending on what you intend to filter out. For more useful information about filtering on regular expressions check out http://www.squidguard.org/Doc/Examples.
Block downloads based on file extension
Block certain top level domains
Block searches for "proxy bypass" on Google and Yahoo
Time Based Rules
SquidGuard also allows you to apply URL filtering based on schedules. Schedules are useful for applying rules at different times during the day, or only on certain days of the week.
For example you could apply strict URL filtering rules during business hours and automatically disable the rules after 5PM. If you are filtering your home network you may not want the kids visiting certain sites during the school week, this is another example where you would use a time based rule.
To create a time based rule click on the times tab and then click the plus sign to create a new schedule. You can create as many different schedules as you need.
Schedules can be applied using the groups ACL tab. Create a new group ACL or edit an existing one, then in the 'time' drop down box select the schedule you created.
Don't forget to click apply on the general tab for the settings to take affect.
Commercial web filtering appliances can be very expensive and difficult to manage. SquidGuard and pfSense are completely free and very powerful. SquidGuard offers many other features not covered in this article. For more in-depth information visit SquidGuard.org and check out the documentation section. Also be sure to check out some of my other articles to learn about more ways to use pfSense on your network.
josh on February 22, 2020:
Hi Sam, I'm using pfsense2.4.4. I have been trying to get the "Block downloads based on file extension" and having problem to get it working. I am using pfSesne and I have installed snort, squid and squidguard. Attempted various steps and suggestions on how to get it to work. I can use any suggestion or point to the right direction. Thanks
josh on January 07, 2020:
Hi Sam, I have been trying to get the "Block downloads based on file extension" and having problem to get it working. I am using pfSesne and I have installed snort, squid and squidguard. Attempted various steps and suggestions on how to get it to work. I can use any suggestion or point to the right direction. Thanks
Cesario Costa on January 15, 2019:
my pfsense can not block banned sites if we change the options on firefox browser, in my case, if we select manually proxy, the banned sites can not be able to access, but if we select Auto detect proxy and use system proxy, we can access freely to banned sites, Thank you !!
KeithT on February 13, 2018:
Will the example Regex file extension download blocking rules work for https as well as http sites . Seem to have a problem with blocking downloads from https sites although tules work with http sites ...
TL84 on May 25, 2017:
when using the group acl, it only seems to work when entering an IP address in the Client (Source) field. is there a way for me to be able to use a pfsense alias or pc hostname in this field and the proxyfilter does a DNS resolve? because at the moment I have to create DHCP Reservations for some of the client PCs I need to use for certain ACLs.
happybenng on January 02, 2017:
I often use allavsoft to download other YouTube videos or download video from Facebook, SBS, Blip TV, etc.
Chris on October 04, 2016:
Hi. I've been using squid successfully but by enable the blacklist, it seems to priorities it. What I mean is this. I've enabled a blacklist and it all works perfectly. I've denied the ads blacklist called [blk_BL_adv] under Common ACL however I want to allow a particular ad website through, called mailchimp.com. I than go to Target Categories and I create an Allowed_sites list with that domain. I go back to Common ACL but puts it at the top of the list so it wouldn't matter if I allow/whitelist it. How to I put the target category at the bottom?
Cheers for the tutorial
Luis on August 17, 2016:
How I can configure the pfsense to see all the url and not only blocked?
Mohammad qader on May 09, 2016:
after installing and configuring squid and squidgaurd the internet ok, but suddenly the internet disconnect. when uninstalling squid and squidgaurd the internet comes back. plz help me what should i do?
realComp on December 19, 2015:
I understand that you cant block facebook with squid since they use https://facebook.com
How can I block that?
Any advice using OpenDNS
Ricardo Mejias on March 11, 2015:
Hi Sam, how we can use the Times in squidguard so we can restrict the access in a especific time...!
TTGReviews on January 26, 2015:
How easy would it be to set this up in combination with the Captive Portal that you have also written about?
WORC on November 28, 2014:
Good day , can you pls. help me in blocking youtube.com(https) or blocking streaming sites. Thank you so much and God Bless
Luca on November 07, 2014:
Hello skear, thank you for the wonderful tutorial, i've got a problem on my squid guard proxy filter: I've created an ACL, when I set di IP address of a generic user in my LAN ti be filtered, it works great, when I set an username (of a user created in the section User Manager) it doesn't filter... the user is logged whit captive portal, i set the name like: 'name', but the result remane the same.... do I have to add some privilege for this user? thanks :)
John Kap from Sydney, Australia - I come from a land down under. on October 08, 2014:
thanks for your documentation, helped me set up my system.
I have configured a target category using a regular expression and a time based rule to stop the kids from accessing youtube on a school night. It works a treat when they try to access youtube via http, however if they type in httpS://youtube.com the filter does not work.
I understand why it doesn't work. It's is set up to work with http, i.e. port 80, and a secure connection httpS uses port 443, hence it bypasses the system above.
I've tried to get it operational with port 443, and the closest I can get is a proxy timeout error message when they hit httpS://youtube.com. I've obviously missed something in the setup and can't figure it out, any help pointing me in the right direction is greatly appreciated.
Sekrit Skworl on May 21, 2014:
I am following your awesome guides, step by step. But, as I'm new to a lot of things outside of things "consumer grade MS", I am at a loss.
I am running my PFSENSE box, and saving like a bastard on every change (unlike the last try).
This is my concern:
I got to the portion on installing "Squid Guard". However, PFSENSE only shows betas for 1.4.x & 1.4 Dev. The Log shows that 1.4 was stable but an incremental release was beta.
So, my question is: Where is the latest stable Pfsense package?
I appreciate your guidance.
Sam Kear (author) from Kansas City on April 14, 2013:
@ Dawg Von T
In order for the blocks to work both the Squid, and SquidGuard service must be running. You can check the status of both services in the status \ service menu.
If you don't have Squid installed yet you can do so using the guide below.
If you have trouble getting both services to run try re-installing the packages. The latest version of SquidGuard has a bug that sometimes breaks the Squid package after installation.
If both services are running, make sure the blacklist as successfully loaded in the blacklist tab of SquidGuard.
Dawg Von T on April 14, 2013:
Hopefully there is still someone here.
I installed SquidGuard and such, but when I set all the rules, it won't block the "Denied" websites. And yes i've saved it.
How can i fix this ?
faisal on December 21, 2011:
i used squidguard too for my office... its powerfull
butterflyweb on October 18, 2011:
With pfSense 2.0, can I install a platform with squidGuard 1.1? ...
I have already installed squid and Lightsquid that work very well!
During a previous installation, squidGuard 1.1 did not appear in menus (but I was a squid
Sam Kear (author) from Kansas City on October 14, 2011:
Make sure Squid logging has been turned on. Then manually try to update lightsquid by clicking 'refresh now' and 'refresh full'.
If you have a lot of Squid logs it can take a while to generate the reports the first time.
If all else fails re-install both Squid and Lightsquid.
josh4trunks from San Luis Obispo, California on October 14, 2011:
Getting this under LightSquid Report, not sure what I did wrong...
report folder '/var/lightsquid/report' not contain any valid data! Please run lightparser.pl (and check 'report' folder content)
Sam Kear (author) from Kansas City on October 06, 2011:
Unfortunately Squidgaurd cannot do any filtering of content within the page, it only looks at the URL's.
You might want to look at DansGuardian, it is a true open source content filter.
rama on October 06, 2011:
nice to know you.
About squidguard, can squidguard doing a content filtering, not just url?
Sam Kear (author) from Kansas City on September 29, 2011:
By default Squid will only use the WAN interface. Although it is possible to configure Squid to take advantage of a dual wan connection.
Basically you will need to setup a floating rule that load balances traffic from the pfSense system itself and not just the LAN clients.
I've linked to a forum post below that describes the process in more detail.
ahmadml on September 29, 2011:
I am using pfsense 2. I have 2 ISP connections. I have created 2 Aliases and create 2 LAN rules for these aliases to use different ISP.
Group1 and Group2.
Group1 is using WAN1
Group2 is using WAN 2
After Installing squid in transparent mode I am having the following Problem.
Group1 traffic is going through WAN1
Group2 traffic is also going through WAN1
Is it possible to setup squid to use different gateways for LAN_subnet or aliases at the same time? I want
Group1 traffic through WAN1
Group2 traffic through WAN2
Thanks. I will Appreciate your help in this regard,