Chapter 4. Connecting With Win2k or XP Using Their Built In IPSec


if your RED IP address changes like the weather, you will have to register your IPCop with one of the dynamic DNS services such as Also note that this cannot be done through the IPCop Web interface, it must be done from a command line, and if you use the connections page of the web interface, it will wipe out all your settings.

Connecting a Win2K/XP box to an IPCop using the built in IPSec of Win2k Pro/XP is accomplished in about ten minutes. While not tested, the same should work for a Windows XP box.


You will have to edit the ipsec.conf and ipsec.secrets which are both placed in the /var/ipcop/vpn directory on your IPCop machine.

In my situation, I have a Win2K box behind an Assante Cable/DSL router connected to a cable modem. The IPCop box protects a private network with a subnet of 192.168.1.x and I am running a subnet of 192.168.10.x at my end. You need different subnets at each end otherwise the routing will not behave properly. By this I mean that you could not setup the network behind the IPCop to be 192.168.1.x and then have your road warrior be 192.168.1.x, it would have to be 192.168.2.x or some other private IP address. In the logging examples, you will see as an IP address. This is a fictional address for this example. My particular machine is and you will see that entered in the conf files, etc. And the IPCop box is Again you will see this in the logs, etc.


My Win2K machine is a different subnet to the IPCop machine.

Now on to the good stuff! First, make sure your Win2K box is ready to do the job, Service Pack 2 must be installed or at least the high encryption pack, it installs 3DES which is needed by IPCop. This is not necessary for XP as it contains 3DES already.

For Windows 2000, get the IPSec policy editor.

For Windows XP you will need the Ipseccmd program: You have to install the Win XP Support tools. They reside on your Win XP CD in the directory \SUPPORT\TOOLS. Just start setup.exe in this directory. You have to select a “Complete installation” to get ipseccmd.

Next download this utility: and extract the contents to the same place that the IPSECPOL.EXE for Win2k was installed to (typically c:\Program Files\Resource Kit\) or where Ipseccmd.exe was installed to for Windows XP.

Also to make sure you know what is going on with the IPCop box. Download and install PuTTY or some other Secure Shell. PuTTY is free and can be downloaded from here.

Make sure you turn on SSH on your IPCop box so that Putty or another Secure Shell can access the command line.

Now, you need to setup the ipsec.conf on both IPCop and the Win2k/XP machine. Here's a sample one for IPCop:

conn roadwarrior
    left=(RED address or dynamic dns name)
    leftsubnet=                           1
    rightsubnet=                       2

Subnet behind IPCop


If you are behind a firewall or other router put private address here otherwise leave blank

In the ipsec.secrets on the IPCop file make sure you have:

(RED address or dynamic dns) : PSK “PreShared secret here”
(RED address or dynamic dns) %any : PSK “PreShared secret here”

Now for the Win2k setup.


The ipsec.conf file that was downloaded, above, needs to be edited now. You will find that it already comes with sample connections inside of it. Erase all of these and replace them with a modified copy of the example, below. Change the connection name and IP addresses.

Here is a sample of a Win2K or XP ipsec.conf file:

conn KDI
    left=(RED address of ipcop or dynamic dns name of ipcop)
    leftsubnet=                           1
    presharedkey=PreShared secret here

Subnet behind IPCop

Now, from a DOS box, change directories to where the IPSECPOL.EXE was installed to (typically c:\Program Files\Resource Kit\) and then type IPSEC.EXE and that will initiate the IPSec connection. It took me two attempts to get this working, but it works and works well if all is configured properly. You should see this from Windows 2K:

    C:\Program Files\Resource Kit>ipsec.exe
IPSec Version 2.1.4 (c) 2001,2002 Marcus Mueller
Getting running Config ...
Microsoft's Windows 2000 identified
Host name is: darrenc
No RAS connections found.
LAN IP address:
Setting up IPSec ...
        Deactivating old policy...
        Removing old policy...
Connection KDI:
        MyTunnel     :
        MyNet        :
        PartnerTunnel: (RED IPCOP address or Dyn DNS Name)
        PartnerNet   :
        CA (ID)      : Preshared Key ******************
        PFS          : y
        Auto         : start
        Auth.Mode    : MD5
        Rekeying     : 3600S/50000K
        Activating policy...
C:\Program Files\Resource Kit>

Next from the Win2K box ping the GREEN IP Address of the IPCop box, after a couple of pings, it should get a reply. (Takes two tries with my setup, I have heard of it taking four or five) To ping type the following:

C:\>ping                          1

Pinging with 32 bytes of data:
Reply from bytes=32 time=51ms TTL=255
Reply from bytes=32 time=60ms TTL=255
Reply from bytes=32 time=50ms TTL=255
Reply from bytes=32 time=50ms TTL=255
Ping statistics for
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss)
Approximate round trip times in milli-seconds:
    Minimum = 50ms, Maximum =  60ms, Average =  52ms

GREEN address of IPCop

Ideally to make sure things are going as planned, have a putty (SSH - Secure Shell) session running to your IPCop box so you can examine /var/log/secure. For more information on SSH and how to set it up, look in the IPCopFAQ for How do I turn on SSH.

As for the IPCop log it should show something like the following:

[email protected]:~ # cat /var/log/secure
…#5: responding to Main Mode from unknown peer
…#3: Peer ID is ID_IPV4_ADDR: ''
…#3: sent MR3, ISAKMP SA established
…#6: responding to Quick Mode
…#6: IPsec SA established
[email protected]:~ # 

The above log can also show you what went wrong, or at least the vital information to post to the list to show us what went wrong so we can help you correct it.

If you fail to connect on the first attempt or try to reconnect after the connection goes idle, I have found that I have to restart the VPN on both ends, on the win2k box type

C:\>ipsec -off

Then on the IPCop, use the web interface to restart the VPN. Now start the Win2K IPSec again.

Now you know how to connect a Win2K box to an IPCop using the built in IPSec of Win2K.