Some Computer Hints


Python Script - UDP Hole Punching

Hole punching is a technique in computer networking for establishing a direct connection between two parties in which one or both are behind firewalls or behind routers that use network address translation (NAT). These days, many (bad) ISPs put their users behind carrier-grade NATs (CGNs) for mitigating IPv4 address exhaustion problem. However, this breaks the end-to-end principle. In other words, if two parties are unfortunate enough to have such poor ISPs, they cannot communicate with each other directly.

In such cases ICMP hole punching, UDP hole punching, or TCP hole punching may help. Some of these methods may need an unrestricted server to initiate a connection which notes endpoint and session information including public IP and port number, along with private IP and port number. I have tried some of these, but I was not able to get good results. Failing to use them, I decided to write my own Python script, which only needs to know (to be given as parameters) the public IP addresses of the parties.

How to Use This Script?

Assume that you have a TCP service (httpd, sshd, VNC server, etc.) running on one party, which we will call the server. On the other hand, we want to access this service from another party, which we will call the client (running http browser, ssh terminal, VNC client, etc.). If both parties are behind a NAT as described above and there is no way for one party to establish a TCP connection to the other party, the script described below will do this.

Requirements:

First, copy the script below into a text file called udphp.py. Then, from command line, go to the folder containing the file and try to run that file with no parameter. If your Python 3 installation and configuration is OK, the script should display some short usage information. For longer help information, try to run it like this:

./udphp.py -h

Assume that you want to access a VNC server running on the server (TCP port 5901) from a VNC client running on the client.

  1. Run the following command on the server:
    ./udphp.py -c 5901 client’s_public_ip_address
  2. Run the following on the client:
    ./udphp.py -l 5901 server’s_public_ip_address
  3. Run the VNC client program on the client and try to connect to localhost:5901.

Obviously, you can change these TCP ports as you need. Important: To keep the script simple, I designed it so that it only supports one TCP session per each run. If a TCP session is lost or disconnected, the script on the failing side exits; eventually the script on the other side will exit also. So, you need to take all the three actions in succession and within 30 seconds or otherwise the first command will timeout. In such a case, it is very important to kill the Python script on both sides and start over.

The script has an option -d that will display extra debug information to stderr. If there is a problem, please first try to use this debug information. In some cases, you may have to adjust the MAXBUF_SIZE appropriately. See explanations given in the source code. Needless to say, any change made on the script should be made on both sides: The server and the client should run the exact copy of the script.

The script uses UDP port 5555 by default for UDP hole punching / bidirectional communication between the server and the client. If this port is busy on one of your hosts, you can use another UDP port (say, 6666) and specify it on both sides like this:

./udphp.py -c 5901 client’s_public_ip_address 6666	# on the server
./udphp.py -l 5901 server’s_public_ip_address 6666	# on the client

The client and the server TCP port need not be the same. For example, your SSH server may run on TCP port 22 on the server. However, you may not be able to use the same TCP port on the client. The following example may be used:

./udphp.py -c 22 client’s_public_ip_address 7777	# on the server
./udphp.py -l 2222 server’s_public_ip_address 7777	# on the client
ssh -p 2222 username@localhost				# on the client

Source Code of the Script

#!/usr/bin/env python3

UDP_PORT = 5555         # Default port for UDP communication
# Please, adjust MAXBUF_SIZE for optimal and WORKING values.
# 40000 looks optimal for my test environment. But, IT MAY NOT WORK IN YOUR CASE!
MAXBUF_SIZE = 40000     # Max size for TCP data. Must be < 2^16 (to fit struct.pack format: 'H')
HEADER_SIZE =    16     # 1:msgtype + 2:seqno + 2:buflen + 11:checksum
#   TOTAL   = 40016     # Max size for UDP data (MAXBUF_SIZE+HEADER_SIZE)
# I repeat: Large UDP packets e.g. > 40KB (or even 30KB) may not be transmitted at all!
# Also, small values like 1.4KB will be very inefficient for graphical apps like VNC!
DIGEST_SIZE = HEADER_SIZE-5   # Only that many bytes from MD5SUM will be used for digest
# Format of UDP data transmitted:
#  [0:1] = Message type indicating:
#   'A' : Positive ack
#   'N' : Negative ack
#   'H' : Heartbeat (keep alive) message
#   'U' : Data message which contains uncompressed buffer
#   'C' : Data message which contains compressed buffer
#  [1:3] = Sequence number to keep track UDP packets
#  [3:5] = Length of (uncompressed original) buffer data
# [5:16] = Digest or checksum of the (uncompressed, original) buffer data
#  [16:] = Buffer containing (TCP) data (possibly compressed)

# === Parse command line arguments:
import argparse
parser = argparse.ArgumentParser(description='''UDP Hole Puncher. Ver. 0.12
\N{COPYRIGHT SIGN} 2019-02-08 Fedon Kadifeli''',
  formatter_class=argparse.RawDescriptionHelpFormatter,
  epilog='''modes of operation:
  server mode: udphp.py -c [-d] tcp_conn_port   client_public_ip [udp_port]
  client mode: udphp.py -l [-d] tcp_listen_port server_public_ip [udp_port]''')
parser.add_argument("-d", "--debug",  help="print extra debug info to stderr", action="store_true")
pmode = parser.add_mutually_exclusive_group(required=True)
pmode.add_argument("-c", "--connect", help="'server' side mode; connect to local TCP port", action="store_true")
pmode.add_argument("-l", "--listen",  help="'client' side mode; listen to local TCP port",  action="store_true")
parser.add_argument("tcp_port", help="TCP port for connection (when running on the server) or TCP port for listening (when running on the client)", type=int)
parser.add_argument("udp_pub_ip", help="peer's public IP")
parser.add_argument("udp_port", help="UDP port for communication with peer (default "+str(UDP_PORT)+")", type=int, nargs='?', default=UDP_PORT)
ARGS = parser.parse_args()
del parser, pmode

# Set up error logging:
import logging
logging.basicConfig(level=logging.DEBUG if ARGS.debug else logging.WARNING,
  format='%(asctime)s.%(msecs)03d %(levelname)s: %(message)s', datefmt='%H:%M:%S')
##logging.debug('Debug!'); logging.info('Info!'); logging.warning('Warn!'); logging.error('Error!')

import socket, time, sys, os, threading, struct, random

MAX_SEQNO_BITS = 0xFFFF     # Must be a number like (2^n-1) < 2^16 (to fit struct.pack format: 'H')
ACKED_SEQNO = -1            # Sequence number of received ack
ACKED_OK = False            # Received ack type
HB_SMSG = b'HELOCLNT'       # Heartbeat messages
HB_RMSG = b'HELOSRVR'
HB_COUNT = 0                # To keep track of heartbeat timeout
TOT_ORIGDATA = 0            # Length of data send (original)
TOT_COMPDATA = 0            # Length of real data send (possibly compressed)
SERVER_MODE = True
if ARGS.listen:             # Client side mode
  SERVER_MODE = False
  HB_SMSG, HB_RMSG = HB_RMSG, HB_SMSG

MY_IP = ''
PEER_IP = socket.gethostbyname(ARGS.udp_pub_ip)
UDP_PORT = ARGS.udp_port
TCP_PORT = ARGS.tcp_port
del ARGS

logging.warning('Server_Mode={} Local_TCP_Port={} Peer_Public_IP={} Peer_UDP_Port={}'.format(SERVER_MODE, TCP_PORT, PEER_IP, UDP_PORT))

# Prepare UDP for hearbeat thread:
udp_sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp_sock.bind((MY_IP, UDP_PORT))

# === Class: Heartbeat_Thread ===
class Heartbeat_Thread(threading.Thread):
  def __init__(self, name=None, args=(), kwargs=None):
    super().__init__()
    self.name = name
  def run(self):
    global HB_COUNT
    logging.warning(self.name+'Started.')
    stat_count=0
    while HB_COUNT<10:
      HB_COUNT+=1       # HB_COUNT is reset to 0 when a heartbeat message is received by UDPReceive_Thread
      udp_sock.sendto(HB_SMSG, (PEER_IP, UDP_PORT))
      logging.debug(self.name+'Sent: '+str(HB_SMSG))
      time.sleep(10 * (2+random.random()) )
      stat_count+=1
      if (TOT_ORIGDATA>0) and (stat_count>15):
        stat_count=0
        logging.warning(self.name+'Cumul. compr. ratio: {:.4}'.format(TOT_COMPDATA/TOT_ORIGDATA))
    logging.error(self.name+'No heartbeat signal for a while: Exiting...')
    os._exit(14)
# End of Heartbeat_Thread

Heartbeat_Thread(name='HTBT: ').start()

tcp_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
if SERVER_MODE:
  tcp_sock.connect(('', TCP_PORT))
  logging.warning('Connected to local TCP port {}'.format(TCP_PORT))
  tcp_conn = tcp_sock
else:
  tcp_sock.bind(('', TCP_PORT))
  logging.warning('Listening for connection on local TCP port {}'.format(TCP_PORT))
  tcp_sock.listen(1)
  tcp_conn, addr = tcp_sock.accept()

ack_event = threading.Event()   # For synchronization of ack receipts

# === Helper functions ===
def calculate_digest(b):
  from hashlib import md5
  return md5(b).digest()[:DIGEST_SIZE]
# calculate_digest

def attempt_compr(buf, buflen):
  from zlib import compress
  MIN_GAIN=100
  global TOT_COMPDATA, TOT_ORIGDATA
  TOT_ORIGDATA += buflen
  if buflen>MIN_GAIN:
    compr_buf = compress(buf)
    compr_buflen = len(compr_buf)
    if compr_buflen<(buflen-MIN_GAIN):
      TOT_COMPDATA += compr_buflen
      return b'C', compr_buf, compr_buflen
  # No significant gain from compression
  TOT_COMPDATA += buflen
  return b'U', buf, buflen
# attempt_compr

def attempt_decompr(buf):
  from zlib import decompress
  try:
    dec_buf = decompress(buf)
    return dec_buf
  except Exception as e:
    logging.warning('Decompress: '+e)
    return b''
# attempt_decompr
# End of Helper functions

# === Class: UDPSend_Thread ===
class UDPSend_Thread(threading.Thread):
  def __init__(self, name=None, args=(), kwargs=None):
    super().__init__()
    self.name = name
  def run(self):
    global ACKED_SEQNO, ACKED_OK
    logging.warning(self.name+'Started.')
    seqno = 0
    while True:
      try:
        inbuf = tcp_conn.recv(MAXBUF_SIZE)
      except:
        logging.error(self.name+'TCP connection lost: Exiting...')
        os._exit(11)
      if not inbuf:
        logging.error(self.name+'TCP connection closed (EOF): Exiting...')
        os._exit(12)
      buflen = len(inbuf)
      digest = calculate_digest(inbuf)
      compr_status, inbuf, cbuf_len = attempt_compr(inbuf, buflen)
      data = struct.pack('!sHH{}s{}s'.format(DIGEST_SIZE, cbuf_len), compr_status, seqno, buflen, digest, inbuf)
      ACKED_OK = False
      for retry in range(30):
        udp_sock.sendto(data,(PEER_IP, UDP_PORT))
        ack_event.clear()
        ack_event.wait(1)
        if (ACKED_SEQNO==seqno) and ACKED_OK:
          break
        logging.warning(self.name+'Retry sending: seqno={}.{} len={} retry={} digest={} inbuf={}'.format(seqno, ACKED_OK, buflen, retry+1, digest.hex(), inbuf[:10]))
      else:
        logging.error(self.name+'No valid response from UDP peer: Exiting...')
        os._exit(15)
      logging.debug(self.name+'Successfully sent: seqno={} len={} retry={} digest={} inbuf={}'.format(seqno, buflen, retry, digest.hex(), inbuf[:10]))
      seqno = (seqno+1) & MAX_SEQNO_BITS
# End of UDPSend_Thread

# === Class: UDPReceive_Thread ===
class UDPReceive_Thread(threading.Thread):
  def __init__(self, name=None, args=(), kwargs=None):
    super().__init__()
    self.name = name
  def run(self):
    logging.warning(self.name+'Started.')
    expected_seqno = 0
    while True:
      data, addr = udp_sock.recvfrom(MAXBUF_SIZE+HEADER_SIZE)
      msg_type = data[:1]
      if msg_type==b'H':
        # Heartbeat message
        global HB_COUNT
        HB_COUNT = 0
        logging.debug(self.name+'Keep alive packet={}'.format(data))    # Log and ignore it
      elif msg_type in b'AN':
        # Positive or negative ack.
        global ACKED_SEQNO, ACKED_OK
        ACKED_OK = msg_type == b'A'
        ACKED_SEQNO, = struct.unpack('!H', data[1:3])
        ack_event.set()
        logging.debug(self.name+'{}tive ack. for seqno={}'.format('Posi' if ACKED_OK else 'Nega', ACKED_SEQNO))
      elif msg_type in b'CU':
        # Compressed or uncompressed data buffer
        sent_seqno, sent_buflen, sent_digest = struct.unpack('!HH{}s'.format(DIGEST_SIZE), data[1:HEADER_SIZE])
        outbuf = data[HEADER_SIZE:]
        if msg_type == b'C':
          outbuf = attempt_decompr(outbuf)
        buflen = len(outbuf)
        digest = calculate_digest(outbuf)
        if (sent_seqno==expected_seqno) and (sent_buflen==buflen) and (sent_digest==digest):
          # Everything is OK!
          try:
            tcp_conn.send(outbuf)
          except:
            logging.error(self.name+'TCP connection lost: Exiting...')
            os._exit(13)
          udp_sock.sendto(struct.pack('!sH', b'A', sent_seqno), (PEER_IP, UDP_PORT))     # Send positive ack.
          logging.debug(self.name+'Valid data: seqno={} len={} digest={} outbuf={}'.format(expected_seqno, buflen, digest.hex(), outbuf[:10]))
          expected_seqno = (expected_seqno+1) & MAX_SEQNO_BITS
        else:
          # Data error or old packet
          if expected_seqno>sent_seqno:
            # Old packet; resend positive ack.
            udp_sock.sendto(struct.pack('!sH', b'A', sent_seqno), (PEER_IP, UDP_PORT))
            packet_err = 'Old'
          else:
            # Corrupted packet; send negative ack.
            udp_sock.sendto(struct.pack('!sH', b'N', expected_seqno), (PEER_IP, UDP_PORT))
            packet_err = 'Corrupted'
          logging.warning(self.name+packet_err+' packet: sent_seqno={} ? expected_seqno={} / sent_len={} ? len={} / sent_digest={} ? digest={}'.format(
            sent_seqno, expected_seqno, sent_buflen, buflen, sent_digest.hex(), digest.hex()))
      else:
        # Message type error
        logging.warning(self.name+'Type error: msg_type={} data: sent_seqno={} ? expected_seqno={} / sent_len={} ? len={} / sent_digest={} ? digest={}'.format(
          msg_type, sent_seqno, expected_seqno, sent_buflen, buflen, sent_digest.hex(), digest.hex()))
        logging.error(self.name+'Invalid message type. Exiting...')
        os._exit(16)
# End of UDPReceive_Thread

UDPSend_Thread(name='SEND: ').start()
UDPReceive_Thread(name='RECV: ').start()