I hear the misconception that a server can only accept 65K connections or a server consumes a port for each accepted connection all the time. Here is a taste of some of them:
So I put together this article to dispel this myth from three directions:
- WhatsApp, a chatting app you have probably used, and Phoenix, a web framework built on top of Elixir, have already demonstrated millions of connections listening to a single port.
- What is theoretically possible based on the TCP/IP protocol
- A simple Java experiment anyone can run on their machine if they are still not convinced.
Jump to the summary at the end if you want to skip the details.
Experiments
The Phoenix framework which achieved 2,000,000 concurrent websocket connections. In the article they demonstrate a chat application where they simulate 2 million users, taking 1 second to broadcast to all users. They also provide details on the technical challenges they hit with their framework to achieve that benchmark. Some of the ideas they shared in their article, I used to write this article, like assigning multiple IPs to overcome the 65k limit on client connections.
WhatsApp also achieved 2,000,000. Unfortunately, they are light on the details. They only reveal the hardware and the OS configuration they used.
Theoretical Max
Some think the limit 216=65,536 because that’s all the ports available in the TCP spec. That limit is true for a single client making outgoing connections to a single IP, port pair. For instance, my laptop will only be able to make 65,536 connections to 172.217.13.174:443 (google.com:443) Google would probably block me before I reached 65k connections. So if you have a use case with intense intertwined communication between two machines using more than 65K concurrent connections, the client will need to connect from a second IP address, or the server needs to make a second port available.
For a server listening on a port, each incoming connection DOES NOT consume a port on the server. The server only consumes the one port that it is listening on. Secondly connections will be coming from multiple IP addresses. In the best case the server will be able to listen to all IP addresses, coming from all ports.
Each tcp connection is uniquely defined by:
- 32 bit source IP (the IP address the connection is coming from)
- 16 bit source port (the port on the source IP address the connection is coming from)
- 32 bit destination IP (the IP address the connection is going to)
- 16 bit destination port (the port on the destination IP address the connection is going to)
Then theoretical limit a server can support on a single port is 248 which is about 1 quadrillion because:
- The server distinguishes the connections from clients’ source IPs and the source ports
- [number of source IP addresses]x[num of source ports]
- 32 bits for the address and 16 bits for the port
- Putting that all together: 232 x 216 = 248.
- Which is about a quadrillion (log(248)/log(10)=14.449)!
Practical Limit
To understand an optimistic practical limit, I put together some experiments trying to open as many TCP connections and have the server send and receive a message on each connection. The workload is nowhere near as practical as Phoenix’s or WhatsApp’s articles but simpler to run if you wanted to try for yourself. You will need to overcome three battles to get the experiment to run: the OS, the JVM, and the TCP/IP protocol.
The experiment
If you’re interested in the source code, take a look here.
The pseudo code is:
Thread 1:
open server socket
for i from 1 to 1 000 000:
accept incoming connection
for i from 1 to 1 000 000
send number i on socket i
for i from 1 to 1 000 000
receive number j on socket i
assert i == j
Thread 2:
for i from 1 to 1 000 000:
open client socket to server
for i from 1 to 1 000 000:
receive number j on socket i
assert i == j
for i from 1 to 1 000 000
send number i on socket i
The Machines
For my machines I tried on my Mac:
2.5 GHz Quad-Core Intel Core i7
16 GB 1600 MHz DDR3
and on my Linux desktop:
AMD FX(tm)-6300 Six-Core Processor
8GiB 1600 MHz
File Descriptors
First battle you’ll encounter is with the operating system. The defaults severely limit file descriptors. You’ll see an error like:
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.base/sun.nio.ch.SocketDispatcher.close(SocketDispatcher.java:70)
at java.base/sun.nio.ch.NioSocketImpl.lambda$closerFor$0(NioSocketImpl.java:1203)
at java.base/jdk.internal.ref.CleanerImpl$PhantomCleanableRef.performCleanup(CleanerImpl.java:178)
at java.base/jdk.internal.ref.PhantomCleanable.clean(PhantomCleanable.java:133)
at java.base/sun.nio.ch.NioSocketImpl.tryClose(NioSocketImpl.java:854)
at java.base/sun.nio.ch.NioSocketImpl.close(NioSocketImpl.java:906)
at java.base/java.net.SocksSocketImpl.close(SocksSocketImpl.java:562)
at java.base/java.net.Socket.close(Socket.java:1585)
at Main.main(Main.java:123)
Caused by: java.io.IOException: Too many open files
at java.base/sun.nio.ch.FileDispatcherImpl.init(Native Method)
at java.base/sun.nio.ch.FileDispatcherImpl.<clinit>(FileDispatcherImpl.java:38)
... 9 more
Each server socket needs two file descriptors:
- A buffer for sending
- A buffer for receiving
The same goes for client connections. As a result, running this experiment on a single machine will require:
- 1,000,000 connection for the client
- 1,000,000 connection for the server
- 2 file descriptors per connection
- = 4,000,000 file descriptors
For a bigSur 11.4 Mac, you can increase the file descriptor limit with:
sudo sysctl kern.maxfiles=2000000 kern.maxfilesperproc=2000000
kern.maxfiles: 49152 -> 2000000
kern.maxfilesperproc: 24576 -> 2000000
sysctl -a | grep maxfiles
kern.maxfiles: 2000000
kern.maxfilesperproc: 1000000
ulimit -Hn 2000000
ulimit -Sn 2000000
as recommended by this stackoverflow answer.
For Ubuntu 20.04, the quickest way is to:
sudo su
# 2^25 should be more than enough
sysctl -w fs.nr_open=33554432
fs.nr_open = 33554432
ulimit -Hn 33554432
ulimit -Sn 33554432
Java File Descriptor Limits
Now that the operating system is complying, the JVM doesn’t like what you’re doing with this experiment either. When you run the experiment you still get the same or similar stack trace.
This stackoverflow answer identifies a JVM flag as the solution:
java -XX:-MaxFDLimit Main 6000
As the above quote from the javadocs says, this is only required for the Mac. I was able to get the experiment to run without the flag on Ubuntu.
Source Ports
It still won’t work. You will encounter a stacktrace like:
Exception in thread "main" java.net.BindException: Can't assign requested
address
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Net.java:555)
at java.base/sun.nio.ch.Net.bind(Net.java:544)
at java.base/sun.nio.ch.NioSocketImpl.bind(NioSocketImpl.java:643)
at
java.base/java.net.DelegatingSocketImpl.bind(DelegatingSocketImpl.java:94)
at java.base/java.net.Socket.bind(Socket.java:682)
at java.base/java.net.Socket.<init>(Socket.java:506)
at java.base/java.net.Socket.<init>(Socket.java:403)
at Main.main(Main.java:137)
The final battle is the TCP/IP specification. Currently we have frozen the server address, server port, and client IP address. That leaves us with only 16 bits of freedom. As a result, we can only open 65k connections.
Our experiment goes way beyond that. We cannot change the server IP nor the server port because that’s the problem we’re exploring with this experiment. That leaves us with changing the client IP, giving access to an additional 32 bits of freedom. As a result, we work around this by conservatively assigning a client IP address for every 5,000 client connections. This is the same technique they used in Phoenix’s experiment.
On bigSur 11.4 you can add a bunch of fake loopback addresses with:
for i in `seq 0 200`; do sudo ifconfig lo0 alias 10.0.0.$i/8 up ; done
To test to make sure you IP addresses are working you can ping them:
for i in `seq 0 200`; do ping -c 1 10.0.0.$i ; done
To remove:
for i in `seq 0 200`; do sudo ifconfig lo0 alias 10.0.0.$i ; done
On Ubuntu 20.04 you need to use the ip tool instead:
for i in `seq 0 200`; do sudo ip addr add 10.0.0.$i/8 dev lo; done
To remove:
for i in `seq 0 200`; do sudo ip addr del 10.0.0.$i/8 dev lo; done
Results
On Mac, I was able to reach 80,000.
However, mysteriously a few minutes after completing the experiment,
my poor Mac crashes everytime without any crash reports in /Library/Logs/DiagnosticReports so I’m not able to diagnose what happened.
The tcp send and receive buffers on my Mac are 131072 bytes:
sysctl net | grep tcp | grep -E '(recv)|(send)'
net.inet.tcp.sendspace: 131072
net.inet.tcp.recvspace: 131072
So it might be because I used 80000 connections*131072 bytes per buffer * 2 input and output buffer * 2 client and server connection bytes which is about 39 GB virtual memory.
Or maybe Mac OS didn’t like me using 80,000*2*2=320,000 file descriptors.
Unfortunately, I’m not familiar with debugging on Mac without a crash report so if anyone has any resources let me know.
On Linux, I was able to reach 840,000! However, while the experiment ran, my mouse movements would take a few seconds to register on my screen. Anything beyond that and my Linux would freeze and become unresponsive.
I used sysstat to investigate what resource was in contention. You can take a look at the graphs sysstat generated here.
I used these commands to get sysstat to record hardware stats on everything and then generate graphs:
sar -o out.840000.sar -A 1 3600 2>&1 > /dev/null &
sadf -g out.840000.sar -- -w -r -u -n SOCK -n TCP -B -S -W > out.840000.svg
Some interesting facts:
MBmemfreebottomed out at 96 MBMBavailwas still 1587MBMBmemusedwas only 1602MB (19.6% of my total 8GB)MBswpusedpeeked at 1086MB (despite memory still available)- 1,680,483 sockets (840k server sockets and 840k client connections plus whatever else was running on my desktop)
- OS decided to start using swap a few seconds into the experiment even though I had more memory
On linux to find the default size of your send and receive buffers you can use:
# minimum, default and maximum memory size values (in bytes)
cat /proc/sys/net/ipv4/tcp_rmem
4096 131072 6291456
cat /proc/sys/net/ipv4/tcp_wmem
4096 16384 4194304
sysctl net.ipv4.tcp_rmem
net.ipv4.tcp_rmem = 4096 131072 6291456
sysctl net.ipv4.tcp_wmem
net.ipv4.tcp_wmem = 4096 16384 4194304
So to support all the connections, I would need 247 GB of virtual memory!
131072 bytes for receive
16384 for write
(131072+16384)*2*840000
=247 GB virtual memory
I suspect the buffers were requested, but since only 4 bytes were needed from each, only a fraction of the buffers were used. Even if I only I load 1 page of memory because I only need to write 4 bytes to write an integer to the buffer:
getconf PAGESIZE
4096
4096 bytes pagesize
(4096+4096)*2*840000
=13 GB
then I use 13GB from touching 2*840000 pages of memory.
I have no idea how this didn’t crash!
I’m happy with 840,000 concurrent connections though.
You could improve upon my result if you have more memory, or further optimize the OS settings like reducing the tcp buffer sizes. How many can you run? Let me know!
Summary
- Phoenix Framework achieved 2,000,000 connections
- WhatsApp achieved 2,000,000 connections
- Theoretical limit is ~1 quadrillion (1,000,000,000,000,000)
- You will run out of source ports (only 216)
- You can fix this by add loopback client IP addresses
- You will run out of file descriptors
- You can fix this by overriding the file descriptor limits of your OS
- Java will also limit the file descriptors
- You can override this by adding the
-XX:MaxFDLimitJVM argument - Practical limit on my 16GB Mac is 80,000
- Practical limit on my 8GB Linux is 840,000