I hear the misconception that a server can only accept 65K connections or a server consumes a port for each accepted connection all the time. Here is a taste of some of them:
There are 65535 TCP port numbers, does that mean only 65535 clients can connect to a TCP server? One might think that this places a hard limit on the number of clients that a single computer/application can maintain.
If there is a limit on the number of ports one machine can have and a socket can only bind to an unused port number, how do servers experiencing extremely high amounts (more than the max port number) of requests handle this? Is it just done by making the system distributed, i.e., many servers on many machines?
So I put together this article to dispel this myth from three directions:
- WhatsApp, a chatting app you have probably used, and Phoenix, a web framework built on top of Elixir, have already demonstrated millions of connections listening to a single port.
- What is theoretically possible based on the TCP/IP protocol
- A simple Java experiment anyone can run on their machine if they are still not convinced.
Jump to the summary at the end if you want to skip the details.
The Phoenix framework which achieved 2,000,000 concurrent websocket connections. In the article they demonstrate a chat application where they simulate 2 million users, taking 1 second to broadcast to all users. They also provide details on the technical challenges they hit with their framework to achieve that benchmark. Some of the ideas they shared in their article, I used to write this article, like assigning multiple IPs to overcome the 65k limit on client connections.
WhatsApp also achieved 2,000,000. Unfortunately, they are light on the details. They only reveal the hardware and the OS configuration they used.
Some think the limit 216=65,536 because that’s all the ports available in the TCP spec. That limit is true for a single client making outgoing connections to a single IP, port pair. For instance, my laptop will only be able to make 65,536 connections to 18.104.22.168:443 (google.com:443) Google would probably block me before I reached 65k connections. So if you have a use case with intense intertwined communication between two machines using more than 65K concurrent connections, the client will need to connect from a second IP address, or the server needs to make a second port available.
For a server listening on a port, each incoming connection DOES NOT consume a port on the server. The server only consumes the one port that it is listening on. Secondly connections will be coming from multiple IP addresses. In the best case the server will be able to listen to all IP addresses, coming from all ports.
Each tcp connection is uniquely defined by:
- 32 bit source IP (the IP address the connection is coming from)
- 16 bit source port (the port on the source IP address the connection is coming from)
- 32 bit destination IP (the IP address the connection is going to)
- 16 bit destination port (the port on the destination IP address the connection is going to)
Then theoretical limit a server can support on a single port is 248 which is about 1 quadrillion because:
- The server distinguishes the connections from clients’ source IPs and the source ports
- [number of source IP addresses]x[num of source ports]
- 32 bits for the address and 16 bits for the port
- Putting that all together: 232 x 216 = 248.
- Which is about a quadrillion (log(248)/log(10)=14.449)!
To understand an optimistic practical limit, I put together some experiments trying to open as many TCP connections and have the server send and receive a message on each connection. The workload is nowhere near as practical as Phoenix’s or WhatsApp’s articles but simpler to run if you wanted to try for yourself. You will need to overcome three battles to get the experiment to run: the OS, the JVM, and the TCP/IP protocol.
If you’re interested in the source code, take a look here.
The pseudo code is:
Thread 1: open server socket for i from 1 to 1 000 000: accept incoming connection for i from 1 to 1 000 000 send number i on socket i for i from 1 to 1 000 000 receive number j on socket i assert i == j Thread 2: for i from 1 to 1 000 000: open client socket to server for i from 1 to 1 000 000: receive number j on socket i assert i == j for i from 1 to 1 000 000 send number i on socket i
For my machines I tried on my Mac:
2.5 GHz Quad-Core Intel Core i7 16 GB 1600 MHz DDR3
and on my Linux desktop:
AMD FX(tm)-6300 Six-Core Processor 8GiB 1600 MHz
First battle you’ll encounter is with the operating system. The defaults severely limit file descriptors. You’ll see an error like:
Exception in thread "main" java.lang.ExceptionInInitializerError at java.base/sun.nio.ch.SocketDispatcher.close(SocketDispatcher.java:70) at java.base/sun.nio.ch.NioSocketImpl.lambda$closerFor$0(NioSocketImpl.java:1203) at java.base/jdk.internal.ref.CleanerImpl$PhantomCleanableRef.performCleanup(CleanerImpl.java:178) at java.base/jdk.internal.ref.PhantomCleanable.clean(PhantomCleanable.java:133) at java.base/sun.nio.ch.NioSocketImpl.tryClose(NioSocketImpl.java:854) at java.base/sun.nio.ch.NioSocketImpl.close(NioSocketImpl.java:906) at java.base/java.net.SocksSocketImpl.close(SocksSocketImpl.java:562) at java.base/java.net.Socket.close(Socket.java:1585) at Main.main(Main.java:123) Caused by: java.io.IOException: Too many open files at java.base/sun.nio.ch.FileDispatcherImpl.init(Native Method) at java.base/sun.nio.ch.FileDispatcherImpl.<clinit>(FileDispatcherImpl.java:38) ... 9 more
Each server socket needs two file descriptors:
- A buffer for sending
- A buffer for receiving
The same goes for client connections. As a result, running this experiment on a single machine will require:
- 1,000,000 connection for the client
- 1,000,000 connection for the server
- 2 file descriptors per connection
- = 4,000,000 file descriptors
For a bigSur 11.4 Mac, you can increase the file descriptor limit with:
sudo sysctl kern.maxfiles=2000000 kern.maxfilesperproc=2000000 kern.maxfiles: 49152 -> 2000000 kern.maxfilesperproc: 24576 -> 2000000 sysctl -a | grep maxfiles kern.maxfiles: 2000000 kern.maxfilesperproc: 1000000 ulimit -Hn 2000000 ulimit -Sn 2000000
as recommended by this stackoverflow answer.
For Ubuntu 20.04, the quickest way is to:
sudo su # 2^25 should be more than enough sysctl -w fs.nr_open=33554432 fs.nr_open = 33554432 ulimit -Hn 33554432 ulimit -Sn 33554432
Java File Descriptor Limits
Now that the operating system is complying, the JVM doesn’t like what you’re doing with this experiment either. When you run the experiment you still get the same or similar stack trace.
This stackoverflow answer identifies a JVM flag as the solution:
-XX:-MaxFDLimit: Disables the attempt to set the soft limit for the number of open file descriptors to the hard limit. By default, this option is enabled on all platforms, but is ignored on Windows. The only time that you may need to disable this is on Mac OS, where its use imposes a maximum of 10240, which is lower than the actual system maximum.
java -XX:-MaxFDLimit Main 6000
As the above quote from the javadocs says, this is only required for the Mac. I was able to get the experiment to run without the flag on Ubuntu.
It still won’t work. You will encounter a stacktrace like:
Exception in thread "main" java.net.BindException: Can't assign requested address at java.base/sun.nio.ch.Net.bind0(Native Method) at java.base/sun.nio.ch.Net.bind(Net.java:555) at java.base/sun.nio.ch.Net.bind(Net.java:544) at java.base/sun.nio.ch.NioSocketImpl.bind(NioSocketImpl.java:643) at java.base/java.net.DelegatingSocketImpl.bind(DelegatingSocketImpl.java:94) at java.base/java.net.Socket.bind(Socket.java:682) at java.base/java.net.Socket.<init>(Socket.java:506) at java.base/java.net.Socket.<init>(Socket.java:403) at Main.main(Main.java:137)
The final battle is the TCP/IP specification. Currently we have frozen the server address, server port, and client IP address. That leaves us with only 16 bits of freedom. As a result, we can only open 65k connections.
Our experiment goes way beyond that. We cannot change the server IP nor the server port because that’s the problem we’re exploring with this experiment. That leaves us with changing the client IP, giving access to an additional 32 bits of freedom. As a result, we work around this by conservatively assigning a client IP address for every 5,000 client connections. This is the same technique they used in Phoenix’s experiment.
On bigSur 11.4 you can add a bunch of fake loopback addresses with:
for i in `seq 0 200`; do sudo ifconfig lo0 alias 10.0.0.$i/8 up ; done
To test to make sure you IP addresses are working you can ping them:
for i in `seq 0 200`; do ping -c 1 10.0.0.$i ; done
for i in `seq 0 200`; do sudo ifconfig lo0 alias 10.0.0.$i ; done
On Ubuntu 20.04 you need to use the
ip tool instead:
for i in `seq 0 200`; do sudo ip addr add 10.0.0.$i/8 dev lo; done
for i in `seq 0 200`; do sudo ip addr del 10.0.0.$i/8 dev lo; done
On Mac, I was able to reach 80,000.
However, mysteriously a few minutes after completing the experiment,
my poor Mac crashes everytime without any crash reports in
/Library/Logs/DiagnosticReports so I’m not able to diagnose what happened.
The tcp send and receive buffers on my Mac are 131072 bytes:
sysctl net | grep tcp | grep -E '(recv)|(send)' net.inet.tcp.sendspace: 131072 net.inet.tcp.recvspace: 131072
So it might be because I used
80000 connections*131072 bytes per buffer * 2 input and output buffer * 2 client and server connection bytes which is about 39 GB virtual memory.
Or maybe Mac OS didn’t like me using
80,000*2*2=320,000 file descriptors.
Unfortunately, I’m not familiar with debugging on Mac without a crash report so if anyone has any resources let me know.
On Linux, I was able to reach 840,000! However, while the experiment ran, my mouse movements would take a few seconds to register on my screen. Anything beyond that and my Linux would freeze and become unresponsive.
I used these commands to get sysstat to record hardware stats on everything and then generate graphs:
sar -o out.840000.sar -A 1 3600 2>&1 > /dev/null & sadf -g out.840000.sar -- -w -r -u -n SOCK -n TCP -B -S -W > out.840000.svg
Some interesting facts:
MBmemfreebottomed out at 96 MB
MBavailwas still 1587MB
MBmemusedwas only 1602MB (19.6% of my total 8GB)
MBswpusedpeeked at 1086MB (despite memory still available)
- 1,680,483 sockets (840k server sockets and 840k client connections plus whatever else was running on my desktop)
- OS decided to start using swap a few seconds into the experiment even though I had more memory
# minimum, default and maximum memory size values (in bytes) cat /proc/sys/net/ipv4/tcp_rmem 4096 131072 6291456 cat /proc/sys/net/ipv4/tcp_wmem 4096 16384 4194304 sysctl net.ipv4.tcp_rmem net.ipv4.tcp_rmem = 4096 131072 6291456 sysctl net.ipv4.tcp_wmem net.ipv4.tcp_wmem = 4096 16384 4194304
So to support all the connections, I would need 247 GB of virtual memory!
131072 bytes for receive 16384 for write (131072+16384)*2*840000 =247 GB virtual memory
I suspect the buffers were requested, but since only 4 bytes were needed from each, only a fraction of the buffers were used. Even if I only I load 1 page of memory because I only need to write 4 bytes to write an integer to the buffer:
getconf PAGESIZE 4096 4096 bytes pagesize (4096+4096)*2*840000 =13 GB
then I use 13GB from touching
2*840000 pages of memory.
I have no idea how this didn’t crash!
I’m happy with 840,000 concurrent connections though.
You could improve upon my result if you have more memory, or further optimize the OS settings like reducing the tcp buffer sizes. How many can you run? Let me know!
- Phoenix Framework achieved 2,000,000 connections
- WhatsApp achieved 2,000,000 connections
- Theoretical limit is ~1 quadrillion (1,000,000,000,000,000)
- You will run out of source ports (only 216)
- You can fix this by add loopback client IP addresses
- You will run out of file descriptors
- You can fix this by overriding the file descriptor limits of your OS
- Java will also limit the file descriptors
- You can override this by adding the
- Practical limit on my 16GB Mac is 80,000
- Practical limit on my 8GB Linux is 840,000