Amazon EC2 + Docker + Resource temporarily unavailable

Error:

runtime/cgo: pthread_create failed: Resource temporarily unavailable

SIGABRT: abort

PC=0x7fcd25acef47 m=8 sigcode=18446744073709551610

goroutine 0 [idle]:

runtime: unknown pc 0x7fcd25acef47

stack: frame={sp:0x7fcd07ffe840, fp:0x0} stack=[0x7fcd077ff288,0x7fcd07ffee88)

Fix:

Check DefaultTasksMax

$ systemctl show --property DefaultTasksMax

Edit system.conf

$ sudo vim /etc/systemd/system.conf 

Increse DefaultTasksMax

DefaultTasksMax=15288

Restart

$ sudo systemctl daemon-reload
$ sudo systemctl daemon-reexec
$ sudo service systemd-logind restart

Check changes

$ systemctl show --property DefaultTasksMax

Add

native.cgroupdriver=cgroupfs on dockerd start config

$ vim /lib/systemd/system/docker.service

Like this

[Service]
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=cgroupfs

Increse file size limit

# edit the following file

$ sudo vim /etc/security/limits.conf

# add following lines to it

* soft     nproc          65535    
* hard     nproc          65535   
* soft     nofile         65535   
* hard     nofile         65535
root soft     nproc          65535    
root hard     nproc          65535   
root soft     nofile         65535   
root hard     nofile         65535

# edit the following file

$ sudo vim /etc/pam.d/common-session

# add this line to it

session required pam_limits.so

# logout and login and try the following command

$ ulimit -n

References

https://github.com/moby/moby/issues/16256

https://medium.com/@muhammadtriwibowo/set-permanently-ulimit-n-open-files-in-ubuntu-4d61064429a

https://www.suse.com/support/kb/doc/?id=000015901