using strace to fix issues

case: wget on SSL

Resolving www.google.com (www.google.com)... 172.217.23.228, 2a00:1450:4014:80d::2004 Connecting to www.google.com (www.google.com)|172.217.23.228|:443... connected. ERROR: cannot verify www.google.com's certificate, issued by `/C=US/O=Google Trust Services/CN=GTS CA 1O1': Unable to locally verify the issuer's authority. To connect to www.google.com insecurely, use `--no-check-certificate'.

The wget runs on a somewhat archaic Synology system, where things are not in their usual location. It is missing the certificates file. But, where it is?

The documentation says it is often in /usr/share/ca-certificates, that wget looks at $SSL_CERT_DIR env variable, and brief search did not find either. Time for the nukes.

strace to the rescue. It allows running a process while watching over its system calls.

So, let's run this:

Same result as before. But now we have a text file /tmp/x where we see, inter alia, everything the process tried to open. (Or run, which we don't look for now.)

The options are:

• -o <filename> - the output file, something convenient
• -s <length> - the limit of number of characters of data to report in the output, the default is too low
• -f - when the process we check is forking daughter subprocesses; this follows the forks too, it is not used here but is useful

The file is a lot of "garbage", heaps of unused information. So let's grep it for the juice.

We find a lot of sequences of failed file opens for libraries or other subprocesses. These happen as the operating system attempts to locate the requested file on a list of possible locations.

open("/lib/half/libdl.so.2", O_RDONLY)  = -1 ENOENT (No such file or directory)
open("/lib/libdl.so.2", O_RDONLY)       = 3
open("/opt/lib/librt.so.1", O_RDONLY)   = -1 ENOENT (No such file or directory)
open("/lib/librt.so.1", O_RDONLY)       = 3

After filtering a lot of /lib and /opt/lib searches for libraries, we come with some useful hints:

open("/opt/etc/wgetrc", O_RDONLY|O_LARGEFILE) = 3
open("/usr/lib/gconv/gconv-modules.cache", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/gconv/gconv-modules", O_RDONLY) = 3
open("/usr/share/zoneinfo/CET-1CEST,M3.5.0,M10.5.0/3", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 3
open("/opt/share/openssl/cert.pem", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/etc/nsswitch.conf", O_RDONLY)    = 3
open("/etc/resolv.conf", O_RDONLY)      = 3
open("/lib/libnss_files.so.2", O_RDONLY) = 3
open("/etc/hosts", O_RDONLY)            = 3
open("/etc/gai.conf", O_RDONLY)         = -1 ENOENT (No such file or directory)
stat64("/opt/share/openssl/certs/111e6273.0", 0xbecc3218) = -1 ENOENT (No such file or directory)

Here we see what wget is trying to find in terms of configuration files, and what she found. The important part for solving the certificate issue is

open("/opt/share/openssl/cert.pem", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)

We also find some potentially useful data, e.g. that she uses gai.conf, apparently with getaddrinfo() call from glibc. Handy for advanced networking tricks.

So now we know the /opt/share/openssl/cert.pem file is missing.

The more usual filename for the generic-for-some-certificate cert.pem is [link?:cacert.pem]. This is a bundle of certificate authority signatures for validation of site certificates. (The certs can be either each in its own file in a directory, or all in one bundle file. We want the latter.)

A good source is from the curl project, converted from the bunch shipped with Mozilla browser:

• https://curl.haxx.se/ca/cacert.pem

So let's download it, preferably on a machine that has valid root certs so we have a reasonable chance to believe it is unmodified. We can also check out https://curl.haxx.se/ca/cacert.pem.sha256 with the hash of the file above and compare it with a hash obtained by

Also possible to extract from local Firefox installation, as of this link: https://curl.haxx.se/docs/caextract.html

So save it to [link?:/opt/share/openssl/cacert.pem], symlink it to [link?:/opt/share/openssl/cert.pem], and test:

Aaaaaand, let's try:

[ <=> ] 12,563  --.-K/s in 0.04s

2020-06-14 18:39:39 (327 KB/s) - `index.html' saved [12563]

Success!

As demonstrated, strace can make the operator's job way easier.