Add probe timeout metric (#1571)
Add a metric to expose the internal probe handler timeout as
probe_timeout_seconds.Related to: https://github.com/prometheus/blackbox_exporter/issues/1570
Signed-off-by: SuperQ superq@gmail.com
版权所有:中国计算机学会技术支持:开源发展技术委员会
京ICP备13000930号-9
京公网安备 11010802032778号
Blackbox exporter
The blackbox exporter allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP, ICMP and gRPC.
Running this software
From binaries
Download the most suitable binary from the releases tab
Then:
Using the docker image
Note: You may want to enable ipv6 in your docker configuration
Checking the results
Visiting http://localhost:9115/probe?target=google.com&module=http_2xx will return metrics for a HTTP probe against google.com. The
probe_successmetric indicates if the probe succeeded. Adding adebug=trueparameter will return debug information for that probe.The probe will also return timing metrics for measuring how long it takes to probe the target.
For example, you can find out how much of the probe timeout is used. This is useful because Prometheus timeouts can never be longer than the scrape interval.
This query will return the ratio of remaining time.
Metrics concerning the operation of the exporter itself are available at the endpoint http://localhost:9115/metrics.
TLS and basic authentication
The Blackbox Exporter supports TLS and basic authentication. This enables better control of the various HTTP endpoints.
To use TLS and/or basic authentication, you need to pass a configuration file using the
--web.config.fileparameter. The format of the file is described in the exporter-toolkit repository.Note that the TLS and basic authentication settings affect all HTTP endpoints: /metrics for scraping, /probe for probing, and the web UI.
Controlling log level for probe logs
The blackbox_exporter has a primary structured logger that is used for logs related to the application itself. Application logs can be controlled by
--log.level, which sets the log level for messages to output, valid levels aredebug,info,warn, anderror.The blackbox_exporter also maintains a second, fully independent structured logger that is used specifically for logging related to the probers (
http,tcp,icmp,dns,grpc) and their output. The scrape probe logger defaults toinfolevel filtering, similar to the primary logger. Prober logs can be controlled with--log.proberExample stderr and debug module output with `--log.prober=info` (default)
Example stderr and debug module output with `--log.prober=debug`
Building the software
Local Build
Building with Docker
After a successful local build:
Configuration
Blackbox exporter is configured via a configuration file and command-line flags (such as what configuration file to load, what port to listen on, and the logging format and level).
Blackbox exporter can reload its configuration file at runtime. If the new configuration is not well-formed, the changes will not be applied. A configuration reload is triggered by sending a
SIGHUPto the Blackbox exporter process or by sending a HTTP POST request to the/-/reloadendpoint.Blackbox exporter also supports automatic configuration reloading. You can enable this feature using the
--config.enable-auto-reloadflag. When enabled, the exporter will automatically check for changes to its configuration file at a specified interval in seconds. The interval can be customized with the--config.auto-reload-intervalflag, which is set to 30 (which is 30 seconds) by default.To view all available command-line flags, run
./blackbox_exporter -h.To specify which configuration file to load, use the
--config.fileflag.Additionally, an example configuration is also available.
HTTP, HTTPS (via the
httpprober), DNS, TCP socket, ICMP and gRPC (see permissions section) are currently supported. Additional modules can be defined to meet your needs.The timeout of each probe is automatically determined from the
scrape_timeoutin the Prometheus config, slightly reduced to allow for network delays. This can be further limited by thetimeoutin the Blackbox exporter config file. If neither is specified, it defaults to 120 seconds.Prometheus Configuration
Blackbox exporter implements the multi-target exporter pattern, so we advice to read the guide Understanding and using the multi-target exporter pattern to get the general idea about the configuration.
The blackbox exporter needs to be passed the target as a parameter, this can be done with relabelling.
Example config:
HTTP probes can accept an additional
hostnameparameter that will setHostheader and TLS SNI. This can be especially useful withdns_sd_config:Permissions
The ICMP probe requires elevated privileges to function:
net.ipv4.ping_group_range, theCAP_NET_RAWcapability or the root user is required.net.ipv4.ping_group_rangeby default in/etc/sysctl.confor similar. If not you can setnet.ipv4.ping_group_range = 0 2147483647to allow any user the ability to use ping.setcap cap_net_raw+ep blackbox_exporterThe UNIX probe requires the process owner to have write permissions (w) to the UNIX socket, and access permissions (x) to the directory structure the socket resides in.