はじめに
Datadog とはサーバのモニタリングなどを行ってくれるサービス。
一般的なメトリクスのモニタリング以外にも log を集約したりなど様々な機能があり、設定次第でそれぞれを有効化/無効化できます。
で今回はメトリクスのモニタリングは別でやってるので APM だけ有効にしたいんだよね〜ということをやっていきます。
Datadog 導入
まず Datadog には agent というものが存在します。
この agent が Datadog の server にメトリクスの情報を送るなどを行ってくれるものになります。
NewRelic などは gem として入れた agent が直接情報を NewRelic の server に送信しますが、Datadog は agent を別でインストールし、それに加えて gem も入れるということになります。この gem が agent に情報を送り agent が server に情報を送るという構成になっています。
APM としての機能だけではなく、server のメトリクス収集なども行ってくれるのでおそらくこういう構成になっているんでしょうね。
kubernetes に agent を install
導入方法はマニュアルの ここ に書いてありますが、k8s 上では daemonset として agent を deploy します。(見てもらえればわかりますが helm を使った install 方法なども書いてあります)
$ kubectl apply -f "https://raw.githubusercontent.com/DataDog/datadog-agent/master/Dockerfiles/manifests/rbac/clusterrole.yaml" $ kubectl apply -f "https://raw.githubusercontent.com/DataDog/datadog-agent/master/Dockerfiles/manifests/rbac/serviceaccount.yaml" $ kubectl apply -f "https://raw.githubusercontent.com/DataDog/datadog-agent/master/Dockerfiles/manifests/rbac/clusterrolebinding.yaml"
必要な RBAC を設定していきます。
次に API key の secret を作成
$ kubectl create secret generic datadog-agent --from-literal api-key="<DATADOG_API_KEY>" --namespace="default"
最後に daemonset を apply
$ cat daemonset.yaml
---
# Source: datadog/templates/install_info-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: datadog-agent-installinfo
labels: {}
annotations:
checksum/install_info: 9f58f4fe71f1b79dfabae5311eb8f5373bd03072d4636e248cd3f581f8752627
data:
install_info: |
---
install_method:
tool: kubernetes sample manifests
tool_version: kubernetes sample manifests
installer_version: kubernetes sample manifests
---
# Source: datadog/templates/daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: datadog-agent
labels: {}
spec:
selector:
matchLabels:
app: datadog-agent
template:
metadata:
labels:
app: datadog-agent
name: datadog-agent
annotations: {}
spec:
containers:
- name: agent
image: "gcr.io/datadoghq/agent:7.23.1"
imagePullPolicy: IfNotPresent
command: ["agent", "run"]
resources: {}
ports:
- containerPort: 8125
name: dogstatsdport
protocol: UDP
env:
- name: DD_API_KEY
valueFrom:
secretKeyRef:
name: "datadog-agent"
key: api-key
- name: DD_KUBERNETES_KUBELET_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: KUBERNETES
value: "yes"
- name: DOCKER_HOST
value: unix:///host/var/run/docker.sock
- name: DD_LOG_LEVEL
value: "INFO"
- name: DD_DOGSTATSD_PORT
value: "8125"
- name: DD_APM_ENABLED
value: "false"
- name: DD_LOGS_ENABLED
value: "false"
- name: DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL
value: "false"
- name: DD_LOGS_CONFIG_K8S_CONTAINER_USE_FILE
value: "true"
- name: DD_HEALTH_PORT
value: "5555"
- name: DD_ENABLE_PAYLOADS_EVENTS
value: "false"
- name: DD_ENABLE_PAYLOADS_SERIES
value: "false"
- name: DD_ENABLE_PAYLOADS_SERVICE_CHECKS
value: "false"
- name: DD_ENABLE_PAYLOADS_SKETCHES
value: "false"
- name: DD_PROCESS_AGENT_ENABLED
value: "false"
volumeMounts:
- name: installinfo
subPath: install_info
mountPath: /etc/datadog-agent/install_info
readOnly: true
- name: config
mountPath: /etc/datadog-agent
- name: runtimesocketdir
mountPath: /host/var/run
mountPropagation: None
readOnly: true
- name: procdir
mountPath: /host/proc
mountPropagation: None
readOnly: true
- name: cgroups
mountPath: /host/sys/fs/cgroup
mountPropagation: None
readOnly: true
livenessProbe:
failureThreshold: 6
httpGet:
path: /live
port: 5555
scheme: HTTP
initialDelaySeconds: 15
periodSeconds: 15
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
failureThreshold: 6
httpGet:
path: /ready
port: 5555
scheme: HTTP
initialDelaySeconds: 15
periodSeconds: 15
successThreshold: 1
timeoutSeconds: 5
- name: trace-agent
image: "gcr.io/datadoghq/agent:7.23.1"
imagePullPolicy: IfNotPresent
command: ["trace-agent", "-config=/etc/datadog-agent/datadog.yaml"]
resources: {}
ports:
- containerPort: 8126
hostPort: 8126
name: traceport
protocol: TCP
env:
- name: DD_API_KEY
valueFrom:
secretKeyRef:
name: "datadog-agent"
key: api-key
- name: DD_KUBERNETES_KUBELET_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: KUBERNETES
value: "yes"
- name: DOCKER_HOST
value: unix:///host/var/run/docker.sock
- name: DD_LOG_LEVEL
value: "INFO"
- name: DD_APM_ENABLED
value: "true"
- name: DD_APM_NON_LOCAL_TRAFFIC
value: "true"
- name: DD_APM_RECEIVER_PORT
value: "8126"
volumeMounts:
- name: config
mountPath: /etc/datadog-agent
- name: runtimesocketdir
mountPath: /host/var/run
mountPropagation: None
readOnly: true
livenessProbe:
initialDelaySeconds: 15
periodSeconds: 15
tcpSocket:
port: 8126
timeoutSeconds: 5
- name: process-agent
image: "gcr.io/datadoghq/agent:7.23.1"
imagePullPolicy: IfNotPresent
command: ["process-agent", "-config=/etc/datadog-agent/datadog.yaml"]
resources: {}
env:
- name: DD_API_KEY
valueFrom:
secretKeyRef:
name: "datadog-agent"
key: api-key
- name: DD_KUBERNETES_KUBELET_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: KUBERNETES
value: "yes"
- name: DOCKER_HOST
value: unix:///host/var/run/docker.sock
- name: DD_LOG_LEVEL
value: "INFO"
- name: DD_ORCHESTRATOR_EXPLORER_ENABLED
value: "false"
volumeMounts:
- name: config
mountPath: /etc/datadog-agent
- name: runtimesocketdir
mountPath: /host/var/run
mountPropagation: None
readOnly: true
- name: cgroups
mountPath: /host/sys/fs/cgroup
mountPropagation: None
readOnly: true
- name: passwd
mountPath: /etc/passwd
- name: procdir
mountPath: /host/proc
mountPropagation: None
readOnly: true
initContainers:
- name: init-volume
image: "gcr.io/datadoghq/agent:7.23.1"
imagePullPolicy: IfNotPresent
command: ["bash", "-c"]
args:
- cp -r /etc/datadog-agent /opt
volumeMounts:
- name: config
mountPath: /opt/datadog-agent
resources: {}
- name: init-config
image: "gcr.io/datadoghq/agent:7.23.1"
imagePullPolicy: IfNotPresent
command: ["bash", "-c"]
args:
- for script in $(find /etc/cont-init.d/ -type f -name '*.sh' | sort) ; do bash $script ; done
volumeMounts:
- name: config
mountPath: /etc/datadog-agent
- name: procdir
mountPath: /host/proc
mountPropagation: None
readOnly: true
- name: runtimesocketdir
mountPath: /host/var/run
mountPropagation: None
readOnly: true
env:
- name: DD_API_KEY
valueFrom:
secretKeyRef:
name: "datadog-agent"
key: api-key
- name: DD_KUBERNETES_KUBELET_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: KUBERNETES
value: "yes"
- name: DOCKER_HOST
value: unix:///host/var/run/docker.sock
resources: {}
volumes:
- name: installinfo
configMap:
name: datadog-agent-installinfo
- name: config
emptyDir: {}
- hostPath:
path: /var/run
name: runtimesocketdir
- hostPath:
path: /proc
name: procdir
- hostPath:
path: /sys/fs/cgroup
name: cgroups
- name: s6-run
emptyDir: {}
- hostPath:
path: /etc/passwd
name: passwd
tolerations:
affinity: {}
serviceAccountName: "datadog-agent"
nodeSelector:
kubernetes.io/os: linux
updateStrategy:
rollingUpdate:
maxUnavailable: 10%
type: RollingUpdate
# Source: datadog/templates/containers-common-env.yaml
# The purpose of this template is to define a minimal set of environment
# variables required to operate dedicated containers in the daemonset
$ kubectl apply -f daemonset.yaml元々の daemonset.yaml と比べてもらえるとわかりますがいくつかの環境変数を設定することで APM のみを有効にしています。
