CockroachDB に 1レコード INSERT した時にアクセスされるファイルのお話

最近話題 (?) の「DB に 1レコード INSERT した時にアクセスされるファイル」について、CockroachDB で検証してみました。

元ネタ

元ネタは以下の Blog です。
MySQLエキスパートyoku0825が目指す、DBAとしての未来像

「インストールされたばかりのMySQLがあるとして、特定テーブルに1件のレコードを最初にINSERTした場合、アクセスが発生するファイルとその理由をすべて教えてください」

カジュアルとはいえ、面談で急にこれを聞かれたらテンパる自信が有ります。

環境

今回は以下の環境 (Docker を利用したローカルの Secure クラスタ) で検証しています。

バージョン

Ubuntu : 20.04
Docker : 20.10.8
CockroachDB : 21.1.7
Container Image : cockroachdb/cockroach

クラスタ (3匹構成)

$ sudo docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS          PORTS                                                  NAMES
605c2a704bbd   cockroachdb/cockroach:v21.1.7   "/cockroach/cockroac…"   15 seconds ago   Up 14 seconds   26257/tcp, 127.0.0.1:8083->8080/tcp                    cockroach-3
054a35a7586a   cockroachdb/cockroach:v21.1.7   "/cockroach/cockroac…"   18 seconds ago   Up 18 seconds   26257/tcp, 127.0.0.1:8082->8080/tcp                    cockroach-2
dc9c30a61132   cockroachdb/cockroach:v21.1.7   "/cockroach/cockroac…"   23 seconds ago   Up 23 seconds   127.0.0.1:26257->26257/tcp, 127.0.0.1:8081->8080/tcp   cockroach-1
d75bfcddf87c   cockroachdb/cockroach:v21.1.7   "tail -f /dev/null"      26 seconds ago   Up 25 seconds   8080/tcp, 26257/tcp                                    cockroach-client

※cockroach-client は sql shell を実行するためのクライアント用コンテナ。

$ sudo docker exec -it cockroach-client bash
[root@cockroach-client cockroach]# 
[root@cockroach-client cockroach]# cockroach node status --certs-dir=certs --host=cockroach-1
  id |      address      |    sql_address    |  build  |         started_at         |         updated_at         |          locality           | is_available | is_live
-----+-------------------+-------------------+---------+----------------------------+----------------------------+-----------------------------+--------------+----------
   1 | cockroach-1:26257 | cockroach-1:26257 | v21.1.7 | 2021-08-28 06:19:05.168392 | 2021-08-28 06:19:59.219781 | region=region-1,zone=zone-a | true         | true
   2 | cockroach-2:26257 | cockroach-2:26257 | v21.1.7 | 2021-08-28 06:19:08.95911  | 2021-08-28 06:19:58.486166 | region=region-1,zone=zone-b | true         | true
   3 | cockroach-3:26257 | cockroach-3:26257 | v21.1.7 | 2021-08-28 06:19:12.531782 | 2021-08-28 06:19:57.557849 | region=region-1,zone=zone-c | true         | true
(3 rows)

CockroachDB では、cockroach start コマンドでの起動時に "-s (--store)" オプションで指定したディレクトリの中に、DB や TABLE に関連するファイル (実データ) が格納されるため、このオプションに指定したディレクトリ配下のファイルアクセスを調べます。

また、コンテナ起動時に (docker の) "-v" オプションでホスト側のディレクトリをコンテナにマウントさせているので、ファイルアクセスについては Docker ホスト側から inotifywait コマンドを使って調べています。

TABLE 作成

最初に、検証に利用する TABLE を作成しておきます。

[root@cockroach-client cockroach]# cockroach sql --certs-dir=certs --host=cockroach-1
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v21.1.7 (x86_64-unknown-linux-gnu, built 2021/08/09 17:55:28, go1.15.14) (same version as client)
# Cluster ID: e77ad798-d3bc-4bd3-91a8-960c3e7549ce
#
# Enter \? for a brief introduction.
#
root@cockroach-1:26257/defaultdb> 
root@cockroach-1:26257/defaultdb> CREATE TABLE t1 (c1 INT, c2 STRING);
CREATE TABLE

Time: 75ms total (execution 75ms / network 0ms)

root@cockroach-1:26257/defaultdb> 
root@cockroach-1:26257/defaultdb> SHOW COLUMNS FROM t1;
  column_name | data_type | is_nullable | column_default | generation_expression |  indices  | is_hidden
--------------+-----------+-------------+----------------+-----------------------+-----------+------------
  c1          | INT8      |    true     | NULL           |                       | {}        |   false
  c2          | STRING    |    true     | NULL           |                       | {}        |   false
  rowid       | INT8      |    false    | unique_rowid() |                       | {primary} |   true
(3 rows)

Time: 53ms total (execution 53ms / network 0ms)

root@cockroach-1:26257/defaultdb>

また、この TABLE の Leaseholder (Raft Leader) を確認しておきます。

root@cockroach-1:26257/defaultdb> SELECT database_name, table_name, replicas, lease_holder FROM crdb_internal.ranges WHERE table_name = 't1';
  database_name | table_name | replicas | lease_holder
----------------+------------+----------+---------------
  defaultdb     | t1         | {1,2,3}  |            1
(1 row)

Time: 14ms total (execution 13ms / network 0ms)

今回の場合、Node ID 1 (cockroach-1) が Leaseholder (Raft の Leader)、Node ID 2/Node ID 3 (cockroach-2/cockroach-3) が replica (Raft の follower) になっているようです。

本命の前に (定期的にアクセスされるファイル)

今回は「1レコード INSERT した時にアクセスされるファイル」を調べたいのですが、どうやら CockroachDB では「何もしなくても定期的にファイルアクセスが発生」しているようです (推測ですが、恐らく Raft 関連の処理や Web UI で確認できるメトリクスを保存する処理に起因してファイルアクセスが発生していると思われます)。

なので、まずは何もしてなくても定期的にアクセスされているファイルを確認しておきます (とりあえず 3分ほどファイルアクセスを確認します)。

Leaseholder (cockroach-1) で確認

# time inotifywait -mr --format '%w%f : %e' ./cockroach-1/ > file_access_cockroach-1
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
^C

real    3m14.764s
user    0m0.032s
sys     0m0.070s

cockroach-1 がアクセスしているファイル

# sort ./file_access_cockroach-1 | uniq -c
     53 ./cockroach-1/ : ACCESS,ISDIR
     41 ./cockroach-1/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-1/ : OPEN,ISDIR
   1704 ./cockroach-1/000011.log : MODIFY
     82 ./cockroach-1/auxiliary : ACCESS,ISDIR
     41 ./cockroach-1/auxiliary : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-1/auxiliary : OPEN,ISDIR
     82 ./cockroach-1/auxiliary/ : ACCESS,ISDIR
     41 ./cockroach-1/auxiliary/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-1/auxiliary/ : OPEN,ISDIR
     82 ./cockroach-1/cockroach-temp755268128 : ACCESS,ISDIR
     41 ./cockroach-1/cockroach-temp755268128 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-1/cockroach-temp755268128 : OPEN,ISDIR
     82 ./cockroach-1/cockroach-temp755268128/ : ACCESS,ISDIR
     41 ./cockroach-1/cockroach-temp755268128/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-1/cockroach-temp755268128/ : OPEN,ISDIR
     82 ./cockroach-1/cockroach-temp755268128/auxiliary : ACCESS,ISDIR
     41 ./cockroach-1/cockroach-temp755268128/auxiliary : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-1/cockroach-temp755268128/auxiliary : OPEN,ISDIR
     82 ./cockroach-1/cockroach-temp755268128/auxiliary/ : ACCESS,ISDIR
     41 ./cockroach-1/cockroach-temp755268128/auxiliary/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-1/cockroach-temp755268128/auxiliary/ : OPEN,ISDIR
     82 ./cockroach-1/logs : ACCESS,ISDIR
     41 ./cockroach-1/logs : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-1/logs : OPEN,ISDIR
     82 ./cockroach-1/logs/ : ACCESS,ISDIR
     41 ./cockroach-1/logs/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-1/logs/ : OPEN,ISDIR
     19 ./cockroach-1/logs/cockroach.cockroach-1.root.2021-08-28T06_19_03Z.000001.log : MODIFY
     82 ./cockroach-1/logs/goroutine_dump : ACCESS,ISDIR
     41 ./cockroach-1/logs/goroutine_dump : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-1/logs/goroutine_dump : OPEN,ISDIR
     82 ./cockroach-1/logs/goroutine_dump/ : ACCESS,ISDIR
     41 ./cockroach-1/logs/goroutine_dump/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-1/logs/goroutine_dump/ : OPEN,ISDIR
     82 ./cockroach-1/logs/heap_profiler : ACCESS,ISDIR
     41 ./cockroach-1/logs/heap_profiler : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-1/logs/heap_profiler : OPEN,ISDIR
     82 ./cockroach-1/logs/heap_profiler/ : ACCESS,ISDIR
     41 ./cockroach-1/logs/heap_profiler/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-1/logs/heap_profiler/ : OPEN,ISDIR

replica (cockroach-2) で確認

# time inotifywait -mr --format '%w%f : %e' ./cockroach-2/ > file_access_cockroach-2
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
^C

real    3m15.499s
user    0m0.073s
sys     0m0.092s

cockroach-2 がアクセスしているファイル

# sort ./file_access_cockroach-2 | uniq -c
     55 ./cockroach-2/ : ACCESS,ISDIR
     41 ./cockroach-2/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/ : OPEN,ISDIR
   1694 ./cockroach-2/000378.log : MODIFY
     82 ./cockroach-2/auxiliary : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/1 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/1 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/1 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/1/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/1/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/1/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/10 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/10 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/10 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/10/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/10/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/10/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/11 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/11 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/11 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/11/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/11/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/11/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/12 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/12 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/12 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/12/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/12/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/12/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/13 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/13 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/13 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/13/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/13/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/13/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/14 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/14 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/14 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/14/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/14/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/14/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/15 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/15 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/15 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/15/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/15/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/15/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/16 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/16 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/16 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/16/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/16/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/16/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/17 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/17 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/17 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/17/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/17/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/17/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/18 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/18 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/18 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/18/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/18/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/18/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/19 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/19 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/19 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/19/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/19/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/19/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/2 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/2 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/2 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/2/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/2/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/2/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/20 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/20 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/20 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/20/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/20/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/20/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/21 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/21 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/21 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/21/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/21/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/21/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/22 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/22 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/22 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/22/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/22/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/22/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/23 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/23 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/23 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/23/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/23/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/23/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/24 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/24 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/24 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/24/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/24/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/24/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/25 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/25 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/25 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/25/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/25/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/25/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/26 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/26 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/26 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/26/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/26/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/26/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/27 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/27 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/27 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/27/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/27/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/27/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/28 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/28 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/28 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/28/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/28/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/28/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/29 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/29 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/29 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/29/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/29/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/29/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/3 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/3 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/3 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/3/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/3/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/3/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/30 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/30 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/30 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/30/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/30/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/30/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/31 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/31 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/31 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/31/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/31/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/31/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/32 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/32 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/32 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/32/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/32/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/32/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/33 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/33 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/33 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/33/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/33/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/33/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/34 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/34 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/34 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/34/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/34/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/34/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/35 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/35 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/35 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/35/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/35/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/35/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/36 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/36 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/36 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/36/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/36/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/36/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/4 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/4 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/4 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/4/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/4/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/4/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/5 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/5 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/5 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/5/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/5/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/5/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/6 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/6 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/6 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/6/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/6/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/6/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/7 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/7 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/7 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/7/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/7/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/7/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/8 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/8 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/8 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/8/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/8/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/8/ : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/9 : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/9 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/9 : OPEN,ISDIR
     82 ./cockroach-2/auxiliary/sstsnapshot/9/ : ACCESS,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/9/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/auxiliary/sstsnapshot/9/ : OPEN,ISDIR
     82 ./cockroach-2/cockroach-temp575432717 : ACCESS,ISDIR
     41 ./cockroach-2/cockroach-temp575432717 : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/cockroach-temp575432717 : OPEN,ISDIR
     82 ./cockroach-2/cockroach-temp575432717/ : ACCESS,ISDIR
     41 ./cockroach-2/cockroach-temp575432717/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/cockroach-temp575432717/ : OPEN,ISDIR
     83 ./cockroach-2/cockroach-temp575432717/auxiliary : ACCESS,ISDIR
     41 ./cockroach-2/cockroach-temp575432717/auxiliary : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/cockroach-temp575432717/auxiliary : OPEN,ISDIR
     83 ./cockroach-2/cockroach-temp575432717/auxiliary/ : ACCESS,ISDIR
     41 ./cockroach-2/cockroach-temp575432717/auxiliary/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/cockroach-temp575432717/auxiliary/ : OPEN,ISDIR
     82 ./cockroach-2/logs : ACCESS,ISDIR
     41 ./cockroach-2/logs : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/logs : OPEN,ISDIR
     82 ./cockroach-2/logs/ : ACCESS,ISDIR
     41 ./cockroach-2/logs/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/logs/ : OPEN,ISDIR
     19 ./cockroach-2/logs/cockroach.cockroach-2.root.2021-08-28T06_19_08Z.000001.log : MODIFY
     82 ./cockroach-2/logs/goroutine_dump : ACCESS,ISDIR
     41 ./cockroach-2/logs/goroutine_dump : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/logs/goroutine_dump : OPEN,ISDIR
     82 ./cockroach-2/logs/goroutine_dump/ : ACCESS,ISDIR
     41 ./cockroach-2/logs/goroutine_dump/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/logs/goroutine_dump/ : OPEN,ISDIR
     82 ./cockroach-2/logs/heap_profiler : ACCESS,ISDIR
     41 ./cockroach-2/logs/heap_profiler : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/logs/heap_profiler : OPEN,ISDIR
     82 ./cockroach-2/logs/heap_profiler/ : ACCESS,ISDIR
     41 ./cockroach-2/logs/heap_profiler/ : CLOSE_NOWRITE,CLOSE,ISDIR
     41 ./cockroach-2/logs/heap_profiler/ : OPEN,ISDIR

replica (cockroach-3) で確認

# time inotifywait -mr --format '%w%f : %e' ./cockroach-3/ > file_access_cockroach-3
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
^C

real    3m17.032s
user    0m0.052s
sys     0m0.116s

cockroach-3 がアクセスしているファイル

# sort ./file_access_cockroach-3 | uniq -c
     61 ./cockroach-3/ : ACCESS,ISDIR
     43 ./cockroach-3/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/ : OPEN,ISDIR
   1712 ./cockroach-3/000379.log : MODIFY
     86 ./cockroach-3/auxiliary : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/1 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/1 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/1 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/1/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/1/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/1/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/10 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/10 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/10 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/10/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/10/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/10/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/11 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/11 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/11 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/11/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/11/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/11/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/12 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/12 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/12 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/12/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/12/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/12/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/13 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/13 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/13 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/13/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/13/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/13/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/14 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/14 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/14 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/14/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/14/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/14/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/15 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/15 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/15 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/15/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/15/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/15/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/16 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/16 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/16 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/16/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/16/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/16/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/17 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/17 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/17 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/17/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/17/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/17/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/18 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/18 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/18 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/18/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/18/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/18/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/19 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/19 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/19 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/19/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/19/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/19/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/2 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/2 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/2 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/2/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/2/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/2/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/20 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/20 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/20 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/20/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/20/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/20/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/21 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/21 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/21 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/21/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/21/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/21/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/22 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/22 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/22 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/22/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/22/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/22/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/23 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/23 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/23 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/23/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/23/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/23/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/24 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/24 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/24 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/24/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/24/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/24/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/25 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/25 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/25 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/25/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/25/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/25/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/26 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/26 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/26 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/26/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/26/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/26/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/27 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/27 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/27 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/27/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/27/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/27/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/28 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/28 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/28 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/28/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/28/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/28/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/29 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/29 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/29 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/29/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/29/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/29/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/3 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/3 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/3 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/3/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/3/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/3/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/30 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/30 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/30 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/30/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/30/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/30/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/31 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/31 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/31 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/31/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/31/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/31/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/32 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/32 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/32 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/32/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/32/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/32/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/33 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/33 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/33 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/33/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/33/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/33/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/34 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/34 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/34 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/34/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/34/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/34/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/35 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/35 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/35 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/35/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/35/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/35/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/36 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/36 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/36 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/36/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/36/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/36/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/4 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/4 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/4 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/4/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/4/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/4/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/5 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/5 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/5 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/5/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/5/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/5/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/6 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/6 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/6 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/6/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/6/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/6/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/7 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/7 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/7 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/7/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/7/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/7/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/8 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/8 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/8 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/8/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/8/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/8/ : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/9 : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/9 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/9 : OPEN,ISDIR
     86 ./cockroach-3/auxiliary/sstsnapshot/9/ : ACCESS,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/9/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/auxiliary/sstsnapshot/9/ : OPEN,ISDIR
     86 ./cockroach-3/cockroach-temp395666710 : ACCESS,ISDIR
     43 ./cockroach-3/cockroach-temp395666710 : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/cockroach-temp395666710 : OPEN,ISDIR
     86 ./cockroach-3/cockroach-temp395666710/ : ACCESS,ISDIR
     43 ./cockroach-3/cockroach-temp395666710/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/cockroach-temp395666710/ : OPEN,ISDIR
     86 ./cockroach-3/cockroach-temp395666710/auxiliary : ACCESS,ISDIR
     43 ./cockroach-3/cockroach-temp395666710/auxiliary : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/cockroach-temp395666710/auxiliary : OPEN,ISDIR
     86 ./cockroach-3/cockroach-temp395666710/auxiliary/ : ACCESS,ISDIR
     43 ./cockroach-3/cockroach-temp395666710/auxiliary/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/cockroach-temp395666710/auxiliary/ : OPEN,ISDIR
     86 ./cockroach-3/logs : ACCESS,ISDIR
     43 ./cockroach-3/logs : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/logs : OPEN,ISDIR
     86 ./cockroach-3/logs/ : ACCESS,ISDIR
     43 ./cockroach-3/logs/ : CLOSE_NOWRITE,CLOSE,ISDIR
     43 ./cockroach-3/logs/ : OPEN,ISDIR
     20 ./cockroach-3/logs/cockroach.cockroach-3.root.2021-08-28T06_19_12Z.000001.log : MODIFY
     84 ./cockroach-3/logs/goroutine_dump : ACCESS,ISDIR
     42 ./cockroach-3/logs/goroutine_dump : CLOSE_NOWRITE,CLOSE,ISDIR
     42 ./cockroach-3/logs/goroutine_dump : OPEN,ISDIR
     84 ./cockroach-3/logs/goroutine_dump/ : ACCESS,ISDIR
     42 ./cockroach-3/logs/goroutine_dump/ : CLOSE_NOWRITE,CLOSE,ISDIR
     42 ./cockroach-3/logs/goroutine_dump/ : OPEN,ISDIR
     88 ./cockroach-3/logs/heap_profiler : ACCESS,ISDIR
     44 ./cockroach-3/logs/heap_profiler : CLOSE_NOWRITE,CLOSE,ISDIR
     44 ./cockroach-3/logs/heap_profiler : OPEN,ISDIR
     88 ./cockroach-3/logs/heap_profiler/ : ACCESS,ISDIR
     44 ./cockroach-3/logs/heap_profiler/ : CLOSE_NOWRITE,CLOSE,ISDIR
     44 ./cockroach-3/logs/heap_profiler/ : OPEN,ISDIR
      1 ./cockroach-3/logs/heap_profiler/memprof.2021-08-28T06_20_32.541.47427880.pprof : DELETE
      1 ./cockroach-3/logs/heap_profiler/memprof.2021-08-28T06_33_22.550.56282280.pprof : CLOSE_WRITE,CLOSE
      1 ./cockroach-3/logs/heap_profiler/memprof.2021-08-28T06_33_22.550.56282280.pprof : CREATE
    309 ./cockroach-3/logs/heap_profiler/memprof.2021-08-28T06_33_22.550.56282280.pprof : MODIFY
      1 ./cockroach-3/logs/heap_profiler/memprof.2021-08-28T06_33_22.550.56282280.pprof : OPEN

cockroach-1 (Leaseholder) と比べて、cockroach-2/cockroach-3 (Replica) の方がアクセスされているファイルが多いように見えます。この差分も気になるところですが、今回の本題からはそれるので一旦気にしないでおきます。

本命 (INSERT 時にアクセスしているファイル)

それでは、本命について調べてみます。先程と同じように、ホスト側で inotfywait を実行した上で、INSERT を実行します。

INSERT 文

root@cockroach-1:26257/defaultdb> BEGIN;
BEGIN

Time: 0ms total (execution 1ms / network 0ms)

root@cockroach-1:26257/defaultdb  OPEN> 
root@cockroach-1:26257/defaultdb  OPEN> INSERT INTO t1 VALUES (111, 'aaa');
INSERT 1

Time: 2ms total (execution 2ms / network 0ms)

root@cockroach-1:26257/defaultdb  OPEN> 
root@cockroach-1:26257/defaultdb  OPEN> COMMIT;
COMMIT

Time: 14ms total (execution 14ms / network 0ms)

root@cockroach-1:26257/defaultdb>

Leaseholder (cockroach-1)

# time inotifywait -mr --format '%w%f : %e' ./cockroach-1/ > INSERT_file_access_cockroach-1
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
^C

real    3m59.656s
user    0m0.040s
sys     0m0.105s

INSERT 時に cockroach-1 がアクセスしているファイル

# sort ./INSERT_file_access_cockroach-1 | uniq -c
     67 ./cockroach-1/ : ACCESS,ISDIR
     52 ./cockroach-1/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-1/ : OPEN,ISDIR
      1 ./cockroach-1/000002.log : MOVED_FROM
      1 ./cockroach-1/000011.log : CLOSE_WRITE,CLOSE
    163 ./cockroach-1/000011.log : MODIFY
     84 ./cockroach-1/000013.sst : ACCESS
      3 ./cockroach-1/000013.sst : CLOSE_NOWRITE,CLOSE
      1 ./cockroach-1/000013.sst : DELETE
      2 ./cockroach-1/000013.sst : OPEN
   1966 ./cockroach-1/000014.log : MODIFY
      1 ./cockroach-1/000014.log : MOVED_TO
      1 ./cockroach-1/000014.log : OPEN
     77 ./cockroach-1/000015.sst : ACCESS
      2 ./cockroach-1/000015.sst : CLOSE_NOWRITE,CLOSE
      1 ./cockroach-1/000015.sst : CLOSE_WRITE,CLOSE
      1 ./cockroach-1/000015.sst : CREATE
      1 ./cockroach-1/000015.sst : DELETE
     65 ./cockroach-1/000015.sst : MODIFY
      3 ./cockroach-1/000015.sst : OPEN
     20 ./cockroach-1/000016.sst : ACCESS
      1 ./cockroach-1/000016.sst : CLOSE_WRITE,CLOSE
      1 ./cockroach-1/000016.sst : CREATE
    134 ./cockroach-1/000016.sst : MODIFY
      2 ./cockroach-1/000016.sst : OPEN
      2 ./cockroach-1/MANIFEST-000001 : MODIFY
    104 ./cockroach-1/auxiliary : ACCESS,ISDIR
     52 ./cockroach-1/auxiliary : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-1/auxiliary : OPEN,ISDIR
    104 ./cockroach-1/auxiliary/ : ACCESS,ISDIR
     52 ./cockroach-1/auxiliary/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-1/auxiliary/ : OPEN,ISDIR
    105 ./cockroach-1/cockroach-temp755268128 : ACCESS,ISDIR
     52 ./cockroach-1/cockroach-temp755268128 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-1/cockroach-temp755268128 : OPEN,ISDIR
    105 ./cockroach-1/cockroach-temp755268128/ : ACCESS,ISDIR
     52 ./cockroach-1/cockroach-temp755268128/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-1/cockroach-temp755268128/ : OPEN,ISDIR
    104 ./cockroach-1/cockroach-temp755268128/auxiliary : ACCESS,ISDIR
     52 ./cockroach-1/cockroach-temp755268128/auxiliary : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-1/cockroach-temp755268128/auxiliary : OPEN,ISDIR
    104 ./cockroach-1/cockroach-temp755268128/auxiliary/ : ACCESS,ISDIR
     52 ./cockroach-1/cockroach-temp755268128/auxiliary/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-1/cockroach-temp755268128/auxiliary/ : OPEN,ISDIR
    104 ./cockroach-1/logs : ACCESS,ISDIR
     52 ./cockroach-1/logs : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-1/logs : OPEN,ISDIR
    104 ./cockroach-1/logs/ : ACCESS,ISDIR
     52 ./cockroach-1/logs/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-1/logs/ : OPEN,ISDIR
      1 ./cockroach-1/logs/cockroach-pebble.cockroach-1.root.2021-08-28T06_19_03Z.000001.log : MODIFY
     24 ./cockroach-1/logs/cockroach.cockroach-1.root.2021-08-28T06_19_03Z.000001.log : MODIFY
    104 ./cockroach-1/logs/goroutine_dump : ACCESS,ISDIR
     52 ./cockroach-1/logs/goroutine_dump : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-1/logs/goroutine_dump : OPEN,ISDIR
    104 ./cockroach-1/logs/goroutine_dump/ : ACCESS,ISDIR
     52 ./cockroach-1/logs/goroutine_dump/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-1/logs/goroutine_dump/ : OPEN,ISDIR
    114 ./cockroach-1/logs/heap_profiler : ACCESS,ISDIR
     57 ./cockroach-1/logs/heap_profiler : CLOSE_NOWRITE,CLOSE,ISDIR
     57 ./cockroach-1/logs/heap_profiler : OPEN,ISDIR
    114 ./cockroach-1/logs/heap_profiler/ : ACCESS,ISDIR
     57 ./cockroach-1/logs/heap_profiler/ : CLOSE_NOWRITE,CLOSE,ISDIR
     57 ./cockroach-1/logs/heap_profiler/ : OPEN,ISDIR
      1 ./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_39_15.201.253542400.txt : DELETE
      1 ./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_39_25.201.254353408.txt : DELETE
      1 ./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_05.202.259219456.txt : CLOSE_WRITE,CLOSE
      1 ./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_05.202.259219456.txt : CREATE
      1 ./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_05.202.259219456.txt : MODIFY
      1 ./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_05.202.259219456.txt : OPEN
      1 ./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_15.202.260571136.txt : CLOSE_WRITE,CLOSE
      1 ./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_15.202.260571136.txt : CREATE
      1 ./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_15.202.260571136.txt : MODIFY
      1 ./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_15.202.260571136.txt : OPEN

cockroach-1 での INSERT 時と平常時の差分

# diff -u <(sort ./file_access_cockroach-1 | uniq) <(sort ./INSERT_file_access_cockroach-1 | uniq)
--- /dev/fd/63    2021-08-28 15:46:27.767770309 +0900
+++ /dev/fd/62    2021-08-28 15:46:27.771772598 +0900
@@ -1,7 +1,29 @@
 ./cockroach-1/ : ACCESS,ISDIR
 ./cockroach-1/ : CLOSE_NOWRITE,CLOSE,ISDIR
 ./cockroach-1/ : OPEN,ISDIR
+./cockroach-1/000002.log : MOVED_FROM
+./cockroach-1/000011.log : CLOSE_WRITE,CLOSE
 ./cockroach-1/000011.log : MODIFY
+./cockroach-1/000013.sst : ACCESS
+./cockroach-1/000013.sst : CLOSE_NOWRITE,CLOSE
+./cockroach-1/000013.sst : DELETE
+./cockroach-1/000013.sst : OPEN
+./cockroach-1/000014.log : MODIFY
+./cockroach-1/000014.log : MOVED_TO
+./cockroach-1/000014.log : OPEN
+./cockroach-1/000015.sst : ACCESS
+./cockroach-1/000015.sst : CLOSE_NOWRITE,CLOSE
+./cockroach-1/000015.sst : CLOSE_WRITE,CLOSE
+./cockroach-1/000015.sst : CREATE
+./cockroach-1/000015.sst : DELETE
+./cockroach-1/000015.sst : MODIFY
+./cockroach-1/000015.sst : OPEN
+./cockroach-1/000016.sst : ACCESS
+./cockroach-1/000016.sst : CLOSE_WRITE,CLOSE
+./cockroach-1/000016.sst : CREATE
+./cockroach-1/000016.sst : MODIFY
+./cockroach-1/000016.sst : OPEN
+./cockroach-1/MANIFEST-000001 : MODIFY
 ./cockroach-1/auxiliary : ACCESS,ISDIR
 ./cockroach-1/auxiliary : CLOSE_NOWRITE,CLOSE,ISDIR
 ./cockroach-1/auxiliary : OPEN,ISDIR
@@ -26,6 +48,7 @@
 ./cockroach-1/logs/ : ACCESS,ISDIR
 ./cockroach-1/logs/ : CLOSE_NOWRITE,CLOSE,ISDIR
 ./cockroach-1/logs/ : OPEN,ISDIR
+./cockroach-1/logs/cockroach-pebble.cockroach-1.root.2021-08-28T06_19_03Z.000001.log : MODIFY
 ./cockroach-1/logs/cockroach.cockroach-1.root.2021-08-28T06_19_03Z.000001.log : MODIFY
 ./cockroach-1/logs/goroutine_dump : ACCESS,ISDIR
 ./cockroach-1/logs/goroutine_dump : CLOSE_NOWRITE,CLOSE,ISDIR
@@ -39,3 +62,13 @@
 ./cockroach-1/logs/heap_profiler/ : ACCESS,ISDIR
 ./cockroach-1/logs/heap_profiler/ : CLOSE_NOWRITE,CLOSE,ISDIR
 ./cockroach-1/logs/heap_profiler/ : OPEN,ISDIR
+./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_39_15.201.253542400.txt : DELETE
+./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_39_25.201.254353408.txt : DELETE
+./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_05.202.259219456.txt : CLOSE_WRITE,CLOSE
+./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_05.202.259219456.txt : CREATE
+./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_05.202.259219456.txt : MODIFY
+./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_05.202.259219456.txt : OPEN
+./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_15.202.260571136.txt : CLOSE_WRITE,CLOSE
+./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_15.202.260571136.txt : CREATE
+./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_15.202.260571136.txt : MODIFY
+./cockroach-1/logs/heap_profiler/memstats.2021-08-28T06_40_15.202.260571136.txt : OPEN

replica (cockroach-2)

# time inotifywait -mr --format '%w%f : %e' ./cockroach-2/ > INSERT_file_access_cockroach-2
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
^C

real    3m58.508s
user    0m0.092s
sys     0m0.123s

INSERT 時に cockroach-2 がアクセスしているファイル

# sort ./INSERT_file_access_cockroach-2 | uniq -c
     74 ./cockroach-2/ : ACCESS,ISDIR
     52 ./cockroach-2/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/ : OPEN,ISDIR
      1 ./cockroach-2/000367.log : MOVED_FROM
      1 ./cockroach-2/000378.log : CLOSE_WRITE,CLOSE
    591 ./cockroach-2/000378.log : MODIFY
     88 ./cockroach-2/000380.sst : ACCESS
      3 ./cockroach-2/000380.sst : CLOSE_NOWRITE,CLOSE
      1 ./cockroach-2/000380.sst : DELETE
      2 ./cockroach-2/000380.sst : OPEN
   1504 ./cockroach-2/000381.log : MODIFY
      1 ./cockroach-2/000381.log : MOVED_TO
      1 ./cockroach-2/000381.log : OPEN
     78 ./cockroach-2/000382.sst : ACCESS
      2 ./cockroach-2/000382.sst : CLOSE_NOWRITE,CLOSE
      1 ./cockroach-2/000382.sst : CLOSE_WRITE,CLOSE
      1 ./cockroach-2/000382.sst : CREATE
      1 ./cockroach-2/000382.sst : DELETE
     68 ./cockroach-2/000382.sst : MODIFY
      3 ./cockroach-2/000382.sst : OPEN
     13 ./cockroach-2/000383.sst : ACCESS
      1 ./cockroach-2/000383.sst : CLOSE_WRITE,CLOSE
      1 ./cockroach-2/000383.sst : CREATE
    130 ./cockroach-2/000383.sst : MODIFY
      2 ./cockroach-2/000383.sst : OPEN
      2 ./cockroach-2/MANIFEST-000001 : MODIFY
    104 ./cockroach-2/auxiliary : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/1 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/1 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/1 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/1/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/1/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/1/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/10 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/10 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/10 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/10/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/10/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/10/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/11 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/11 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/11 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/11/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/11/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/11/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/12 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/12 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/12 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/12/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/12/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/12/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/13 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/13 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/13 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/13/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/13/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/13/ : OPEN,ISDIR
    103 ./cockroach-2/auxiliary/sstsnapshot/14 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/14 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/14 : OPEN,ISDIR
    103 ./cockroach-2/auxiliary/sstsnapshot/14/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/14/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/14/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/15 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/15 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/15 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/15/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/15/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/15/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/16 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/16 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/16 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/16/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/16/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/16/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/17 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/17 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/17 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/17/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/17/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/17/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/18 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/18 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/18 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/18/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/18/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/18/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/19 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/19 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/19 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/19/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/19/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/19/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/2 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/2 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/2 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/2/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/2/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/2/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/20 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/20 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/20 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/20/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/20/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/20/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/21 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/21 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/21 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/21/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/21/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/21/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/22 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/22 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/22 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/22/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/22/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/22/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/23 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/23 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/23 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/23/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/23/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/23/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/24 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/24 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/24 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/24/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/24/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/24/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/25 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/25 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/25 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/25/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/25/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/25/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/26 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/26 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/26 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/26/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/26/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/26/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/27 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/27 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/27 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/27/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/27/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/27/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/28 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/28 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/28 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/28/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/28/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/28/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/29 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/29 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/29 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/29/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/29/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/29/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/3 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/3 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/3 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/3/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/3/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/3/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/30 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/30 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/30 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/30/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/30/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/30/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/31 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/31 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/31 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/31/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/31/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/31/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/32 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/32 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/32 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/32/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/32/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/32/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/33 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/33 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/33 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/33/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/33/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/33/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/34 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/34 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/34 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/34/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/34/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/34/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/35 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/35 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/35 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/35/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/35/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/35/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/36 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/36 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/36 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/36/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/36/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/36/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/4 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/4 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/4 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/4/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/4/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/4/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/5 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/5 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/5 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/5/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/5/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/5/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/6 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/6 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/6 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/6/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/6/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/6/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/7 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/7 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/7 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/7/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/7/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/7/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/8 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/8 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/8 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/8/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/8/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/8/ : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/9 : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/9 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/9 : OPEN,ISDIR
    104 ./cockroach-2/auxiliary/sstsnapshot/9/ : ACCESS,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/9/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/auxiliary/sstsnapshot/9/ : OPEN,ISDIR
    104 ./cockroach-2/cockroach-temp575432717 : ACCESS,ISDIR
     52 ./cockroach-2/cockroach-temp575432717 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/cockroach-temp575432717 : OPEN,ISDIR
    104 ./cockroach-2/cockroach-temp575432717/ : ACCESS,ISDIR
     52 ./cockroach-2/cockroach-temp575432717/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/cockroach-temp575432717/ : OPEN,ISDIR
    104 ./cockroach-2/cockroach-temp575432717/auxiliary : ACCESS,ISDIR
     52 ./cockroach-2/cockroach-temp575432717/auxiliary : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/cockroach-temp575432717/auxiliary : OPEN,ISDIR
    104 ./cockroach-2/cockroach-temp575432717/auxiliary/ : ACCESS,ISDIR
     52 ./cockroach-2/cockroach-temp575432717/auxiliary/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/cockroach-temp575432717/auxiliary/ : OPEN,ISDIR
    104 ./cockroach-2/logs : ACCESS,ISDIR
     52 ./cockroach-2/logs : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/logs : OPEN,ISDIR
    104 ./cockroach-2/logs/ : ACCESS,ISDIR
     52 ./cockroach-2/logs/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/logs/ : OPEN,ISDIR
      1 ./cockroach-2/logs/cockroach-pebble.cockroach-2.root.2021-08-28T06_19_08Z.000001.log : MODIFY
     24 ./cockroach-2/logs/cockroach.cockroach-2.root.2021-08-28T06_19_08Z.000001.log : MODIFY
    104 ./cockroach-2/logs/goroutine_dump : ACCESS,ISDIR
     52 ./cockroach-2/logs/goroutine_dump : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/logs/goroutine_dump : OPEN,ISDIR
    104 ./cockroach-2/logs/goroutine_dump/ : ACCESS,ISDIR
     52 ./cockroach-2/logs/goroutine_dump/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-2/logs/goroutine_dump/ : OPEN,ISDIR
    120 ./cockroach-2/logs/heap_profiler : ACCESS,ISDIR
     60 ./cockroach-2/logs/heap_profiler : CLOSE_NOWRITE,CLOSE,ISDIR
     60 ./cockroach-2/logs/heap_profiler : OPEN,ISDIR
    120 ./cockroach-2/logs/heap_profiler/ : ACCESS,ISDIR
     60 ./cockroach-2/logs/heap_profiler/ : CLOSE_NOWRITE,CLOSE,ISDIR
     60 ./cockroach-2/logs/heap_profiler/ : OPEN,ISDIR
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_30_08.976.244809728.txt : DELETE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_30_18.976.246157312.txt : DELETE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_39_38.983.246288384.txt : DELETE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_39_48.984.247365632.txt : DELETE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_39_58.984.248446976.txt : DELETE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_08.984.249528320.txt : CLOSE_WRITE,CLOSE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_08.984.249528320.txt : CREATE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_08.984.249528320.txt : DELETE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_08.984.249528320.txt : MODIFY
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_08.984.249528320.txt : OPEN
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_18.985.250609664.txt : CLOSE_WRITE,CLOSE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_18.985.250609664.txt : CREATE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_18.985.250609664.txt : DELETE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_18.985.250609664.txt : MODIFY
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_18.985.250609664.txt : OPEN
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_28.985.251420672.txt : CLOSE_WRITE,CLOSE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_28.985.251420672.txt : CREATE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_28.985.251420672.txt : MODIFY
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_28.985.251420672.txt : OPEN
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_38.985.252772352.txt : CLOSE_WRITE,CLOSE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_38.985.252772352.txt : CREATE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_38.985.252772352.txt : MODIFY
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_38.985.252772352.txt : OPEN
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_48.985.253313024.txt : CLOSE_WRITE,CLOSE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_48.985.253313024.txt : CREATE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_48.985.253313024.txt : MODIFY
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_48.985.253313024.txt : OPEN
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_58.985.254394368.txt : CLOSE_WRITE,CLOSE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_58.985.254394368.txt : CREATE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_58.985.254394368.txt : MODIFY
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_58.985.254394368.txt : OPEN
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_41_08.985.255471616.txt : CLOSE_WRITE,CLOSE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_41_08.985.255471616.txt : CREATE
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_41_08.985.255471616.txt : MODIFY
      1 ./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_41_08.985.255471616.txt : OPEN

cockroach-2 での INSERT 時と平常時の差分

# diff -u <(sort ./file_access_cockroach-2 | uniq) <(sort ./INSERT_file_access_cockroach-2 | uniq)
--- /dev/fd/63    2021-08-28 15:49:21.260992890 +0900
+++ /dev/fd/62    2021-08-28 15:49:21.264996153 +0900
@@ -1,7 +1,29 @@
 ./cockroach-2/ : ACCESS,ISDIR
 ./cockroach-2/ : CLOSE_NOWRITE,CLOSE,ISDIR
 ./cockroach-2/ : OPEN,ISDIR
+./cockroach-2/000367.log : MOVED_FROM
+./cockroach-2/000378.log : CLOSE_WRITE,CLOSE
 ./cockroach-2/000378.log : MODIFY
+./cockroach-2/000380.sst : ACCESS
+./cockroach-2/000380.sst : CLOSE_NOWRITE,CLOSE
+./cockroach-2/000380.sst : DELETE
+./cockroach-2/000380.sst : OPEN
+./cockroach-2/000381.log : MODIFY
+./cockroach-2/000381.log : MOVED_TO
+./cockroach-2/000381.log : OPEN
+./cockroach-2/000382.sst : ACCESS
+./cockroach-2/000382.sst : CLOSE_NOWRITE,CLOSE
+./cockroach-2/000382.sst : CLOSE_WRITE,CLOSE
+./cockroach-2/000382.sst : CREATE
+./cockroach-2/000382.sst : DELETE
+./cockroach-2/000382.sst : MODIFY
+./cockroach-2/000382.sst : OPEN
+./cockroach-2/000383.sst : ACCESS
+./cockroach-2/000383.sst : CLOSE_WRITE,CLOSE
+./cockroach-2/000383.sst : CREATE
+./cockroach-2/000383.sst : MODIFY
+./cockroach-2/000383.sst : OPEN
+./cockroach-2/MANIFEST-000001 : MODIFY
 ./cockroach-2/auxiliary : ACCESS,ISDIR
 ./cockroach-2/auxiliary : CLOSE_NOWRITE,CLOSE,ISDIR
 ./cockroach-2/auxiliary : OPEN,ISDIR
@@ -248,6 +270,7 @@
 ./cockroach-2/logs/ : ACCESS,ISDIR
 ./cockroach-2/logs/ : CLOSE_NOWRITE,CLOSE,ISDIR
 ./cockroach-2/logs/ : OPEN,ISDIR
+./cockroach-2/logs/cockroach-pebble.cockroach-2.root.2021-08-28T06_19_08Z.000001.log : MODIFY
 ./cockroach-2/logs/cockroach.cockroach-2.root.2021-08-28T06_19_08Z.000001.log : MODIFY
 ./cockroach-2/logs/goroutine_dump : ACCESS,ISDIR
 ./cockroach-2/logs/goroutine_dump : CLOSE_NOWRITE,CLOSE,ISDIR
@@ -261,3 +284,38 @@
 ./cockroach-2/logs/heap_profiler/ : ACCESS,ISDIR
 ./cockroach-2/logs/heap_profiler/ : CLOSE_NOWRITE,CLOSE,ISDIR
 ./cockroach-2/logs/heap_profiler/ : OPEN,ISDIR
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_30_08.976.244809728.txt : DELETE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_30_18.976.246157312.txt : DELETE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_39_38.983.246288384.txt : DELETE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_39_48.984.247365632.txt : DELETE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_39_58.984.248446976.txt : DELETE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_08.984.249528320.txt : CLOSE_WRITE,CLOSE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_08.984.249528320.txt : CREATE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_08.984.249528320.txt : DELETE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_08.984.249528320.txt : MODIFY
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_08.984.249528320.txt : OPEN
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_18.985.250609664.txt : CLOSE_WRITE,CLOSE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_18.985.250609664.txt : CREATE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_18.985.250609664.txt : DELETE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_18.985.250609664.txt : MODIFY
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_18.985.250609664.txt : OPEN
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_28.985.251420672.txt : CLOSE_WRITE,CLOSE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_28.985.251420672.txt : CREATE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_28.985.251420672.txt : MODIFY
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_28.985.251420672.txt : OPEN
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_38.985.252772352.txt : CLOSE_WRITE,CLOSE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_38.985.252772352.txt : CREATE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_38.985.252772352.txt : MODIFY
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_38.985.252772352.txt : OPEN
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_48.985.253313024.txt : CLOSE_WRITE,CLOSE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_48.985.253313024.txt : CREATE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_48.985.253313024.txt : MODIFY
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_48.985.253313024.txt : OPEN
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_58.985.254394368.txt : CLOSE_WRITE,CLOSE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_58.985.254394368.txt : CREATE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_58.985.254394368.txt : MODIFY
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_40_58.985.254394368.txt : OPEN
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_41_08.985.255471616.txt : CLOSE_WRITE,CLOSE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_41_08.985.255471616.txt : CREATE
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_41_08.985.255471616.txt : MODIFY
+./cockroach-2/logs/heap_profiler/memstats.2021-08-28T06_41_08.985.255471616.txt : OPEN

replica (cockroach-3)

# time inotifywait -mr --format '%w%f : %e' ./cockroach-3/ > INSERT_file_access_cockroach-3
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
^C

real    3m58.410s
user    0m0.068s
sys     0m0.148s

INSERT 時に cockroach-3 がアクセスしているファイル

# sort ./INSERT_file_access_cockroach-3 | uniq -c
     68 ./cockroach-3/ : ACCESS,ISDIR
     52 ./cockroach-3/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/ : OPEN,ISDIR
      1 ./cockroach-3/000368.log : MOVED_FROM
      1 ./cockroach-3/000379.log : CLOSE_WRITE,CLOSE
    567 ./cockroach-3/000379.log : MODIFY
    116 ./cockroach-3/000381.sst : ACCESS
      3 ./cockroach-3/000381.sst : CLOSE_NOWRITE,CLOSE
      1 ./cockroach-3/000381.sst : DELETE
      2 ./cockroach-3/000381.sst : OPEN
   1506 ./cockroach-3/000382.log : MODIFY
      1 ./cockroach-3/000382.log : MOVED_TO
      1 ./cockroach-3/000382.log : OPEN
     80 ./cockroach-3/000383.sst : ACCESS
      2 ./cockroach-3/000383.sst : CLOSE_NOWRITE,CLOSE
      1 ./cockroach-3/000383.sst : CLOSE_WRITE,CLOSE
      1 ./cockroach-3/000383.sst : CREATE
      1 ./cockroach-3/000383.sst : DELETE
     66 ./cockroach-3/000383.sst : MODIFY
      3 ./cockroach-3/000383.sst : OPEN
     15 ./cockroach-3/000384.sst : ACCESS
      1 ./cockroach-3/000384.sst : CLOSE_WRITE,CLOSE
      1 ./cockroach-3/000384.sst : CREATE
    126 ./cockroach-3/000384.sst : MODIFY
      2 ./cockroach-3/000384.sst : OPEN
      2 ./cockroach-3/MANIFEST-000001 : MODIFY
    104 ./cockroach-3/auxiliary : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/1 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/1 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/1 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/1/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/1/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/1/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/10 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/10 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/10 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/10/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/10/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/10/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/11 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/11 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/11 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/11/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/11/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/11/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/12 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/12 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/12 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/12/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/12/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/12/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/13 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/13 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/13 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/13/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/13/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/13/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/14 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/14 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/14 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/14/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/14/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/14/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/15 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/15 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/15 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/15/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/15/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/15/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/16 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/16 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/16 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/16/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/16/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/16/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/17 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/17 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/17 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/17/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/17/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/17/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/18 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/18 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/18 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/18/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/18/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/18/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/19 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/19 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/19 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/19/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/19/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/19/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/2 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/2 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/2 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/2/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/2/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/2/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/20 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/20 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/20 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/20/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/20/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/20/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/21 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/21 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/21 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/21/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/21/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/21/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/22 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/22 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/22 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/22/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/22/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/22/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/23 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/23 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/23 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/23/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/23/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/23/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/24 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/24 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/24 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/24/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/24/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/24/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/25 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/25 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/25 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/25/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/25/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/25/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/26 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/26 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/26 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/26/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/26/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/26/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/27 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/27 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/27 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/27/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/27/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/27/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/28 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/28 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/28 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/28/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/28/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/28/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/29 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/29 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/29 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/29/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/29/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/29/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/3 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/3 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/3 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/3/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/3/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/3/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/30 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/30 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/30 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/30/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/30/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/30/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/31 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/31 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/31 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/31/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/31/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/31/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/32 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/32 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/32 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/32/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/32/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/32/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/33 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/33 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/33 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/33/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/33/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/33/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/34 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/34 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/34 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/34/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/34/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/34/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/35 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/35 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/35 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/35/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/35/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/35/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/36 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/36 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/36 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/36/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/36/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/36/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/4 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/4 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/4 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/4/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/4/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/4/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/5 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/5 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/5 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/5/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/5/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/5/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/6 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/6 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/6 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/6/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/6/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/6/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/7 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/7 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/7 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/7/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/7/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/7/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/8 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/8 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/8 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/8/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/8/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/8/ : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/9 : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/9 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/9 : OPEN,ISDIR
    104 ./cockroach-3/auxiliary/sstsnapshot/9/ : ACCESS,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/9/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/auxiliary/sstsnapshot/9/ : OPEN,ISDIR
    104 ./cockroach-3/cockroach-temp395666710 : ACCESS,ISDIR
     52 ./cockroach-3/cockroach-temp395666710 : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/cockroach-temp395666710 : OPEN,ISDIR
    104 ./cockroach-3/cockroach-temp395666710/ : ACCESS,ISDIR
     52 ./cockroach-3/cockroach-temp395666710/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/cockroach-temp395666710/ : OPEN,ISDIR
    104 ./cockroach-3/cockroach-temp395666710/auxiliary : ACCESS,ISDIR
     52 ./cockroach-3/cockroach-temp395666710/auxiliary : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/cockroach-temp395666710/auxiliary : OPEN,ISDIR
    104 ./cockroach-3/cockroach-temp395666710/auxiliary/ : ACCESS,ISDIR
     52 ./cockroach-3/cockroach-temp395666710/auxiliary/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/cockroach-temp395666710/auxiliary/ : OPEN,ISDIR
    104 ./cockroach-3/logs : ACCESS,ISDIR
     52 ./cockroach-3/logs : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/logs : OPEN,ISDIR
    104 ./cockroach-3/logs/ : ACCESS,ISDIR
     52 ./cockroach-3/logs/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/logs/ : OPEN,ISDIR
      2 ./cockroach-3/logs/cockroach-pebble.cockroach-3.root.2021-08-28T06_19_12Z.000001.log : MODIFY
     24 ./cockroach-3/logs/cockroach.cockroach-3.root.2021-08-28T06_19_12Z.000001.log : MODIFY
    104 ./cockroach-3/logs/goroutine_dump : ACCESS,ISDIR
     52 ./cockroach-3/logs/goroutine_dump : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/logs/goroutine_dump : OPEN,ISDIR
    104 ./cockroach-3/logs/goroutine_dump/ : ACCESS,ISDIR
     52 ./cockroach-3/logs/goroutine_dump/ : CLOSE_NOWRITE,CLOSE,ISDIR
     52 ./cockroach-3/logs/goroutine_dump/ : OPEN,ISDIR
    118 ./cockroach-3/logs/heap_profiler : ACCESS,ISDIR
     59 ./cockroach-3/logs/heap_profiler : CLOSE_NOWRITE,CLOSE,ISDIR
     59 ./cockroach-3/logs/heap_profiler : OPEN,ISDIR
    118 ./cockroach-3/logs/heap_profiler/ : ACCESS,ISDIR
     59 ./cockroach-3/logs/heap_profiler/ : CLOSE_NOWRITE,CLOSE,ISDIR
     59 ./cockroach-3/logs/heap_profiler/ : OPEN,ISDIR
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_30_12.564.245551104.txt : DELETE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_39_32.557.246333440.txt : DELETE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_39_42.557.247955456.txt : DELETE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_39_52.557.248496128.txt : DELETE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_02.558.249307136.txt : DELETE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_12.558.250114048.txt : CLOSE_WRITE,CLOSE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_12.558.250114048.txt : CREATE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_12.558.250114048.txt : DELETE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_12.558.250114048.txt : MODIFY
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_12.558.250114048.txt : OPEN
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_22.558.251195392.txt : CLOSE_WRITE,CLOSE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_22.558.251195392.txt : CREATE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_22.558.251195392.txt : MODIFY
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_22.558.251195392.txt : OPEN
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_32.559.252276736.txt : CLOSE_WRITE,CLOSE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_32.559.252276736.txt : CREATE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_32.559.252276736.txt : MODIFY
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_32.559.252276736.txt : OPEN
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_42.559.253898752.txt : CLOSE_WRITE,CLOSE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_42.559.253898752.txt : CREATE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_42.559.253898752.txt : MODIFY
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_42.559.253898752.txt : OPEN
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_52.558.254980096.txt : CLOSE_WRITE,CLOSE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_52.558.254980096.txt : CREATE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_52.558.254980096.txt : MODIFY
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_52.558.254980096.txt : OPEN
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_41_02.559.255791104.txt : CLOSE_WRITE,CLOSE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_41_02.559.255791104.txt : CREATE
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_41_02.559.255791104.txt : MODIFY
      1 ./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_41_02.559.255791104.txt : OPEN

cockroach-3 での INSERT 時と平常時の差分

# diff -u <(sort ./file_access_cockroach-3 | uniq) <(sort ./INSERT_file_access_cockroach-3 | uniq)
--- /dev/fd/63    2021-08-28 15:49:43.289889149 +0900
+++ /dev/fd/62    2021-08-28 15:49:43.289889149 +0900
@@ -1,7 +1,29 @@
 ./cockroach-3/ : ACCESS,ISDIR
 ./cockroach-3/ : CLOSE_NOWRITE,CLOSE,ISDIR
 ./cockroach-3/ : OPEN,ISDIR
+./cockroach-3/000368.log : MOVED_FROM
+./cockroach-3/000379.log : CLOSE_WRITE,CLOSE
 ./cockroach-3/000379.log : MODIFY
+./cockroach-3/000381.sst : ACCESS
+./cockroach-3/000381.sst : CLOSE_NOWRITE,CLOSE
+./cockroach-3/000381.sst : DELETE
+./cockroach-3/000381.sst : OPEN
+./cockroach-3/000382.log : MODIFY
+./cockroach-3/000382.log : MOVED_TO
+./cockroach-3/000382.log : OPEN
+./cockroach-3/000383.sst : ACCESS
+./cockroach-3/000383.sst : CLOSE_NOWRITE,CLOSE
+./cockroach-3/000383.sst : CLOSE_WRITE,CLOSE
+./cockroach-3/000383.sst : CREATE
+./cockroach-3/000383.sst : DELETE
+./cockroach-3/000383.sst : MODIFY
+./cockroach-3/000383.sst : OPEN
+./cockroach-3/000384.sst : ACCESS
+./cockroach-3/000384.sst : CLOSE_WRITE,CLOSE
+./cockroach-3/000384.sst : CREATE
+./cockroach-3/000384.sst : MODIFY
+./cockroach-3/000384.sst : OPEN
+./cockroach-3/MANIFEST-000001 : MODIFY
 ./cockroach-3/auxiliary : ACCESS,ISDIR
 ./cockroach-3/auxiliary : CLOSE_NOWRITE,CLOSE,ISDIR
 ./cockroach-3/auxiliary : OPEN,ISDIR
@@ -248,6 +270,7 @@
 ./cockroach-3/logs/ : ACCESS,ISDIR
 ./cockroach-3/logs/ : CLOSE_NOWRITE,CLOSE,ISDIR
 ./cockroach-3/logs/ : OPEN,ISDIR
+./cockroach-3/logs/cockroach-pebble.cockroach-3.root.2021-08-28T06_19_12Z.000001.log : MODIFY
 ./cockroach-3/logs/cockroach.cockroach-3.root.2021-08-28T06_19_12Z.000001.log : MODIFY
 ./cockroach-3/logs/goroutine_dump : ACCESS,ISDIR
 ./cockroach-3/logs/goroutine_dump : CLOSE_NOWRITE,CLOSE,ISDIR
@@ -261,8 +284,33 @@
 ./cockroach-3/logs/heap_profiler/ : ACCESS,ISDIR
 ./cockroach-3/logs/heap_profiler/ : CLOSE_NOWRITE,CLOSE,ISDIR
 ./cockroach-3/logs/heap_profiler/ : OPEN,ISDIR
-./cockroach-3/logs/heap_profiler/memprof.2021-08-28T06_20_32.541.47427880.pprof : DELETE
-./cockroach-3/logs/heap_profiler/memprof.2021-08-28T06_33_22.550.56282280.pprof : CLOSE_WRITE,CLOSE
-./cockroach-3/logs/heap_profiler/memprof.2021-08-28T06_33_22.550.56282280.pprof : CREATE
-./cockroach-3/logs/heap_profiler/memprof.2021-08-28T06_33_22.550.56282280.pprof : MODIFY
-./cockroach-3/logs/heap_profiler/memprof.2021-08-28T06_33_22.550.56282280.pprof : OPEN
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_30_12.564.245551104.txt : DELETE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_39_32.557.246333440.txt : DELETE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_39_42.557.247955456.txt : DELETE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_39_52.557.248496128.txt : DELETE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_02.558.249307136.txt : DELETE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_12.558.250114048.txt : CLOSE_WRITE,CLOSE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_12.558.250114048.txt : CREATE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_12.558.250114048.txt : DELETE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_12.558.250114048.txt : MODIFY
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_12.558.250114048.txt : OPEN
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_22.558.251195392.txt : CLOSE_WRITE,CLOSE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_22.558.251195392.txt : CREATE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_22.558.251195392.txt : MODIFY
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_22.558.251195392.txt : OPEN
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_32.559.252276736.txt : CLOSE_WRITE,CLOSE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_32.559.252276736.txt : CREATE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_32.559.252276736.txt : MODIFY
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_32.559.252276736.txt : OPEN
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_42.559.253898752.txt : CLOSE_WRITE,CLOSE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_42.559.253898752.txt : CREATE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_42.559.253898752.txt : MODIFY
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_42.559.253898752.txt : OPEN
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_52.558.254980096.txt : CLOSE_WRITE,CLOSE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_52.558.254980096.txt : CREATE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_52.558.254980096.txt : MODIFY
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_40_52.558.254980096.txt : OPEN
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_41_02.559.255791104.txt : CLOSE_WRITE,CLOSE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_41_02.559.255791104.txt : CREATE
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_41_02.559.255791104.txt : MODIFY
+./cockroach-3/logs/heap_profiler/memstats.2021-08-28T06_41_02.559.255791104.txt : OPEN

差分確認

定期的にアクセスされていたファイルと INSERT 時にアクセスされていたファイルを比較し、INSERT 時にのみアクセスされていたファイルを確認してみようと思います。

XXXXXX.log ファイル

拡張子が ".log" になっているファイルは、エラーメッセージ等を保存する「ログファイル」ではなく、 WAL (Write Ahead Log) です。この WAL は、CockroachDB の Storage Engine として利用されている Pebble が出力しているファイルです。

こちらについては起動直後 (INSERT していないタイミング) でも定期的にアクセスが発生していることが確認できていますが、CockroachDB では Raft 関連のデータ (Raft Log) や Web UI (DB Console) で参照できるクラスタのメトリクスの情報も他の TABLE と同じように KV 形式のデータとして Pebble に保存されます。

そのため、ユーザが INSERT/UPDATE/DELETE 等を実行していなくても定期的に WAL への書き込みが発生している状況であり、INSERT の瞬間にアクセスされたことを明確に観測できている訳では無いのですが、CockroachDB は ACID (今回の場合は特に Durability の部分) 準拠の DB であるため、仕組み上 INSERT 時に WAL (XXXXXX.log) ファイルへのアクセス (Write) が発生することは間違いないと思われます。

※strace でも確認してみましたが、どこかのタイミング (他のレコードを INSERT したタイミング？) で openat(2) して掴みっぱなしにしているのか、INSERT 時に WAL を openat(2) している形跡が確認できませんでした...

XXXXXX.sst ファイル

拡張子が ".sst" になっているファイルは、SSTable (Sorted String Table) と呼ばれるファイルです。こちらも、CockroachDB の Storage Engine である Pebble が出力しているものであり、TABLE の実データが含まれている LSM Tree 実装のファイルです。

しかし、何度か検証してみたところ「INSERT の瞬間に SSTable (XXXXXX.sst) にアクセスしている訳ではない」という感じのようです。

Pebble は LSM Tree 実装の KVS であるため、INSERT されたデータは前述した WAL (XXXXXX.log) に書き込まれた上で、メモリ上の memtable に格納されます。そして、memtable が一定のサイズ以上になった場合や、定期的に実行される memtable のフラッシュ処理時に、memtable の内容が SSTable (XXXXXX.sst) にフラッシュされます。

そのため、厳密に言うと SSTable (XXXXXX.sst) は INSERT 時にアクセスされるファイルではありませんが、INSERT した際に memtable のサイズが上限値に達した場合は、SSTable (XXXXXX.sst) へのアクセス (Write) が発生する動作になると思われます。

MANIFEST-000001 ファイル

こいつについては正体がわかりませんでした...

Pebble のログに以下のような出力が有るので、こいつも Pebble 関連のファイルだと思うのですが、INSERT の処理に関連しているのか否かは分かりませんでした...

# grep MANIFEST cockroach-pebble.log
I210828 06:19:03.852876 82 3@vendor/github.com/cockroachdb/pebble/version_set.go:156 ⋮ [n?,pebble,s?] 1  [JOB 1] MANIFEST created 000001
I210828 06:19:03.884580 82 3@vendor/github.com/cockroachdb/pebble/version_set.go:156 ⋮ [n?,pebble,s?] 4  [JOB 1] MANIFEST created 000001

また、このファイルはバイナリデータなのですが、strings や hexdump で中身を覗いてみたところ cockroach_comparator という単語が見えたので、そのあたりをヒントにして別の機会に調べてみようと思います。

# file cockroach-1/MANIFEST-000001 
cockroach-1/MANIFEST-000001: data

# strings MANIFEST-000001 | head -n 1
cockroach_comparator

# hexdump -C MANIFEST-000001 | head -n 2
00000000  bf 13 d7 16 1a 00 01 01  14 63 6f 63 6b 72 6f 61  |.........cockroa|
00000010  63 68 5f 63 6f 6d 70 61  72 61 74 6f 72 03 02 04  |ch_comparator...|

logs/ ディレクトリ配下のファイル

以下の形式で出力されているファイルは TEXT 形式のファイルであり、いわゆる「ログファイル」です。そのため、これらは定常的にアクセスされており、INSERT に直接関連したアクセスではないと思われます。

./cockroach-1/logs/cockroach-pebble.cockroach-1.root.2021-08-28T06_19_03Z.000001.log
./cockroach-1/logs/cockroach.cockroach-1.root.2021-08-28T06_19_03Z.000001.log

ただし、INSERT に伴い memtable のフラッシュや WAL の recycle が発生すると、それらの処理を実行した旨のメッセージがログファイルに出力されるため、INSERT 時にファイルアクセスが発生する可能性もあります。

• memtable のフラッシュ (SSTable 作成)

I210828 07:56:15.365857 381443 3@vendor/github.com/cockroachdb/pebble/compaction.go:1985 ⋮ [n1,pebble,s1] 92  [JOB 44] flushing: sstable created 000036
I210828 07:56:15.670978 381475 3@vendor/github.com/cockroachdb/pebble/compaction.go:1985 ⋮ [n1,pebble,s1] 95  [JOB 46] compacting: sstable created 000037

• WAL の作成 (recycled)

I210828 07:56:15.365219 253 3@vendor/github.com/cockroachdb/pebble/db.go:1449 ⋮ [n1,pebble,s1] 90  [JOB 43] WAL created 000035 (recycled 000017)

また、logs/heap_profiler/ 配下のファイルについて詳細は調べられていないのですが、ファイル名にタイムスタンプの情報を含んでいることから、定期的にアクセス (更新) されているファイルのようであり、INSERT の処理には直接関連していなさそうです。

ちなみに、.txt になっているファイルの中身を見ると以下のようになっており、メモリ関連の統計情報を扱っているファイルであるようです。

# head -n 30 memstats.2021-08-28T07_34_25.259.282726400.txt
Go memory stats:
{
  "Alloc": 58207552,
  "TotalAlloc": 8454525400,
  "Sys": 147476488,
  "Lookups": 0,
  "Mallocs": 42250539,
  "Frees": 42004715,
  "HeapAlloc": 58207552,
  "HeapSys": 128581632,
  "HeapIdle": 65052672,
  "HeapInuse": 63528960,
  "HeapReleased": 58875904,
  "HeapObjects": 245824,
  "StackInuse": 5636096,
  "StackSys": 5636096,
  "MSpanInuse": 653616,
  "MSpanSys": 1015808,
  "MCacheInuse": 13888,
  "MCacheSys": 16384,
  "BuckHashSys": 2235814,
  "GCSys": 8726760,
  "OtherSys": 1263994,
  "NextGC": 65079600,
  "LastGC": 1630136045376634882,
  "PauseTotalNs": 39247543,
  "PauseNs": [
    59176,
    88149,
    182033,

おまけ

CockroachDB 起動 (DB 初期化) 直後に作成されているファイルは以下のようになっていました。SSTable (XXXXXX.sst ファイル) は初期状態では存在していないようです。

CockroachDB 起動 (DB 初期化) 直後に作成されているファイル

$ sudo ls -laFR cockroach-1
cockroach-1:
total 360516
drwxr-xr-x 5 root root    4096 Aug 29 10:51 ./
drwxr-xr-x 5 root root    4096 Aug 29 10:50 ../
-rw-r----- 1 root root  147639 Aug 29 10:50 000002.log
-rw-r----- 1 root root  456550 Aug 29 10:50 000004.log
-rw-r----- 1 root root  800183 Aug 29 10:50 000005.log
-rw-r----- 1 root root 1761113 Aug 29 10:51 000006.log
-rw-r----- 1 root root  286878 Aug 29 10:51 000007.log
-rw-r----- 1 root root      16 Aug 29 10:50 CURRENT
-rw-r----- 1 root root       0 Aug 29 10:50 LOCK
-rw-r----- 1 root root      44 Aug 29 10:50 MANIFEST-000001
-rw-r----- 1 root root    2224 Aug 29 10:50 OPTIONS-000003
drwxr-x--- 2 root root    4096 Aug 29 10:50 auxiliary/
drwxr-xr-x 3 root root    4096 Aug 29 10:50 cockroach-temp096599273/
-rw-r----- 1 root root      17 Aug 29 10:50 cockroach.advertise-addr
-rw-r----- 1 root root      17 Aug 29 10:50 cockroach.advertise-sql-addr
-rw-r----- 1 root root      16 Aug 29 10:50 cockroach.http-addr
-rw-r----- 1 root root      10 Aug 29 10:50 cockroach.listen-addr
-rw-r----- 1 root root      10 Aug 29 10:50 cockroach.sql-addr
drwxr-x--- 4 root root    4096 Aug 29 10:50 logs/
-rw-r----- 1 root root      50 Aug 29 10:50 temp-dirs-record.txt

cockroach-1/auxiliary:
total 8
drwxr-x--- 2 root root 4096 Aug 29 10:50 ./
drwxr-xr-x 5 root root 4096 Aug 29 10:51 ../

cockroach-1/cockroach-temp096599273:
total 24
drwxr-xr-x 3 root root 4096 Aug 29 10:50 ./
drwxr-xr-x 5 root root 4096 Aug 29 10:51 ../
-rw-r----- 1 root root    0 Aug 29 10:50 000002.log
-rw-r----- 1 root root   16 Aug 29 10:50 CURRENT
-rw-r----- 1 root root    0 Aug 29 10:50 LOCK
-rw-r----- 1 root root   50 Aug 29 10:50 MANIFEST-000001
-rw-r----- 1 root root 2159 Aug 29 10:50 OPTIONS-000003
-rw-r----- 1 root root    0 Aug 29 10:50 TEMP_DIR.LOCK
drwxr-x--- 2 root root 4096 Aug 29 10:50 auxiliary/

cockroach-1/cockroach-temp096599273/auxiliary:
total 8
drwxr-x--- 2 root root 4096 Aug 29 10:50 ./
drwxr-xr-x 3 root root 4096 Aug 29 10:50 ../

cockroach-1/logs:
total 136
drwxr-x--- 4 root root   4096 Aug 29 10:50 ./
drwxr-xr-x 5 root root   4096 Aug 29 10:51 ../
-rw-r----- 1 root root   2242 Aug 29 10:51 cockroach-pebble.cockroach-1.root.2021-08-29T01_50_45Z.000001.log
lrwxrwxrwx 1 root root     65 Aug 29 10:50 cockroach-pebble.log -> cockroach-pebble.cockroach-1.root.2021-08-29T01_50_45Z.000001.log
-rw-r----- 1 root root    973 Aug 29 10:50 cockroach-stderr.cockroach-1.root.2021-08-29T01_50_45Z.000001.log
lrwxrwxrwx 1 root root     65 Aug 29 10:50 cockroach-stderr.log -> cockroach-stderr.cockroach-1.root.2021-08-29T01_50_45Z.000001.log
-rw-r----- 1 root root 105762 Aug 29 10:51 cockroach.cockroach-1.root.2021-08-29T01_50_45Z.000001.log
lrwxrwxrwx 1 root root     58 Aug 29 10:50 cockroach.log -> cockroach.cockroach-1.root.2021-08-29T01_50_45Z.000001.log
drwxr-x--- 2 root root   4096 Aug 29 10:50 goroutine_dump/
drwxr-x--- 2 root root   4096 Aug 29 10:51 heap_profiler/

cockroach-1/logs/goroutine_dump:
total 8
drwxr-x--- 2 root root 4096 Aug 29 10:50 ./
drwxr-x--- 4 root root 4096 Aug 29 10:50 ../

cockroach-1/logs/heap_profiler:
total 72
drwxr-x--- 2 root root  4096 Aug 29 10:51 ./
drwxr-x--- 4 root root  4096 Aug 29 10:50 ../
-rw-r----- 1 root root 37375 Aug 29 10:50 memprof.2021-08-29T01_50_57.174.37237424.pprof
-rw-r----- 1 root root  9186 Aug 29 10:50 memstats.2021-08-29T01_50_57.184.150753280.txt
-rw-r----- 1 root root  9312 Aug 29 10:51 memstats.2021-08-29T01_51_07.174.175755264.txt

まとめ

CockroachDB に 1レコード INSERT すると...

• データの永続化のために WAL (XXXXXX.log ファイル) にアクセスする。
• (INSERT で memtable のサイズが上限を超えると) memtable を永続化するために SSTable (XXXXXX.sst ファイル) にアクセスすると思われる。
• (詳細は不明だが) MANIFEST-000001 というファイルにアクセスしてるっぽい...？

という感じになってました。

Pebble なんもわからん (´・ω・`)