[tidb-x] tiflash wn crash repeatedly after recovery network partition from dfs

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run ch
2、simulated dfs network partition last for 10mins
chaos time: 2025/12/21 11:35:58.043 +08:00 ~ 2025/12/21 11:45:58.044 +08:00
3、recovery fault

2. What did you expect to see? (Required)

after recovery fault，all tiflash are normal

3. What did you see instead (Required)

after recovery fault，tiflash wn crash repeatedly and can not recovery
{"namespace":"ha-test-tiflash-tps-8040398-1-848","log":"[2025/12/21 11:47:19.611 +08:00] [ERROR] [BaseDaemon.cpp:560] [\"\\n 0x556d665d42fe\\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+133485310]\\n \\tlibs/libdaemon/src/BaseDaemon.cpp:211\\n 0x7fed4af75bf0\\t<unknown symbol> [libc.so.6+257008]\\n 0x556d671ade7a\\tvoid DB::S3::TiFlashS3Client::setBucketAndKeyWithRoot<Aws::S3::Model::PutObjectRequest>(Aws::S3::Model::PutObjectRequest&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) const [tiflash+145911418]\\n \\tdbms/src/Storages/S3/S3Common.h:87\\n 0x556d671a460e\\tDB::S3::uploadFile(DB::S3::TiFlashS3Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::EncryptionPath const&, std::__1::shared_ptr<DB::FileProvider> const&, int) [tiflash+145872398]\\n \\tdbms/src/Storages/S3/S3Common.cpp:638\\n 0x556d66fcb54d\\tstd::__1::__packaged_task_func<DB::DM::Remote::DataStoreS3::putDMFileLocalFiles(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, DB::DM::Remote::DMFileOID const&)::$_2, std::__1::allocator<DB::DM::Remote::DataStoreS3::putDMFileLocalFiles(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, DB::DM::Remote::DMFileOID const&)::$_2>, void ()>::operator()() (.0657bb016a1310a986a4a2750184c5f4) [tiflash+143934797]\\n \\tdbms/src/Storages/DeltaMerge/Remote/DataStore/DataStoreS3.cpp:85\\n 0x556d60b3b895\\tstd::__1::packaged_task<void ()>::operator()() [tiflash+38488213]\\n \\t/usr/local/bin/../include/c++/v1/future:1891\\n 0x556d60b39a79\\tDB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::worker(std::__1::__list_iterator<DB::ThreadFromGlobalPoolImpl<false>, void*>) [tiflash+38480505]\\n \\t/usr/local/bin/../include/c++/v1/__functional/function.h:517\\n 0x556d60b3c3a3\\tstd::__1::__function::__func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()>, void ()>::operator()() [tiflash+38491043]\\n \\tdbms/src/Common/UniThreadPool.cpp:169\\n 0x556d60b3af68\\tvoid* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) [tiflash+38485864]\\n \\t/usr/local/bin/../include/c++/v1/__functional/function.h:517\\n 0x7fed4afc119a\\tstart_thread [libc.so.6+565658]\"] [source=BaseDaemon] [thread_id=1101]\n","pod":"tc-tiflash-0","container":"serverlog","time":"2025-12-21T03:47:19.972866215Z","stream":"stdout"}

4. What is your TiFlash version? (Required)

/tiflash/tiflash version
TiFlash
Release Version: v9.0.0-beta.2.pre-99-g0ddabdef5
Edition: Enterprise
Git Commit Hash: 0ddabde
Git Branch: HEAD
UTC Build Time: 2025-12-12 07:57:30
Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto next-gen hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake
Profile: RELWITHDEBINFO
Compiler: clang++ 17.0.6

Raft Proxy
Git Commit Hash: 2505f2f8d3061d8e61aa6f4ff4b91ade95a50785
Git Commit Branch: HEAD
UTC Build Time: ""
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Storage Engine: tiflash
Prometheus Prefix: tiflash_proxy_
Profile: release

[JaySon-Huang]

The direct reason tiflash crash repeatedly is the minio in testing env is full.

testbed: "ha-test-tiflash-tps-8040398-1-848" AND pod: "tc-tiflash-0" | FATAL

{"namespace":"ha-test-tiflash-tps-8040398-1-848","log":"
[2025/12/21 11:55:50.959 +08:00] [FATAL] [Exception.cpp:98] [\"Code: 11000, e.displayText() = DB::Exception: S3 PutObject failed, local_fname=/data0/tiflash/data/ks_1_t_2737/stable/dmf_1101126/1.dat bucket=tiflash root=tiflash/ key=s1562/data/ks_1_t_2737/dmf_1101126/1.dat s3error=UNKNOWN s3exception_name=XMinioStorageFull s3msg=Unable to parse ExceptionName: XMinioStorageFull Message: Storage backend has reached its minimum free drive threshold. Please delete a few objects to proceed. request_id=18831ECE731FC33F: keyspace=1 physical_table_id=2737: (while preHandleSnapshot region_id=5109, index=259193, term=11), e.what() = DB::Exception, Stack trace:
  0x561fc74c314c    DB::Exception DB::S3::fromS3Error<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(Aws::S3::S3Error const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) [tiflash+145916236]
                    dbms/src/Common/StackTrace.cpp:25
  0x561fc74b976b    DB::S3::uploadFile(DB::S3::TiFlashS3Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::EncryptionPath const&, std::__1::shared_ptr<DB::FileProvider> const&, int) [tiflash+145876843]
                    dbms/src/Storages/S3/S3Common.cpp:666
  0x561fc72df54d    std::__1::__packaged_task_func<DB::DM::Remote::DataStoreS3::putDMFileLocalFiles(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, DB::DM::Remote::DMFileOID const&)::$_2, std::__1::allocator<DB::DM::Remote::DataStoreS3::putDMFileLocalFiles(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, DB::DM::Remote::DMFileOID const&)::$_2>, void ()>::operator()() (.0657bb016a1310a986a4a2750184c5f4) [tiflash+143934797]
                    dbms/src/Storages/DeltaMerge/Remote/DataStore/DataStoreS3.cpp:85
  0x561fc0e4f895    std::__1::packaged_task<void ()>::operator()() [tiflash+38488213]
                    /usr/local/bin/../include/c++/v1/future:1891
  0x561fc0e4da79    DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::worker(std::__1::__list_iterator<DB::ThreadFromGlobalPoolImpl<false>, void*>) [tiflash+38480505]
                    /usr/local/bin/../include/c++/v1/__functional/function.h:517
  0x561fc0e503a3    std::__1::__function::__func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()>, void ()>::operator()() [tiflash+38491043]
                    dbms/src/Common/UniThreadPool.cpp:169
  0x561fc0e4ef68    void* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) [tiflash+38485864]
                    /usr/local/bin/../include/c++/v1/__functional/function.h:517
  0x7fbdd2b7a19a    start_thread [libc.so.6+565658]
  0x7fbdd2bff240    __clone3 [libc.so.6+1110592]\"] [source=\"RawCppPtr DB::PreHandleSnapshot(EngineStoreServerWrap *, BaseBuffView, uint64_t, SSTViewVec, uint64_t, uint64_t)\"] [thread_id=856]\n","pod":"tc-tiflash-0","container":"serverlog","time":"2025-12-21T03:55:51.213412162Z","stream":"stdout"}

And the reason why minio in testing env is full, seems related to that S3GC thread hangs since 03:05:58.

logging:

Explore-logs-2025-12-23 17_11_13.txt.zip

# testbed: "ha-test-tiflash-tps-8040398-1-848" AND pod: "tc-tiflash-1" NOT pod: "secondary-tc-tiflash-1" | "thread_id=43"

2025-12-21 03:05:50.109	{"log":"[2025/12/21 03:05:50.081 +08:00] [DEBUG] [S3GCManager.cpp:197] [\"run gc, gc_store_id=1561 timepoint=2025-12-20T19:05:50Z\"] [thread_id=43]\n","stream":"stdout","container":"serverlog","time":"2025-12-20T19:05:50.109308788Z","pod":"tc-tiflash-1","namespace":"ha-test-tiflash-tps-8040398-1-848"}
...
2025-12-21 03:05:50.109	{"log":"[2025/12/21 03:05:50.086 +08:00] [INFO] [S3GCManager.cpp:213] [\"latest manifest, gc_store_id=1561 upload_seq=308 key=s1561/manifest/mf_308\"] [thread_id=43]\n","stream":"stdout","container":"serverlog","time":"2025-12-20T19:05:50.10931537Z","pod":"tc-tiflash-1","namespace":"ha-test-tiflash-tps-8040398-1-848"}
...
2025-12-21 03:05:58.110	{"log":"[2025/12/21 03:05:57.726 +08:00] [INFO] [S3GCManager.cpp:741] [\"remove outdated manifest, key=s1561/manifest/mf_298 mtime=2025-12-20T18:59:29Z\"] [thread_id=43]\n","pod":"tc-tiflash-1","container":"serverlog","time":"2025-12-20T19:05:58.110861686Z","namespace":"ha-test-tiflash-tps-8040398-1-848","stream":"stdout"}
2025-12-21 03:05:58.110	{"log":"[2025/12/21 03:05:57.792 +08:00] [INFO] [S3GCManager.cpp:509] [\"delmark exist, datafile=s1561/data/dat_206_0 mark_mtime=2025-12-20T18:05:31Z now=2025-12-20T19:05:50Z diff_sec=3619.081 expired=true\"] [source=\"remove_key=s1561/data/dat_206_0\"] [thread_id=43]\n","pod":"tc-tiflash-1","container":"serverlog","time":"2025-12-20T19:05:58.110984937Z","namespace":"ha-test-tiflash-tps-8040398-1-848","stream":"stdout"}
2025-12-21 03:05:58.110	{"log":"[2025/12/21 03:05:57.794 +08:00] [INFO] [S3GCManager.cpp:606] [\"datafile deleted\"] [source=\"remove_key=s1561/data/dat_206_0\"] [thread_id=43]\n","pod":"tc-tiflash-1","container":"serverlog","time":"2025-12-20T19:05:58.110996077Z","namespace":"ha-test-tiflash-tps-8040398-1-848","stream":"stdout"}
...
2025-12-21 03:05:59.111	{"log":"[2025/12/21 03:05:58.630 +08:00] [INFO] [S3GCManager.cpp:509] [\"delmark exist, datafile=s1561/data/ks_1_t_2729/dmf_523 mark_mtime=2025-12-20T18:08:20Z now=2025-12-20T19:05:50Z diff_sec=3450.081 expired=false\"] [source=\"remove_key=s1561/data/ks_1_t_2729/dmf_523\"] [thread_id=43]\n","pod":"tc-tiflash-1","container":"serverlog","time":"2025-12-20T19:05:59.111411046Z","stream":"stdout","namespace":"ha-test-tiflash-tps-8040398-1-848"}
2025-12-21 03:05:59.111	{"log":"[2025/12/21 03:05:58.630 +08:00] [INFO] [S3GCManager.cpp:509] [\"delmark exist, datafile=s1561/data/ks_1_t_2729/dmf_545 mark_mtime=2025-12-20T18:08:20Z now=2025-12-20T19:05:50Z diff_sec=3450.081 expired=false\"] [source=\"remove_key=s1561/data/ks_1_t_2729/dmf_545\"] [thread_id=43]\n","pod":"tc-tiflash-1","container":"serverlog","time":"2025-12-20T19:05:59.111414157Z","stream":"stdout","namespace":"ha-test-tiflash-tps-8040398-1-848"}
2025-12-21 03:05:59.111	{"log":"[2025/12/21 03:05:58.631 +08:00] [DEBUG] [S3Common.cpp:881] [\"listPrefix prefix=s1561/data/, keys=1000, total_keys=4000, next_token=dGlmbGFzaC9zMTU2MS9kYXRhL2tzXzFfdF8yNzI5L2RtZl81OTc2Lzkuc2l6ZS5kYXRbbWluaW9fY2FjaGU6djIsaWQ6MWIxYjQ3MTItZjJlOC00MTRlLWFjMzItYzg5MDE3YTQ1OWFiLHA6MCxzOjBd\"] [source=\"bucket=tiflash root=tiflash/\"] [thread_id=43]\n","pod":"tc-tiflash-1","container":"serverlog","time":"2025-12-20T19:05:59.111420297Z","stream":"stdout","namespace":"ha-test-tiflash-tps-8040398-1-848"}
2025-12-21 03:05:59.111	{"log":"[2025/12/21 03:05:58.804 +08:00] [DEBUG] [S3Common.cpp:881] [\"listPrefix prefix=s1561/data/, keys=1000, total_keys=5000, next_token=dGlmbGFzaC9zMTU2MS9kYXRhL2tzXzFfdF8yNzI5L2RtZl82MDIxLzE4LmRhdFttaW5pb19jYWNoZTp2MixpZDoxYjFiNDcxMi1mMmU4LTQxNGUtYWMzMi1jODkwMTdhNDU5YWIscDowLHM6MF0=\"] [source=\"bucket=tiflash root=tiflash/\"] [thread_id=43]\n","pod":"tc-tiflash-1","container":"serverlog","time":"2025-12-20T19:05:59.111498502Z","stream":"stdout","namespace":"ha-test-tiflash-tps-8040398-1-848"}
# then no more logging about "thread_id=43" or "S3GCManager"

metrics