Skip to content

Conversation

@hubgeter
Copy link
Contributor

What problem does this PR solve?

Problem Summary:
This PR fixes the semantics of reading Parquet INT96 values. Previously, INT96 was interpreted as a time zone–aware type, causing the read results to vary with the set timezone configuration.
This PR changes the behavior to always convert INT96 values to UTC, treating them as time zone–independent timestamps.

Release note

Parse Parquet INT96 physical type data into a no-timezone datetime.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 31, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31901 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 666297a2c335cd13b722c2b4d535e8f716ab6bf2, data reload: false

------ Round 1 ----------------------------------
q1	17636	4216	4042	4042
q2	2031	366	237	237
q3	10185	1270	734	734
q4	10204	898	316	316
q5	7553	2159	1915	1915
q6	194	171	137	137
q7	946	807	668	668
q8	9283	1430	1150	1150
q9	5510	4736	4721	4721
q10	6815	1810	1428	1428
q11	488	303	290	290
q12	736	742	602	602
q13	17826	3819	3065	3065
q14	296	287	274	274
q15	594	515	518	515
q16	707	689	644	644
q17	720	749	617	617
q18	6650	6505	6381	6381
q19	1112	961	618	618
q20	402	373	247	247
q21	3138	2562	2356	2356
q22	1102	1055	944	944
Total cold run time: 104128 ms
Total hot run time: 31901 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4175	4062	4042	4042
q2	338	395	333	333
q3	2128	2589	2222	2222
q4	1324	1749	1297	1297
q5	4078	4014	3992	3992
q6	208	170	129	129
q7	1890	1845	2102	1845
q8	2610	2464	2361	2361
q9	7224	7161	7124	7124
q10	2499	2784	2338	2338
q11	529	486	458	458
q12	730	769	622	622
q13	3622	4045	3391	3391
q14	406	301	291	291
q15	542	500	498	498
q16	660	657	649	649
q17	1138	1344	1347	1344
q18	7858	7934	8137	7934
q19	920	911	932	911
q20	2095	2018	1911	1911
q21	4697	4477	4237	4237
q22	1139	1086	988	988
Total cold run time: 50810 ms
Total hot run time: 48917 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174160 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 666297a2c335cd13b722c2b4d535e8f716ab6bf2, data reload: false

query5	4490	593	449	449
query6	332	227	213	213
query7	4224	478	279	279
query8	351	254	242	242
query9	8758	2600	2637	2600
query10	492	379	318	318
query11	15178	15043	15060	15043
query12	181	118	115	115
query13	1272	505	416	416
query14	6216	2957	2733	2733
query14_1	2631	2641	2678	2641
query15	198	195	179	179
query16	984	403	469	403
query17	1123	736	570	570
query18	2530	425	333	333
query19	225	221	185	185
query20	117	118	112	112
query21	224	151	115	115
query22	3951	4002	3824	3824
query23	15981	15694	15438	15438
query23_1	15435	15568	15500	15500
query24	7366	1579	1238	1238
query24_1	1216	1201	1181	1181
query25	528	469	401	401
query26	1228	262	157	157
query27	2779	457	291	291
query28	4549	2201	2194	2194
query29	776	523	429	429
query30	313	236	214	214
query31	803	627	556	556
query32	78	71	69	69
query33	528	326	290	290
query34	891	882	540	540
query35	755	801	697	697
query36	903	887	853	853
query37	127	99	78	78
query38	2725	2679	2618	2618
query39	778	743	730	730
query39_1	733	714	710	710
query40	217	138	121	121
query41	73	68	66	66
query42	104	105	107	105
query43	457	449	421	421
query44	1339	762	765	762
query45	191	186	181	181
query46	872	964	617	617
query47	1447	1530	1378	1378
query48	334	337	257	257
query49	647	429	340	340
query50	663	295	232	232
query51	3754	3819	3804	3804
query52	113	112	100	100
query53	300	333	277	277
query54	305	289	260	260
query55	81	81	76	76
query56	302	318	317	317
query57	1017	1064	978	978
query58	270	268	262	262
query59	2092	2141	2172	2141
query60	336	328	302	302
query61	192	181	182	181
query62	383	375	324	324
query63	304	268	270	268
query64	5128	1360	995	995
query65	3761	3688	3768	3688
query66	1433	438	311	311
query67	14934	15281	15815	15281
query68	8237	1010	725	725
query69	510	339	313	313
query70	1048	903	941	903
query71	355	304	279	279
query72	6130	3413	3508	3413
query73	767	723	316	316
query74	8810	8796	8645	8645
query75	2835	2845	2499	2499
query76	3889	1066	662	662
query77	546	386	277	277
query78	9865	9997	9080	9080
query79	1214	929	618	618
query80	669	585	519	519
query81	494	264	230	230
query82	212	145	107	107
query83	263	248	248	248
query84	258	120	109	109
query85	882	525	462	462
query86	319	315	316	315
query87	2872	2858	2758	2758
query88	3197	2324	2291	2291
query89	404	356	332	332
query90	1960	154	159	154
query91	179	164	155	155
query92	70	65	61	61
query93	1008	919	573	573
query94	568	335	289	289
query95	586	344	307	307
query96	588	448	205	205
query97	2310	2381	2291	2291
query98	220	196	196	196
query99	600	577	513	513
Total cold run time: 253913 ms
Total hot run time: 174160 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 27.17 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 666297a2c335cd13b722c2b4d535e8f716ab6bf2, data reload: false

query1	0.06	0.05	0.05
query2	0.11	0.06	0.05
query3	0.25	0.10	0.09
query4	1.61	0.11	0.11
query5	0.27	0.24	0.26
query6	1.15	0.64	0.66
query7	0.04	0.02	0.03
query8	0.05	0.04	0.04
query9	0.58	0.51	0.49
query10	0.57	0.55	0.56
query11	0.16	0.11	0.11
query12	0.16	0.13	0.13
query13	0.63	0.60	0.59
query14	0.97	0.99	0.98
query15	0.82	0.80	0.78
query16	0.42	0.42	0.37
query17	1.03	1.05	1.08
query18	0.22	0.21	0.21
query19	1.95	1.91	1.89
query20	0.02	0.02	0.01
query21	15.45	0.30	0.14
query22	4.86	0.05	0.05
query23	16.18	0.30	0.10
query24	0.94	0.61	0.40
query25	0.11	0.08	0.08
query26	0.13	0.14	0.12
query27	0.06	0.06	0.04
query28	4.97	1.05	0.88
query29	12.60	3.97	3.18
query30	0.28	0.13	0.12
query31	2.81	0.67	0.38
query32	3.23	0.57	0.48
query33	2.97	3.03	3.08
query34	16.64	5.09	4.48
query35	4.45	4.48	4.45
query36	0.65	0.50	0.49
query37	0.11	0.06	0.06
query38	0.08	0.04	0.04
query39	0.04	0.04	0.03
query40	0.19	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 97.98 s
Total hot run time: 27.17 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 78.05% (32/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.39% (18964/35518)
Line Coverage 39.26% (175969/448170)
Region Coverage 33.82% (136138/402479)
Branch Coverage 34.76% (58796/169164)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 78.05% (32/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.15% (24357/34723)
Line Coverage 56.51% (252588/446978)
Region Coverage 51.06% (207627/406642)
Branch Coverage 52.49% (89083/169726)

@hubgeter
Copy link
Contributor Author

hubgeter commented Jan 4, 2026

run buildall

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

TPC-H: Total hot run time: 32002 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ea91c2dbc71d1824222d0ca9cf3c6844d48a0504, data reload: false

------ Round 1 ----------------------------------
q1	17644	4265	4063	4063
q2	2036	360	246	246
q3	10153	1287	706	706
q4	10207	809	312	312
q5	7492	2091	1863	1863
q6	190	171	138	138
q7	953	776	656	656
q8	9289	1431	1167	1167
q9	5207	4509	4498	4498
q10	6809	1817	1390	1390
q11	518	290	286	286
q12	768	742	582	582
q13	17796	3780	3100	3100
q14	290	287	261	261
q15	596	514	512	512
q16	674	692	609	609
q17	667	789	569	569
q18	6982	6578	6763	6578
q19	1173	998	624	624
q20	422	395	272	272
q21	3223	2590	2520	2520
q22	1135	1089	1050	1050
Total cold run time: 104224 ms
Total hot run time: 32002 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4350	4306	4220	4220
q2	356	425	317	317
q3	2197	2785	2474	2474
q4	1389	1844	1401	1401
q5	4640	4377	4387	4377
q6	213	163	127	127
q7	1976	1878	1789	1789
q8	2705	2424	2361	2361
q9	7056	7279	7304	7279
q10	2533	2776	2290	2290
q11	545	459	434	434
q12	669	744	546	546
q13	3269	3753	3060	3060
q14	278	292	256	256
q15	527	491	486	486
q16	628	631	602	602
q17	1092	1301	1331	1301
q18	7380	7322	7355	7322
q19	817	778	818	778
q20	1877	2078	1834	1834
q21	4500	4415	4077	4077
q22	1105	1048	964	964
Total cold run time: 50102 ms
Total hot run time: 48295 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172457 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ea91c2dbc71d1824222d0ca9cf3c6844d48a0504, data reload: false

query5	4976	585	452	452
query6	320	220	217	217
query7	4210	445	258	258
query8	326	232	245	232
query9	8782	2599	2629	2599
query10	535	373	320	320
query11	15294	15289	14969	14969
query12	175	121	113	113
query13	1269	485	386	386
query14	6469	2962	2663	2663
query14_1	2610	2588	2630	2588
query15	208	192	175	175
query16	994	491	480	480
query17	1078	689	570	570
query18	2617	439	338	338
query19	228	225	196	196
query20	124	117	114	114
query21	213	141	125	125
query22	4176	4004	4100	4004
query23	15939	15574	15443	15443
query23_1	15408	15430	15551	15430
query24	7369	1519	1141	1141
query24_1	1180	1151	1190	1151
query25	558	490	431	431
query26	1234	278	160	160
query27	2747	448	294	294
query28	4582	2118	2124	2118
query29	787	552	447	447
query30	316	239	212	212
query31	804	624	561	561
query32	82	68	69	68
query33	587	326	266	266
query34	900	857	511	511
query35	748	782	691	691
query36	847	885	834	834
query37	121	91	79	79
query38	2702	2717	2649	2649
query39	780	752	733	733
query39_1	714	708	719	708
query40	216	130	116	116
query41	66	60	64	60
query42	105	101	96	96
query43	430	471	405	405
query44	1309	728	725	725
query45	186	183	178	178
query46	846	945	579	579
query47	1451	1470	1364	1364
query48	306	319	232	232
query49	605	419	326	326
query50	626	266	203	203
query51	3792	3756	3807	3756
query52	107	107	96	96
query53	294	327	261	261
query54	278	295	248	248
query55	78	71	69	69
query56	294	291	283	283
query57	1070	985	926	926
query58	264	242	245	242
query59	2068	2101	2030	2030
query60	318	309	291	291
query61	159	166	158	158
query62	391	372	320	320
query63	300	264	267	264
query64	4879	1297	988	988
query65	3760	3704	3710	3704
query66	1391	415	307	307
query67	15359	15363	15134	15134
query68	6223	990	705	705
query69	486	343	307	307
query70	1002	945	868	868
query71	360	298	278	278
query72	6085	3442	3392	3392
query73	761	730	307	307
query74	8814	8755	8584	8584
query75	2792	2814	2477	2477
query76	3908	1036	653	653
query77	516	377	273	273
query78	9853	9743	9152	9152
query79	1604	844	592	592
query80	645	566	463	463
query81	508	260	225	225
query82	212	142	110	110
query83	253	255	235	235
query84	260	128	94	94
query85	894	510	459	459
query86	381	328	324	324
query87	2840	2852	2749	2749
query88	3191	2232	2218	2218
query89	374	349	326	326
query90	2170	146	143	143
query91	170	164	140	140
query92	83	63	63	63
query93	1615	900	525	525
query94	570	316	293	293
query95	579	374	306	306
query96	585	458	201	201
query97	2299	2351	2272	2272
query98	228	209	195	195
query99	584	593	523	523
Total cold run time: 253740 ms
Total hot run time: 172457 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 26.9 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ea91c2dbc71d1824222d0ca9cf3c6844d48a0504, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.05
query3	0.27	0.09	0.08
query4	1.61	0.11	0.10
query5	0.27	0.27	0.26
query6	1.16	0.66	0.66
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.56	0.50	0.50
query10	0.55	0.56	0.55
query11	0.15	0.10	0.09
query12	0.13	0.10	0.10
query13	0.61	0.58	0.59
query14	0.94	0.95	0.95
query15	0.80	0.80	0.81
query16	0.39	0.40	0.40
query17	1.08	1.05	1.01
query18	0.23	0.22	0.21
query19	1.93	1.85	1.84
query20	0.02	0.01	0.01
query21	15.40	0.27	0.14
query22	5.24	0.05	0.05
query23	16.04	0.30	0.10
query24	1.43	0.35	0.35
query25	0.08	0.07	0.07
query26	0.14	0.13	0.13
query27	0.09	0.05	0.08
query28	3.69	1.06	0.88
query29	12.53	3.85	3.13
query30	0.28	0.13	0.12
query31	2.83	0.65	0.40
query32	3.25	0.57	0.46
query33	3.01	3.06	3.07
query34	16.73	5.11	4.41
query35	4.49	4.46	4.44
query36	0.63	0.50	0.49
query37	0.12	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.14	0.13
query41	0.09	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 97.35 s
Total hot run time: 26.9 s

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@hubgeter
Copy link
Contributor Author

hubgeter commented Jan 8, 2026

run buildall

@hubgeter
Copy link
Contributor Author

hubgeter commented Jan 9, 2026

run buildall

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.57% (1784/2242)
Line Coverage 64.75% (31726/48997)
Region Coverage 65.40% (15774/24120)
Branch Coverage 55.98% (8379/14968)

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

TPC-H: Total hot run time: 31788 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4cf9a7acc6d85ce46169845bccf775720b3fc25e, data reload: false

------ Round 1 ----------------------------------
q1	17610	4229	4143	4143
q2	2022	368	258	258
q3	10532	1342	720	720
q4	10343	811	313	313
q5	9718	2168	1833	1833
q6	248	171	136	136
q7	933	813	656	656
q8	9279	1430	1120	1120
q9	5275	4601	4718	4601
q10	6856	1802	1430	1430
q11	506	302	284	284
q12	752	743	572	572
q13	17812	3875	3104	3104
q14	306	308	277	277
q15	593	521	506	506
q16	711	690	632	632
q17	670	742	613	613
q18	6819	6514	6446	6446
q19	1103	967	592	592
q20	391	358	244	244
q21	2997	2411	2321	2321
q22	1037	1023	987	987
Total cold run time: 106513 ms
Total hot run time: 31788 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4158	4100	4063	4063
q2	356	398	334	334
q3	2121	2616	2220	2220
q4	1322	1747	1324	1324
q5	4091	4081	4076	4076
q6	219	170	130	130
q7	1918	2188	1883	1883
q8	2613	2480	2468	2468
q9	7231	7219	7269	7219
q10	2548	2742	2323	2323
q11	571	478	475	475
q12	728	748	591	591
q13	3668	4236	3516	3516
q14	295	306	290	290
q15	566	512	492	492
q16	652	710	620	620
q17	1169	1368	1430	1368
q18	7950	7748	7982	7748
q19	927	806	833	806
q20	1983	2047	2009	2009
q21	4748	4203	4171	4171
q22	1059	1033	974	974
Total cold run time: 50893 ms
Total hot run time: 49100 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172805 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4cf9a7acc6d85ce46169845bccf775720b3fc25e, data reload: false

query5	4370	568	435	435
query6	360	242	246	242
query7	4215	473	271	271
query8	343	254	243	243
query9	8768	2645	2622	2622
query10	531	383	331	331
query11	15359	15096	14893	14893
query12	171	119	115	115
query13	1272	507	373	373
query14	6227	3047	2832	2832
query14_1	2689	2631	2659	2631
query15	207	190	170	170
query16	998	480	452	452
query17	1085	655	560	560
query18	2421	431	334	334
query19	220	223	191	191
query20	122	116	115	115
query21	220	143	123	123
query22	3900	4137	3978	3978
query23	16043	15556	15383	15383
query23_1	15481	15344	15338	15338
query24	7360	1528	1178	1178
query24_1	1193	1180	1195	1180
query25	535	450	395	395
query26	1253	269	165	165
query27	2750	447	283	283
query28	4614	2113	2113	2113
query29	794	566	440	440
query30	315	244	212	212
query31	773	622	541	541
query32	75	69	71	69
query33	530	332	274	274
query34	911	912	528	528
query35	730	757	676	676
query36	836	895	771	771
query37	124	96	76	76
query38	2738	2708	2757	2708
query39	789	776	746	746
query39_1	712	718	712	712
query40	220	132	119	119
query41	67	63	64	63
query42	105	103	102	102
query43	471	449	413	413
query44	1317	746	736	736
query45	187	182	175	175
query46	853	955	599	599
query47	1447	1435	1407	1407
query48	311	331	245	245
query49	605	417	342	342
query50	637	278	207	207
query51	3803	3748	3951	3748
query52	107	111	98	98
query53	289	334	271	271
query54	323	271	255	255
query55	81	80	70	70
query56	298	301	293	293
query57	1000	1014	915	915
query58	268	250	255	250
query59	2158	2221	2159	2159
query60	317	320	293	293
query61	158	158	155	155
query62	397	359	316	316
query63	298	265	277	265
query64	5039	1304	1003	1003
query65	3784	3702	3689	3689
query66	1442	415	316	316
query67	15885	14675	14724	14675
query68	2734	1034	752	752
query69	453	365	327	327
query70	1019	968	981	968
query71	348	309	283	283
query72	6138	3478	3464	3464
query73	584	717	305	305
query74	8806	8767	8636	8636
query75	2792	2852	2438	2438
query76	2845	1072	662	662
query77	335	370	281	281
query78	9738	9948	9165	9165
query79	1108	917	599	599
query80	648	581	477	477
query81	484	267	231	231
query82	1367	144	109	109
query83	373	254	236	236
query84	263	119	103	103
query85	882	508	471	471
query86	381	320	314	314
query87	2883	2921	2772	2772
query88	3279	2243	2225	2225
query89	399	362	337	337
query90	1931	154	154	154
query91	170	165	149	149
query92	72	65	63	63
query93	957	895	529	529
query94	457	317	293	293
query95	576	337	364	337
query96	594	462	203	203
query97	2299	2409	2306	2306
query98	237	201	197	197
query99	604	572	494	494
Total cold run time: 248928 ms
Total hot run time: 172805 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants