Index usage stats

Every sql server developer/dba should have some “utility” scripts, for looking at particular point.
One of this script is about index usage , that is looking to a table and get information about index, how they are used, statistics …


set nocount on
DECLARE 
	@objname nvarchar(128);
select OBJECT_NAME(i.object_id) as tblName
	,i.name	
	,i.is_disabled as esteDezactivat
	,STUFF(ca.IndexColumnsNames,1,1,'') as IndexColumns
	,STUFF(ia.IncludedColumnsNames,1,1,'') as IncludedColumns
	,STATS_DATE(ius.object_id, i.index_id) as  Stat_Date
	,ps.reserved_page_count as noPages
	,ius.user_seeks
	,ius.user_scans
	,ius.user_lookups
	,ius.last_user_seek
	,ius.last_user_scan
	--,i.*
	,case when i.index_id  =1 then 1 else 0 end as is_clustered
	,i.is_unique 
	,i.is_primary_key
	,i.is_unique_constraint
	,i.is_disabled
	,i.has_filter
	,i.filter_definition
	,ps.data_compression_desc
	,'ALTER INDEX ' +  QUOTENAME(i.name) + ' ON ' + QUOTENAME(OBJECT_SCHEMA_NAME(i.object_id )) + '.'+ QUOTENAME(OBJECT_NAME(i.object_id)) + ' DISABLE;' as DisableStmt
	,'DROP INDEX ' +  QUOTENAME(i.name) + ' ON ' + QUOTENAME(OBJECT_SCHEMA_NAME(i.object_id )) + '.'+ QUOTENAME(OBJECT_NAME(i.object_id)) + ';'  as dropStmt
from
	sys.objects as o
	left join sys.indexes as i
		on o.object_id = i.object_id
	left join sys.dm_db_index_usage_stats as ius
		ON ius.object_id = i.object_id
			AND ius.index_id = i.index_id	
			AND ius.database_id = DB_ID(DB_Name())
	cross apply
		(select ', '+ c.name +''
			from sys.index_columns as ic
					inner join sys.columns as c
						on c.object_id = ic.object_id
						and c.column_id = ic.column_id
			where ic.object_id = i.object_id
				and ic.index_id = i.index_id
				and ic.is_included_column = 0
			order by ic.key_ordinal	
			for xml path('')
		)ca(IndexColumnsNames)
	outer apply
		(select ', '+ c.name +''
			from sys.index_columns as ic
					inner join sys.columns as c
						on c.object_id = ic.object_id
						and c.column_id = ic.column_id
			where ic.object_id = i.object_id
				and ic.index_id = i.index_id
				and ic.is_included_column = 1
			order by ic.index_column_id ASC
			for xml path('')
		)ia(IncludedColumnsNames)
	left join (select		
					ps.object_id
					,ps.index_id
					, sum(ps.reserved_page_count) as reserved_page_count
					,p.data_compression_desc as data_compression_desc
				from sys.dm_db_partition_stats  as ps
					inner join sys.partitions as p
						ON ps.partition_id = p.partition_id
						AND ps.object_id = p.object_id
				group by ps.object_id,ps.index_id,p.data_compression_desc
				)ps
		on ps.object_id = i.object_id
			and ps.index_id = i.index_id
where
	1=1
	--ius.user_seeks  = 0  
	--and ius.user_scans = 0
	--and ius.user_lookups = 0
	--and i.type > 1 -- nonclustered index
	--and i.is_unique = 0 -- is not unique
	--and i.is_primary_key = 0 -- not PK
	--and i.is_unique_constraint = 0 --not Unique Constraint
	/********and i.is_disabled = 0 -- is active*******/
	--and i.has_filter = 0
	--and database_id = DB_ID(DB_Name())  -- database actuala
	and OBJECT_NAME(o.object_id) = @objname 
order by OBJECT_NAME(ius.object_id),IndexColumns,IncludedColumns, QUOTENAME(i.name);


Thanks,
S

Puzzle CE

Sql server 2019, I have this query, and I was wondered from where is coming the estimate of 19140:

select
rezProcesat
from tblRezervari
where 1=1
and rezProcesat is null
OPTION
(
QUERYTRACEON 3604, QUERYTRACEON 2363 –New CE Stats Report
)

And the estimate for it is: 19140


the output from TF2363 :

Begin selectivity computation

Input tree:

LogOp_Select

  CStCollBaseTable(ID=1, CARD=1.84981e+07 TBL: tblRezervari)

  ScaOp_Comp x_cmpIs

      ScaOp_Identifier QCOL: [Stefan_2019].[dbo].[tblRezervari].rezProcesat

      ScaOp_Const TI(datetime,Null,ML=8) XVAR(datetime,Not Owned,Value=NULL)

Plan for computation:

CSelCalcColumnInInterval

  Column: QCOL: [Stefan_2019].[dbo].[tblRezervari].rezProcesat

Loaded histogram for column COL: KeyCo2 from stats with id 13

Selectivity: 0.00103472

Stats collection generated:

CStCollFilter(ID=2, CARD=19140.4)

  CStCollBaseTable(ID=1, CARD=1.84981e+07 TBL: tblRezervari)

End selectivity computation


So, the value 19140 is base on 0.00103472 * 1.84981e+07


But , from where did we get the Selectivity: 0.00103472 ?
I have a filtered index on that column, with filtered clause on exact that predicate.



Now, the next value : CARD=1.84981e+07, translated to 18498100

This should be the cardinality of the table… but I don’t from where is that value.


I don’t exactly know from where the exact value 18498100 come from , but this probably would be a future post after more research

S

Statistics steps

How many steps , a statistic can have?
200 ?

Online documentation, is saying that 200 steps.

DBCC Show Statistics is saying 200 steps here.

Also, the dm_db_stats_histogram, is saying the same here

But , checking an index, I get: 201 , where first row is having the NULL and additional 200 steps.

steps, from indexes of a table.

S

Importance of Statistics and Update of it

How important is to have statistics up to date versus fragmentations.

I start with a fresh copy of WideWorldImporters , call it WideWorldImporters_Test, in case of already existing database.

use master
go

restore database WideWorldImporters_Test
from disk = ‘D:\Kituri\SQL Server\WideWorldImporters-Full.bak’
with
move ‘WWI_Primary’ to ‘D:\Databases\DATA\WideWorldImporters_Test.mdf’,
move ‘WWI_UserData’ to ‘D:\Databases\DATA\WideWorldImporters_Test_UserData.ndf’,
move ‘WWI_Log’ to ‘D:\Databases\DATA\WideWorldImporters_Test.ldf’,
move ‘WWI_InMemory_Data_1’ to ‘D:\Databases\DATA\WideWorldImporters_Test_InMemory_Data_1’,
replace, stats=10

Here is a simple query:

select *
from Sales.Orders
where PickedByPersonID = 1

The execution plan is:

Let’s take a look at fragmentation:

select i.name as idxName
,ips.avg_fragmentation_in_percent
,ips.fragment_count
,ips.avg_page_space_used_in_percent
,ips.page_count
,STATS_DATE(o.object_id, i.index_id) as statsDate
from sys.dm_db_index_physical_stats(db_id(‘WideWorldImporters_test’),OBJECT_ID(‘Sales.Orders’),NULL,NULL,’DETAILED’) as ips
inner join sys.objects as o
ON o.object_id = ips.object_id
inner join sys.indexes as i
ON o.object_id = i.object_id
AND ips.index_id = i.index_id
where
i.name =’FK_Sales_Orders_PickedByPersonID’

Output:

99 precent (external/logical fragmentation) with 53 percent (internal/physical fragmentation)
So, high fragmentation, and the index (FK_Sales_Orders_PickedByPersonID) is not picked.

Now, let’s update the statistic of this index :
update statistics Sales.Orders (FK_Sales_Orders_PickedByPersonID) With fullScan

Looking, again , at the fragmentation :

The index statistics are updated, but the fragmentation is still high.
Key Question: the execution plan is still the same , or …

select *
from Sales.Orders
where PickedByPersonID = 1
Execution plan has changed.


Conclusion:

1 – Have statistics up to date, is the most important thing
2 – Fragmentation is important, but not as statistics.
3 – Rebuilding an index is doing :
  • Put in order the pages
  • Update statistics with FULL SCAN

So, first do an update statistics with full scan, then run the query. If this doesn’t take you over the cross line, the next step is to rebuild it. And if this doesn’t take you , over the line, maybe you need to rewrite the Query or call someone.

S

Statistics update 4 – cached plans

Previous  post was about  statistics update and cached plans. Moving next, what happens when we have modification

Q: – It will  trigger the invalidation of existing execution plan, when there is a modification or what happens when we update the statistics (that have modifications) ?

A: – Let see!

The flow :

  1. Clear the cache – drop freeProcCach
  2. Check the cached plans sys.dm_exec_cached_plans and Recompilations/sec
  3. Run a sp =Website.SearchForCustomers
  4. Check the cached_plan and Recompilations/sec
  5. Do some modification: insert/update/delete
  6. Run the update statistics statement
  7. Check the cached_plan and Recompilations/sec
  8. Run sp =Website.SearchForCustomers
  9. Check the cached_plan and Recompilations/sec

Step1,2:

DBCC FreeProcCache;
go

SELECT cp.*
FROM sys.dm_exec_cached_plans as cp
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_db
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_obj
WHERE
pa_db.attribute =’dbid’
AND pa_db.value =DB_ID(‘WideWorldImporters’)
AND pa_obj.attribute =’objectid’
AND pa_obj.value =OBJECT_ID(‘Website.SearchForCustomers’)

(0 rows affected)

SELECT
counter_name,cntr_value
FROM sys.dm_os_performance_counters as pc
WHERE pc.counter_name = ‘SQL Re-Compilations/sec’

SQL Re-Compilations/sec 23

 

Step3:

EXEC Website.SearchForCustomers @SearchText =N’Aly’,@MaximumRowsToReturn =3

JSON_F52E2B61-18A1-11d1-B105-00805F49916B
{“Customers”:[{“CustomerID”:1056,”CustomerName”:”Kalyani Benjaree”,”PhoneNumber”:”(212) 555-0100″,”FaxNumber”:”(212) 555-0101″,”ct”:[{“CityName”:”Bowmansville”,”p”:[{“PrimaryContactFullName”:”Kalyani Benjaree”,”PrimaryContactPreferredName”:”Kalyani”}]}]}]}

 

Step4: Check the cached_plan and Re-compilations counter

SELECT cp.usecounts
,cp.plan_handle
FROM sys.dm_exec_cached_plans as cp
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_db
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_obj
WHERE
pa_db.attribute =’dbid’
AND pa_db.value =DB_ID(‘WideWorldImporters’)
AND pa_obj.attribute =’objectid’
AND pa_obj.value =OBJECT_ID(‘Website.SearchForCustomers’)

(1 row affected)

usecounts plan_handle
1 0x0500050061562F38304AC17EB501000001000000000000000000000000000000000000000000000000000000

SELECT
counter_name,cntr_value
FROM sys.dm_os_performance_counters as pc
WHERE pc.counter_name = ‘SQL Re-Compilations/sec’

SQL Re-Compilations/sec 23

 

Step5: do modification

BEGIN TRAN
UPDATE TOP (5000) sales.Customers
SET CustomerName = CustomerName + N’sbse’
ROLLBACK TRAN
(663 rows affected)

Step6: run update statistics

UPDATE STATISTICS Sales.Customers WITH FULLSCAN;

Commands completed successfully.

 

Step7: check cached_plan and Recompilations counter

SELECT cp.*
FROM sys.dm_exec_cached_plans as cp
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_db
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_obj
WHERE
pa_db.attribute =’dbid’
AND pa_db.value =DB_ID(‘WideWorldImporters’)
AND pa_obj.attribute =’objectid’
AND pa_obj.value =OBJECT_ID(‘Website.SearchForCustomers’)

(1 row affected)

plan_handle
0x0500050061562F38304AC17EB501000001000000000000000000000000000000000000000000000000000000

SELECT
counter_name,cntr_value
FROM sys.dm_os_performance_counters as pc
WHERE pc.counter_name = ‘SQL Re-Compilations/sec’

SQL Re-Compilations/sec 23

No changes after the update statistics statement in Re-compilations counter.

 

 

Step8: Run , again the sp = Website.SearchForCustomers

EXEC Website.SearchForCustomers @SearchText =N’Aly’,@MaximumRowsToReturn =3
(1 row affected)

JSON_F52E2B61-18A1-11d1-B105-00805F49916B
{“Customers”:[{“CustomerID”:1056,”CustomerName”:”Kalyani Benjaree”,”PhoneNumber”:”(212) 555-0100″,”FaxNumber”:”(212) 555-0101″,”ct”:[{“CityName”:”Bowmansville”,”p”:[{“PrimaryContactFullName”:”Kalyani Benjaree”,”PrimaryContactPreferredName”:”Kalyani”}]}]}]}

Step9: check cached_plan and Re-compilations counter

SELECT cp.*
FROM sys.dm_exec_cached_plans as cp
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_db
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_obj
WHERE
pa_db.attribute =’dbid’
AND pa_db.value =DB_ID(‘WideWorldImporters’)
AND pa_obj.attribute =’objectid’
AND pa_obj.value =OBJECT_ID(‘Website.SearchForCustomers’)

(1 row affected)

bucketid refcounts usecounts size_in_bytes memory_object_address cacheobjtype objtype plan_handle pool_id parent_plan_handle
30650 2 2 303104 0x000001B572C94060 Compiled Plan Proc 0x0500050061562F38304AC17EB501000001000000000000000000000000000000000000000000000000000000 2 NULL
you see the plan handle its:  0x0500050061562F38304AC17EB501000001000000000000000000000000000000000000000000000000000000
0x0500050061562F38304AC17EB501000001000000000000000000000000000000000000000000000000000000 – previous plan handle

and see the use counts: 2…

usecounts refcounts
2 2

SELECT
counter_name,cntr_value
FROM sys.dm_os_performance_counters as pc
WHERE pc.counter_name = ‘SQL Re-Compilations/sec’

SQL Re-Compilations/sec 24

 

Conclusion:

When the update statistics statement it’s trigger, and have modification in statistics object, the plan it’s not evicted from cache. Can be reused.

The changes are in Re-compilations counter.And this counter is increased NOT after the update statistics statement, but AFTER the execution of the stored procedure/query.

 

S

Statistics update 3 – cached plan

What happens when you  launch an update statistics for a table?

Q: – It will  trigger a recompilation of the sp and then the invalidation of existing execution plan ?

A: – It depends! A plan is evicted from cache when :

  • FreeProcCache or
  • sp_Recompile
  • FlushProcInDB
  • changing db compatibility mode
  • change sp_Configure server option
  • etc…

 

When there is no changes, the flow will be:

update statistics —> stop.

When there is changes in statistics :

update statistics —>will trigger a —->plan invalidated —-> recompilation.

No cached plan is marked for a recompilation or marked as invalid. You can use extended events : sqlserver.sql_statement_recompile or sys.dm_os_performance_counters – >SQL Re-Compliations/sec to see what plan is invalidated and triggered for recompilation.

  1. Clear the cache – drop freeProcCach
  2. Check the cached plans sys.dm_exec_cached_plans
  3. Run a sp =Website.SearchForCustomers
  4. Check the cached_plan
  5. Run the update statistics statement
  6. Check the cached_plan
  7. Run , again the sp = Website.SearchForCustomers
  8. Check the cached_plan , to see the plan handle.
  9. Check recompilation with sys.dm_os_performance_counters

As you can see, the stored procedure it’s only a select sp. So insert/update/delete. So , the table it’s not modified, and there for, the statistics objects are not modified. Key point here : statistics objects are not modified. So no need of an recompilation.

Step1,2:

DBCC FreeProcCache;
go

SELECT cp.*
FROM sys.dm_exec_cached_plans as cp
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_db
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_obj
WHERE
pa_db.attribute =’dbid’
AND pa_db.value =DB_ID(‘WideWorldImporters’)
AND pa_obj.attribute =’objectid’
AND pa_obj.value =OBJECT_ID(‘Website.SearchForCustomers’)

(0 rows affected)

SELECT
counter_name,cntr_value
FROM sys.dm_os_performance_counters as pc
WHERE pc.counter_name = ‘SQL Re-Compilations/sec’
or pc.counter_name = ‘SQL Compilations/sec’
/*SQL Re-Compilations/sec 23 */

In my case Re-Compilations/sec = 23

 

Step3:

EXEC Website.SearchForCustomers @SearchText =N’Aly’,@MaximumRowsToReturn =3

JSON_F52E2B61-18A1-11d1-B105-00805F49916B
{“Customers”:[{“CustomerID”:1056,”CustomerName”:”Kalyani Benjaree”,”PhoneNumber”:”(212) 555-0100″,”FaxNumber”:”(212) 555-0101″,”ct”:[{“CityName”:”Bowmansville”,”p”:[{“PrimaryContactFullName”:”Kalyani Benjaree”,”PrimaryContactPreferredName”:”Kalyani”}]}]}]}

Step4: Check the cached_plan

SELECT cp.*
FROM sys.dm_exec_cached_plans as cp
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_db
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_obj
WHERE
pa_db.attribute =’dbid’
AND pa_db.value =DB_ID(‘WideWorldImporters’)
AND pa_obj.attribute =’objectid’
AND pa_obj.value =OBJECT_ID(‘Website.SearchForCustomers’)

(1 row affected)

plan_handle
0x0500050061562F38304AC17EB501000001000000000000000000000000000000000000000000000000000000

Step5: run update statistics

UPDATE STATISTICS Sales.Customers WITH FULLSCAN;

Commands completed successfully.

 

Step6: check cached_plan

SELECT cp.*
FROM sys.dm_exec_cached_plans as cp
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_db
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_obj
WHERE
pa_db.attribute =’dbid’
AND pa_db.value =DB_ID(‘WideWorldImporters’)
AND pa_obj.attribute =’objectid’
AND pa_obj.value =OBJECT_ID(‘Website.SearchForCustomers’)

(1 row affected)

plan_handle
0x0500050061562F38304AC17EB501000001000000000000000000000000000000000000000000000000000000

SELECT
counter_name,cntr_value
FROM sys.dm_os_performance_counters as pc
WHERE pc.counter_name = ‘SQL Re-Compilations/sec’
or pc.counter_name = ‘SQL Compilations/sec’
/*SQL Re-Compilations/sec 23 */

 

 

The plan is there.

Step7: Run , again the sp = Website.SearchForCustomers

EXEC Website.SearchForCustomers @SearchText =N’Aly’,@MaximumRowsToReturn =3
(1 row affected)

JSON_F52E2B61-18A1-11d1-B105-00805F49916B
{“Customers”:[{“CustomerID”:1056,”CustomerName”:”Kalyani Benjaree”,”PhoneNumber”:”(212) 555-0100″,”FaxNumber”:”(212) 555-0101″,”ct”:[{“CityName”:”Bowmansville”,”p”:[{“PrimaryContactFullName”:”Kalyani Benjaree”,”PrimaryContactPreferredName”:”Kalyani”}]}]}]}

Step8: check cached_plan

SELECT cp.*
FROM sys.dm_exec_cached_plans as cp
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_db
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) as pa_obj
WHERE
pa_db.attribute =’dbid’
AND pa_db.value =DB_ID(‘WideWorldImporters’)
AND pa_obj.attribute =’objectid’
AND pa_obj.value =OBJECT_ID(‘Website.SearchForCustomers’)

(1 row affected)

bucketid refcounts usecounts size_in_bytes memory_object_address cacheobjtype objtype plan_handle pool_id parent_plan_handle
30650 2 2 303104 0x000001B572C94060 Compiled Plan Proc 0x0500050061562F38304AC17EB501000001000000000000000000000000000000000000000000000000000000 2 NULL
you see the plan handle its:  0x0500050061562F38304AC17EB501000001000000000000000000000000000000000000000000000000000000
0x0500050061562F38304AC17EB501000001000000000000000000000000000000000000000000000000000000 – previous plan handle

and see the use counts: 2…

usecounts refcounts
2 2

Because no modification on statistics, the plan is not invalidated.

Step9:

SELECT *
  FROM sys.dm_os_performance_counters as pc
  WHERE counter_name =‘SQL Re-Compilations/sec’
/*SQL Re-Compilations/sec 23 */

Next post, when we have modifications , what happens with recompilation and plan cached ?

 

#tsql #t-sql #sqlserver #msSqlServer #Recompilation #statistics #update

S

Statistics update 2

Previous post about statistics was talking about updating them , base on some value, value that exists in a utility table.

The present post is referring also about update statistics, but now base on a percentage value, threshold . Maybe this is more flexible . This value could also be present in a utility table , it’s up to anyone how to have it.

In my case, I use a value of 5.0, so that I will not update all objects, only some of them. Like I said, this value, could be extracted from a utility table , and could be unique for an particularly statistics object (stats_id) , or an object( object_id ) or database…

 

The starting point is the calculation of the percentage value :  sp.modification_counter/sp.rows

where :

  •  sp.modification_count = actual value of modification
  • sp.rows = number of rows in that object

The actual formula is :

CAST((sp.modification_counter/(sp.rows * 1.0)) *100.0  AS decimal(6,4))

 

This query will list all the statistics of the object

SELECT
OBJECT_SCHEMA_NAME(s.object_id)
+ ‘.’ + OBJECT_NAME(s.object_id) as objectName
, s.name as statisticsName
,s.object_id as objectID
,s.stats_id as statsID
,sp.modification_counter
,sp.rows as noRows
,CAST((sp.modification_counter/(sp.rows *1.0)) *100.0 AS decimal(6,4)) as modification_Percent
,sp.last_updated
,sc.colNames
FROM sys.stats as s
CROSS APPLY sys.dm_db_stats_properties(s.object_id,s.stats_id) as sp
CROSS APPLY(SELECT STRING_AGG(c.name,’,’) as colNames
FROM sys.stats_columns as sc
INNER JOIN sys.columns as c
ON sc.column_id = c.column_id
and sc.object_id = c.object_id
WHERE sc.object_id = s.object_id
and sc.stats_id = s.stats_id
) as sc
WHERE s.object_id = OBJECT_ID(‘Sales.Orders’)

 

objectName statisticsName objectID statsID modification_counter noRows modification_Percent last_updated colNames
Sales.Orders PK_Sales_Orders 1154103152 1 3966 69629 5.6959 2016-06-02 10:39:54.2533333 OrderID
Sales.Orders FK_Sales_Orders_CustomerID 1154103152 2 0 73595 0.0000 2018-07-25 10:52:18.0733333 CustomerID
Sales.Orders FK_Sales_Orders_SalespersonPersonID 1154103152 3 0 73595 0.0000 2018-07-25 10:52:18.4233333 SalespersonPersonID
Sales.Orders FK_Sales_Orders_PickedByPersonID 1154103152 4 4751 71018 6.6899 2016-06-02 10:43:25.6400000 PickedByPersonID
Sales.Orders FK_Sales_Orders_ContactPersonID 1154103152 5 2012 71583 2.8107 2016-06-02 10:45:05.4033333 ContactPersonID
Sales.Orders _WA_Sys_0000000E_44CA3770 1154103152 6 0 73595 0.0000 2018-07-25 10:53:22.7400000 PickingCompletedWhen
Sales.Orders _WA_Sys_0000000A_44CA3770 1154103152 7 3495 70100 4.9857 2016-06-02 10:41:14.3700000 IsUndersupplyBackordered

Adding the  additional predicate on the percentage …

WHERE s.object_id = OBJECT_ID(‘Sales.Orders’)
AND CAST((sp.modification_counter/(sp.rows *1.0)) *100.0 AS decimal(6,4)) > CAST(5.0 AS decimal(6,4))

will have this records outputted :

objectName statisticsName modification_counter noRows modification_Percent
Sales.Orders PK_Sales_Orders 3966 69629 5.6959
Sales.Orders FK_Sales_Orders_PickedByPersonID 4751 71018 6.6899

Now, just prepare the dynamic sql for update

 
DECLARE @nvc_SQLText AS nvarchar(4000) = N”;

SELECT
@nvc_SQLText = STRING_AGG(N’UPDATE STATISTICS ‘
+ OBJECT_SCHEMA_NAME(s.object_id)
+ N’.’ + OBJECT_NAME(s.object_id)
+N'(‘ + s.name +N’)’
+ N’ WITH FULLSCAN;’ + char(10) ,N”)
FROM sys.stats as s
CROSS APPLY sys.dm_db_stats_properties(s.object_id,s.stats_id) as sp
WHERE s.object_id = OBJECT_ID(‘Sales.Orders’)
AND CAST((sp.modification_counter/(sp.rows *1.0)) *100.0 AS decimal(6,4)) > CAST(5.0 AS decimal(6,4))

SELECT @nvc_SQLText

 

output:

UPDATE STATISTICS Sales.Orders(PK_Sales_Orders) WITH FULLSCAN;
UPDATE STATISTICS Sales.Orders(FK_Sales_Orders_PickedByPersonID) WITH FULLSCAN;

 

and run it :

EXEC sp_executeSQL @nvc_SQLText;

 

 

Now, check if you have some objects over the threshold :

 
SELECT
OBJECT_SCHEMA_NAME(s.object_id)
+ ‘.’ + OBJECT_NAME(s.object_id) as objectName
,s.name as statisticsName
,sp.modification_counter
,sp.rows as noRows ,CAST((sp.modification_counter/(sp.rows *1.0)) *100.0 AS decimal(6,4)) as modification_Percent
FROM sys.stats as s
CROSS APPLY sys.dm_db_stats_properties(s.object_id,s.stats_id) as sp
CROSS APPLY(SELECT STRING_AGG(c.name,’,’) as colNames
FROM sys.stats_columns as sc
INNER JOIN sys.columns as c
ON sc.column_id = c.column_id
and sc.object_id = c.object_id
WHERE sc.object_id = s.object_id
and sc.stats_id = s.stats_id
) as sc

WHERE s.object_id = OBJECT_ID(‘Sales.Orders’)
AND CAST((sp.modification_counter/(sp.rows *1.0)) *100.0 AS decimal(6,4))
> CAST(5.0 AS decimal(6,4))

 

output :

No rows.

#t-sql #sqlserver #statistics #update

 

 

S

 

 

 

 

Statistics update

How about a home made script that is checking the statistics modification counter, and base on this value, check it with a company / object threshold and if this value is above that threshold, trigger an update of that particular statistics.

For this we must have a utility table that hold the threshold for which will trigger the update. This is kept on object_id and stat_id ( for a more flexibility) but could be only one per database or one per object_id …

ex:  I have a statistics  : _WA_Sys_0000000E_44CA3770 , if this has a value = 4563 and the threshold it’s like 3000. In this case we should trigger , for this statistics, an update.

 

Here are the steps:

  1. – List the statistics (for indexes and for columns) of an object. In this case for Sales.Orders
  2. – Show the field that holds the modification counter associated
    with this statistics
  3. – Do an update/delete in a transaction with rollback – for field  PickingCompletedWhen
  4. – Create utility table
  5. – Create dynamic string to run the update statistics
  6. – Check the value after update statistics

 

I will start with step 3   , the utility table.This should be save in a “utility” database.

use db_workspace
go

DROP TABLE IF EXISTS dbo.utility_stats;
GO

CREATE TABLE dbo.utility_stats
(
uts_id INT IDENTITY(1,1) NOT NULL
,uts_object_id INT NOT NULL
,uts_stats_id INT NOT NULL
,uts_threshold INT NULL
);
GO

ALTER TABLE dbo.utility_stats
ADD CONSTRAINT pk_utility_stats__uts_object_id_uts_stats_id
PRIMARY KEY CLUSTERED(uts_object_id,uts_stats_id);
GO

 

Step 1: List the statistics (for indexes and for columns) of an object.
In this case for Sales.Orders

SELECT
OBJECT_SCHEMA_NAME(s.object_id)
+ ‘.’ + OBJECT_NAME(s.object_id) as objectName
, s.name as statisticsName
,s.object_id as objectID
FROM sys.stats as s
WHERE s.object_id = OBJECT_ID(‘Sales.Orders’)

objectName statisticsName objectID
Sales.Orders PK_Sales_Orders 1154103152
Sales.Orders FK_Sales_Orders_CustomerID 1154103152
Sales.Orders FK_Sales_Orders_SalespersonPersonID 1154103152
Sales.Orders FK_Sales_Orders_PickedByPersonID 1154103152
Sales.Orders FK_Sales_Orders_ContactPersonID 1154103152
Sales.Orders _WA_Sys_0000000E_44CA3770 1154103152
Sales.Orders _WA_Sys_0000000A_44CA3770 1154103152

Step 2: Show the fields that holds the modification counter associated
with this statistics

SELECT
OBJECT_SCHEMA_NAME(s.object_id)
+ ‘.’ + OBJECT_NAME(s.object_id) as objectName
, s.name as statisticsName
,s.object_id as objectID
,s.stats_id as statsID
,sp.modification_counter
,sp.last_updated
,sc.colNames
FROM sys.stats as s
CROSS APPLY sys.dm_db_stats_properties(s.object_id,s.stats_id) as sp
CROSS APPLY(SELECT STRING_AGG(c.name,’,’) as colNames
FROM sys.stats_columns as sc
INNER JOIN sys.columns as c
ON sc.column_id = c.column_id
and sc.object_id = c.object_id
WHERE sc.object_id = s.object_id
and sc.stats_id = s.stats_id
) as sc
WHERE s.object_id = OBJECT_ID(‘Sales.Orders’)

objectName statisticsName objectID statsID modification_counter last_updated colNames
Sales.Orders PK_Sales_Orders 1154103152 1 3966 2016-06-02 10:39:54.2533333 OrderID
Sales.Orders FK_Sales_Orders_CustomerID 1154103152 2 0 2018-07-25 10:52:18.0733333 CustomerID
Sales.Orders FK_Sales_Orders_SalespersonPersonID 1154103152 3 0 2018-07-25 10:52:18.4233333 SalespersonPersonID
Sales.Orders FK_Sales_Orders_PickedByPersonID 1154103152 4 4751 2016-06-02 10:43:25.6400000 PickedByPersonID
Sales.Orders FK_Sales_Orders_ContactPersonID 1154103152 5 2012 2016-06-02 10:45:05.4033333 ContactPersonID
Sales.Orders _WA_Sys_0000000E_44CA3770 1154103152 6 0 2018-07-25 10:53:22.7400000 PickingCompletedWhen
Sales.Orders _WA_Sys_0000000A_44CA3770 1154103152 7 3495 2016-06-02 10:41:14.3700000 IsUndersupplyBackordered

Step 3 : Do an update/delete in a transaction with rollback – for field PickingCompletedWhen
DECLARE @i_modification_counter_before int = 0
,@i_modification_counter_after int = 0;

SELECT
@i_modification_counter_before = sp.modification_counter
FROM sys.stats as s
CROSS APPLY sys.dm_db_stats_properties(s.object_id,s.stats_id) as sp
WHERE s.object_id = OBJECT_ID(‘Sales.Orders’)
AND s.stats_id = 6
BEGIN TRAN
UPDATE TOP(1000) Sales.Orders
SET PickingCompletedWhen = SYSDATETIME()

SELECT
@i_modification_counter_after = sp.modification_counter
FROM sys.stats as s
CROSS APPLY sys.dm_db_stats_properties(s.object_id,s.stats_id) as sp
WHERE s.object_id = OBJECT_ID(‘Sales.Orders’)
AND s.stats_id = 6

ROLLBACK TRAN
SELECT @i_modification_counter_before as beforeCounter
,@i_modification_counter_after as afterCounter
,@i_modification_counter_after-@i_modification_counter_before as differ

beforeCounter afterCounter differ
7480 8480 1000

 

Step 4 : utility table and populate with thresholds:

INSERT INTO db_workspace.dbo.utility_stats(uts_object_id,uts_stats_id,uts_threshold)
SELECT
s.object_id as objectID
,s.stats_id as statsID
,5000
FROM WideWorldImporters.sys.stats as s
WHERE s.object_id = OBJECT_ID(‘WideWorldImporters.Sales.Orders’);
GO

 

SELECT * FROM db_workspace.dbo.utility_stats
/*
uts_id uts_object_id uts_stats_id uts_threshold
1 1154103152 1 5000
2 1154103152 2 5000
3 1154103152 3 5000
4 1154103152 4 5000
5 1154103152 5 5000
6 1154103152 6 5000
7 1154103152 7 5000
*/

Step 5: Create dynamic string to run the update statistics , for the statistics with modification counter above the threshold.

 

DECLARE @nvc_SQLText AS nvarchar(4000) = N”;

SELECT
@nvc_SQLText = STRING_AGG(N’ UPDATE STATISTICS ‘
+ OBJECT_SCHEMA_NAME(s.object_id)
+ N’.’ + OBJECT_NAME(s.object_id)
+N'(‘ + s.name +N’)’
+ N’ WITH FULLSCAN ‘ + char(10) + char(13) ,N’;’)
FROM sys.stats as s
CROSS APPLY sys.dm_db_stats_properties(s.object_id,s.stats_id) as sp
LEFT JOIN db_workspace.dbo.utility_stats as us
ON us.uts_object_id = s.object_id
AND us.uts_stats_id = s.stats_id
WHERE s.object_id = OBJECT_ID(‘Sales.Orders’)
AND sp.modification_counter >= us.uts_threshold

SELECT @nvc_SQLText

/* output :
UPDATE STATISTICS Sales.Orders(_WA_Sys_0000000E_44CA3770) WITH FULLSCAN
*/

EXEC sp_executeSQL @nvc_SQLText

GO

 

Step 6: Check the value after update statistics, for that particullary statistics:
_WA_Sys_0000000E_44CA3770

use WideWorldImporters
go

SELECT
OBJECT_SCHEMA_NAME(s.object_id)
+ ‘.’ + OBJECT_NAME(s.object_id) as objectName
, s.name as statisticsName
,s.object_id as objectID
,s.stats_id as statsID
,sp.modification_counter
,sp.last_updated
,sc.colNames
,us.uts_threshold
FROM sys.stats as s
CROSS APPLY sys.dm_db_stats_properties(s.object_id,s.stats_id) as sp
CROSS APPLY(SELECT STRING_AGG(c.name,’,’) as colNames
FROM sys.stats_columns as sc
INNER JOIN sys.columns as c
ON sc.column_id = c.column_id
and sc.object_id = c.object_id
WHERE sc.object_id = s.object_id
and sc.stats_id = s.stats_id
) as sc
LEFT JOIN db_workspace.dbo.utility_stats as us
ON us.uts_object_id = s.object_id
AND us.uts_stats_id = s.stats_id
WHERE s.object_id = OBJECT_ID(‘Sales.Orders’)

/*
objectName statisticsName objectID statsID modification_counter last_updated colNames uts_threshold
Sales.Orders _WA_Sys_0000000E_44CA3770 1154103152 6 0 2018-07-25 10:53:22.7400000 PickingCompletedWhen 5000
*/

That’s it fox!

 

S

 

Page Split

Page split occurs when there is not enough space in a particular data page to insert/update a specific record. There are 3 ways that generate Page splits:

  • End/Sequential/Identity Insert  = good page split = when we insert at the end of the table . ex: imagine you have an identity field  which is also PK and clustered index , and all new records would have id+1
  • Mid lvl table Inserts = bad page split
  • Updates = bad page split = when we’re updating a record on a page that don’t have enough space to accommodate the extra space. For this needs extra space and voila , page split

For bad page split, this is bad because there is additional information that needs to be save in the transaction log.

  • adding a new page
  • move half of records to the new page (insert into the new page and delete from old page  = mark it as delete )
  • update link pointers in the pages

 

Here is a sample of it:

We have a simple table , with id field clustered :

–create a table with an id and some string , so that 2 records will be stored in a page
drop table if exists dbo.pageSplit;
create table dbo.pageSplit
(
id int not null,
myChar char(3500) not null,
constraint pk_pageSplit_id PRIMARY KEY CLUSTERED(id)
)
go

Now ,let start inserting one record, starting with value of id = 3 , and see the log data generated.

begin transaction
insert into dbo.pageSplit(id,myChar)
values(3,REPLICATE(‘3’,3500));

select database_transaction_log_bytes_used from sys.dm_tran_database_transactions where database_id = DB_ID(‘db_workspace’)
–3796
commit transaction

 

Log data generated

pageSplit_Insert_Value_3

 

Next step, insert value 2:

begin transaction
insert into dbo.pageSplit(id,myChar)
values(2,REPLICATE(‘2’,3500));

select database_transaction_log_bytes_used from sys.dm_tran_database_transactions where database_id = DB_ID(‘db_workspace’)
–3756
commit transaction

pageSplit_Insert_Value_2

pageSplit_Insert_Value_2_after

 

Step 3, insert a record with value 1.

begin transaction
insert into dbo.pageSplit(id,myChar)
values(1,REPLICATE(‘1’,3500));

select database_transaction_log_bytes_used from sys.dm_tran_database_transactions where database_id = DB_ID(‘db_workspace’)
–5784
commit transaction

Log data generated is 5784, almost double (70%) .wow…

You can monitor it with perform counters:

SQLServer:AccessMethods:Page Splits/Sec

Or from dmvs:

select cntr_value
from sys.dm_os_performance_counters
where counter_name = ‘Page Splits/sec’

Code here

S

 

 

 

 

 

 

 

Stats

Statistics are one of the most important objects./aspect from SQL Server.

Today, some  small knowledge about it. For deeper and more information, go search Microsoft site for it.

Statistics are the base when optimizer will choose, when it will generate an execution plan.

Statistics can be created:

  •  automatically (if you have AUTO_CREATE_STATISTICS = ON) for an index or if optimizer think it will benefit. ( image a where clause : WHERE city = ‘XXX’ – no index on it, and optimizer will choose to create statistics on it)
  • manually –   CREATE STATISTICS nameStats

You can view this information  and different aspect with following commands:

  • DBCC Show_Statistics
    • WITH STATS HEADER ; WITH HISTROGRAM
  • sys.dm_db_stats_properties
  • sys.dm_db_stats_histrogram
  • sys.stats
  • sys.stats_columns
  • sys.dm_db_incremental_stats_properties
  • STATS_DATE  – this will present last date time when was updated

If you have the option AUTO_UPDATE_STATISTICS = ON, then there is a threshold that will trigger an update of your statistics:

  • regular tables:
    • 0 rows in it – any modification will trigger
    • (0 , 500]  rows –  threshold will be  when you have 500+ modification
    • (500, …) – 20% + 500
  • temp table #
    • [0,6) , threshold  will be 6
    • [6,500] , threshold  = 500
    • (500,…)  – 20% + 500
  • @ variable tables
    • no threshold for recompilation

To change this recompiled threshold, you can use Trace Flag 2371. This is included on , in sql server 2016, compatibility level 130.

So , next time when a query is executed (one that is using the statistics involved), it is checking if statistics are up-to-date. NOW , there are outdated and the compiled plan is removed from cache, then a recompilation is triggered and  recompilation will trigger the update of statistics.

Also, when a query is compiled for the first time, then statistics is also checked for up-do-date. if they are outdated, then also will trigger the auto Update.

To view the modifications in a table, you can use different approach:

  • modification_counter from sys.dm_db_stats_properties
  • rowmodctr from sys.sysindexes – this is an old approach  and could be removed in future
  • leaf_XXX_count columns from sys.dm_db_index_operational_stats

 

Starting with sql server 2014,  we have something new – Incremental Statistics – applying  at partition level.  This means that is scanning the partition to get/update statistics; including the 20% threshold.
SET AUTO_CREATE_STATISTICS ON(INCREMENTAL = ON)  – setting at db level
or at index level –  WITH (STATISTICS_INCREMENTAL=ON)
You can check it with sys.stats – column is_incremental

 

UPDATE STATISTICS can be :

  • FULL SCAN
  • SAMPLE

Rebuilding an index will update statistics with full scan.

 

More information can be written about statistics. This is only my view about it.

ps: I hope this info are accurate.

 

 

S