Filtered index gotcha

We added a filtered index to a table today and a SQL Agent job that updates that table started to fail with this error message:

Msg 1934, Sev 16, State 1, Line 290 : UPDATE failed because the following SET options have incorrect settings: ‘QUOTED_IDENTIFIER’. Verify that SET options are correct for use with indexed views and/or indexes on computed columns and/or filtered indexes and/or query notifications and/or XML data type methods and/or spatial index operations. [SQLSTATE 42000]

It turns out that you must have certain options enabled in your session in order to use or update a filtered index.

All these must be ON:

  • ANSI_NULLS
  • ANSI_PADDING
  • ANSI_WARNINGS
  • ARITHABORT
  • CONCAT_NULL_YIELDS_NULL
  • QUOTED_IDENTIFIER

And this one must be OFF:

  • NUMERIC_ROUNDABORT

See Books Online for details: https://msdn.microsoft.com/en-us/library/ms188783.aspx (it’s about half way down the page in the Remarks section).

So what happened here? Well it looks like SQL Agent does not, by default, set QUOTED_IDENTIFIER on. I can’t find any clear official information on this but that’s what I’m seeing in SQL Profiler. I also couldn’t find any way to override this in SQL Agent’s configuration settings.

Anyway, I added SET QUOTED_IDENTIFIER ON at the start of the job step and it worked fine.

Handling service failures

There are times when a SQL-related service has a problem and just stops. Windows services are like that sometimes. Unless you do some additional configuration, you’re not going to know about it until users start shouting at you. Here’s what you can do to make that less likely.

Open up Services and double click on the SQL Server service. By default the Recovery tab looks like this:

20160304_default_recovery_tab

If this services stops unexpectedly you won’t know. Not until your phone starts ringing off the hook!

Here’s how I suggest you improve things:

20160304_configured_recovery_tab

The first time the service fails the system will wait a minute and try to restart it. A second failure will get the same treatment. If it fails a third time you need to know so the third failure runs a PowerShell script to send an alert email.

This won’t kick in if the service is stopped by hand or by something like a SQL Server service pack. It’s only for when things go wrong.

The full contents of the Command line parameters box goes like this:

-file "\\server_name\share_name\email_service_crash.ps1" -ServerName "server_name" -ServiceName "SQL Server"

Clearly you’ll need to edit the file path, server name and service name to suit your environment.

The powershell script, email_service_crash.ps1, is this:

Param (
    [Parameter(Mandatory=$True)]
    [String]$ServerName,

    [Parameter(Mandatory=$True)]
    [String]$ServiceName
)

$from = "from_address"
$to = "to_address"
$subject = $ServiceName + " service is down on " + $ServerName
$body = "Find out why and get it started again."

$creds = new-object System.Net.NetworkCredential("smtp_user_name", "smtp_user_password")
$smtp = new-object System.Net.Mail.SmtpClient("smtp_server_name")
$smtp.UseDefaultCredentials = "False"
$smtp.Credentials = $creds
$smtp.Send($from, $to, $subject, $body)

Again, you’ll need to edit some of the variables and parameters: $from, $to, and the SMTP server, user and password.

Set the script up and test it in a cmd window like this:

powershell.exe -file "\\server_name\share_name\email_service_crash.ps1" -ServerName "server_name" -ServiceName "SQL Server"

Once you’ve got it working, you can take the string from ‘-file’ onwards and use it in the service Recovery tab.

You might be asking what the checkbox Enable actions for stops with errors means. Good question. The documentation is no help but it turns out there are some situations where this stuff won’t kick in unless that box is ticked. For more details have a look at this question on Server Fault: http://serverfault.com/questions/72318/set-up-recovery-actions-to-take-place-when-a-service-fails

You can re-use this in the Recovery tab for each service you want to cover. Just remember to change the -ServiceName parameter you pass to the PowerShell script.

In praise of StatisticsParser.com

Here’s a great free addition to your query tuning toolbox that you may well not have heard about: StatisticsParser.com. It hasn’t been mentioned nearly enough in my opinion.

Here’s how the author, Richie Rump, puts it:

Sometimes reading SQL Server ouput from Statistics IO and Statistics Time can be a total drag. This page will help with that. Just paste in the output of Statistics IO and/or Statistics Time and press Parse. Your output will be formatted and totaled. Enjoy.

Here’s a quick look at what it does.

If you want to see how long each part of a query takes or what logical reads are involved you can set a couple of options on: statistics io and statistics time.

set statistics io on;
set statistics time on;

-- List all indexes in the current DB
select	t.name as TableName
	, i.name as IndexName
	, i.*
from	sys.indexes i
	inner join sys.objects t on t.object_id = i.object_id
where	i.type > 0
	and t.type = 'U'
	and t.is_ms_shipped = 0
	and t.name not like 'sys%'
order by t.name asc
	, i.type desc
	, i.name asc;

This adds extra output to the Messages tab when you run your query:

SQL Server parse and compile time:
   CPU time = 0 ms, elapsed time = 14 ms.

 SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 0 ms.

 SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 0 ms.

(1161 row(s) affected)
Table 'syspalvalues'. Scan count 0, logical reads 2322, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'syssingleobjrefs'. Scan count 0, logical reads 2403, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'sysschobjs'. Scan count 1, logical reads 2566, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'sysidxstats'. Scan count 634, logical reads 1472, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

 SQL Server Execution Times:
   CPU time = 47 ms,  elapsed time = 190 ms.

It’s not that readable, even for a simple example like this. Here’s where StatisticsParser comes in. Paste in this output and hit the Parse button to see the information laid out nice and clearly:

20160122_statisticsparser

I think this is brilliant.

Thank you, Richie.

SQL Server service startup failure

I just had a SQL Server service that didn’t start after a reboot of the machine. Error message: “The service did not start due to a logon failure. [0x8007042d]”. Hmm.

In the Local Security Policy editor the service account was no longer listed under the user rights assignment for Logon as a service. I discovered I had no permission to change that because it’s managed by a domain policy.

A search found this thread which helped me to sort it out. I tried resetting the service account from the SQL Server Configuration Manager and that didn’t fix the Logon as a service rights so I still had the same error. But when I reset the service account from Services in the Control Panel instead that re-applied the Logon as a service rights and the service started.

I still have no idea why that permission dropped off but at least now I know how to get it back again. It’s progress of a sort.

Developer database permissions

Here’s a suggestion on how you can grant developers plenty of permissions on a dev or test database without just adding them to the db_owner role (which is going too far).

I found a post by Brent Ozar talking about podcasts he likes. Thanks for the tips, Brent. I checked them out and I particularly like the SQL Data Partners podcast. Thanks for the excellent listening, Carlos. I’m working through the episodes and episode 8 ‘The Principal of Least Privilege’ with Robert Verrell (aka SQL Cowbell) really got me thinking.

Robert talked about why adding developers to the db_owner role is a bad idea, even on a dev server. Then he proposed adding them to a custom high-powered database role, db_developer, instead. This way they have enough rights to do things they need to but hopefully they can’t destroy the server. He blogged about it here.

I like the idea but we regularly refresh our our test databases by restoring from production backups and sanitising sensitive data. This process would undo the db_developer role so I wrapped it up in a stored procedure to make it easy to re-run on each relevant database at the end of the refresh job:

if exists (	select 	1
		from 	dbo.sysobjects
		where 	id = object_id(N'dbo.dba_AddDBDeveloperRole')
			and objectproperty(id, N'IsProcedure') = 1    )
	drop proc dbo.dba_AddDBDeveloperRole;
go

set ansi_nulls on;
set quoted_identifier on;
go

create proc dbo.dba_AddDBDeveloperRole
(
	@DBName		sysname
	, @Login	sysname
	, @DebugMode	bit	= 0
)
as

/*
---------------------------------------------------------------------------------------------------------------------------------------------------
Version : 1.01
Date    : 22/12/2015

Grant developers lots of permissions without the really dangerous ones like db_owner or sysadmin.
Based on this blog post: http://sqlcowbell.com/wordpress/why-nobody-ever-needs-the-db_owner-role/.

Original script from that post:

CREATE ROLE [db_developer] AUTHORIZATION [dbo]
 GRANT ALTER ANY APPLICATION ROLE TO [db_developer]
 GRANT ALTER ANY ASSEMBLY TO [db_developer]
 GRANT ALTER ANY DATABASE DDL TRIGGER TO [db_developer]
 GRANT ALTER ANY DATASPACE TO [db_developer]
 GRANT ALTER ANY FULLTEXT CATALOG TO [db_developer]
 GRANT ALTER ANY MESSAGE TYPE TO [db_developer]
 GRANT ALTER ANY SCHEMA TO [db_developer]
 GRANT CREATE AGGREGATE TO [db_developer]
 GRANT CREATE ASSEMBLY TO [db_developer]
 GRANT CREATE DATABASE DDL EVENT NOTIFICATION TO [db_developer]
 GRANT CREATE DEFAULT TO [db_developer]
 GRANT CREATE FULLTEXT CATALOG TO [db_developer]
 GRANT CREATE FUNCTION TO [db_developer]
 GRANT CREATE PROCEDURE TO [db_developer]
 GRANT CREATE ROLE TO [db_developer]
 GRANT CREATE RULE TO [db_developer]
 GRANT CREATE SCHEMA TO [db_developer]
 GRANT CREATE SERVICE TO [db_developer]
 GRANT CREATE SYNONYM TO [db_developer]
 GRANT CREATE TABLE TO [db_developer]
 GRANT CREATE TYPE TO [db_developer]
 GRANT CREATE VIEW TO [db_developer]
 GRANT CREATE XML SCHEMA COLLECTION TO [db_developer]
 GRANT DELETE TO [db_developer]
 GRANT EXECUTE TO [db_developer]
 GRANT INSERT TO [db_developer]
 GRANT REFERENCES TO [db_developer]
 GRANT SELECT TO [db_developer]
 GRANT SHOWPLAN TO [db_developer]
 GRANT UPDATE TO [db_developer]
 GRANT VIEW DATABASE STATE TO [db_developer]
 GRANT VIEW DEFINITION TO [db_developer]
---------------------------------------------------------------------------------------------------------------------------------------------------
*/

set nocount on;

declare @SQL nvarchar(4000);

print 'Database: ' + @DBName;

print 'Create role';
set @SQL =	'use [' + @DBName + ']'
		+ ' if not exists (select 1 from sys.database_principals where type_desc = ''DATABASE_ROLE'' and name = ''db_developer'')'
		+ ' create role [db_developer] authorization [dbo]';
if @DebugMode = 1
	print @SQL;
else
	exec sp_executesql @SQL;

print 'Grant permissions to the role';
set @SQL =	'use [' + @DBName + ']'
		+ ' grant alter any application role to [db_developer];'
		+ ' grant alter any assembly to [db_developer];'
		+ ' grant alter any database ddl trigger to [db_developer];'
		+ ' grant alter any dataspace to [db_developer];'
		+ ' grant alter any fulltext catalog to [db_developer];'
		+ ' grant alter any message type to [db_developer];'
		+ ' grant alter any schema to [db_developer];'
		+ ' grant create aggregate to [db_developer];'
		+ ' grant create assembly to [db_developer];'
		+ ' grant create database ddl event notification to [db_developer];'
		+ ' grant create default to [db_developer];'
		+ ' grant create fulltext catalog to [db_developer];'
		+ ' grant create function to [db_developer];'
		+ ' grant create procedure to [db_developer];'
		+ ' grant create role to [db_developer];'
		+ ' grant create rule to [db_developer];'
		+ ' grant create schema to [db_developer];'
		+ ' grant create service to [db_developer];'
		+ ' grant create synonym to [db_developer];'
		+ ' grant create table to [db_developer];'
		+ ' grant create type to [db_developer];'
		+ ' grant create view to [db_developer];'
		+ ' grant create xml schema collection to [db_developer];'
		+ ' grant delete to [db_developer];'
		+ ' grant execute to [db_developer];'
		+ ' grant insert to [db_developer];'
		+ ' grant references to [db_developer];'
		+ ' grant select to [db_developer];'
		+ ' grant showplan to [db_developer];'
		+ ' grant update to [db_developer];'
		+ ' grant view database state to [db_developer];'
		+ ' grant view definition to [db_developer];';
if @DebugMode = 1
	print @SQL;
else
	exec sp_executesql @SQL;

print 'Create a user for the login';
set @SQL =	'use [' + @DBName + ']'
		+ ' if not exists (select 1 from sys.database_principals where type_desc = ''SQL_USER'' and name = ''' + @Login + ''')'
		+ ' create user [' + @Login + '] for login [' + @Login + '] with default_schema = [dbo];';
if @DebugMode = 1
	print @SQL;
else
	exec sp_executesql @SQL;

print 'Add user to role';
set @SQL =	'use [' + @DBName + ']'
		+ ' if not exists ('
		+ 'select 1'
		+ ' from sys.database_role_members rm'
		+ ' inner join sys.database_principals r on r.principal_id = rm.role_principal_id'
		+ ' inner join sys.database_principals m on m.principal_id = rm.member_principal_id'
		+ ' where r.type_desc = ''DATABASE_ROLE'''
		+ ' and r.name = ''db_developer'''
		+ ' and m.name = ''' + @Login + ''''
		+ ')'
		+ ' alter role [db_developer] add member [ITDev];';
if @DebugMode = 1
	print @SQL;
else
	exec sp_executesql @SQL;

go

The proc is re-runnable and only does the bits that need to be done. It doesn’t check each permission because you don’t get an error if it’s already granted.

Note that the actual permissions bit is just a list so you could easily comment out any you don’t want to grant. Indeed you really should look at all these permissions and see what they mean.

If you set @DebugMode = 1 it will just print out the SQL instead of executing it. This was very handy during development and you could use it if you change anything or if you want to run the changes past somebody for compliance checking.

You can call it for a specific database like this:

exec master.dbo.dba_AddDBDeveloperRole
	@DBName = 'DatabaseX'
	, @Login = 'LoginY';

Or you could apply it to all user databases like this:

print 'Grant rights to DevTeam'

declare @DBName sysname;

declare db_cursor cursor
local forward_only
for
select	name
from	sys.databases
where	name not in ('master', 'model', 'msdb', 'tempdb')
	and state_desc = 'ONLINE'
	and is_read_only = 0
order by name;

open db_cursor;

	fetch next from db_cursor
	into 	@DBName;

	while @@fetch_status = 0
	begin
		exec master.dbo.dba_AddDBDeveloperRole
			@DBName = @DBName
			, @Login = 'DevTeam';

		fetch next from db_cursor
		into 	@DBName;
	end

close db_cursor;
deallocate db_cursor;

Tweak the WHERE clause to exclude other databases.

You can download the code here.

Reporting and the Edge browser

I recently updated my dev machine to Windows 10. I like it a lot more than Windows 8.1. I was keen to see whether Reporting works in the new Edge browser and it turns out it works just fine. Apart from a strange inability to edit a data source in Report Manager.

When you update anything on a data source you have to re-enter the login’s password but in Edge the page keeps insisting the password is wrong and refuses to save the change.

So I recommend you edit data sources in Internet Explorer for now.

sp_help_revlogin revisited

I believe I’ve come up with an improvement to sp_help_revlogin.

Transferring a login from one server to another is one of those learning experiences we all go through. If you just do CREATE LOGIN on another server the login in the master database usually gets a different ID. When you restore the database on the other server, the ID of the user in the database doesn’t match the ID of the login in master and so the login can’t access the database. (And, as they say, ‘hilarity ensues’.)

A better way to move the login is to use a Microsoft stored procedure sp_help_revlogin (read more here: http://support.microsoft.com/kb/918992/) to generate a script that will re-create the login with the same ID. But I noticed it doesn’t include the login’s default language, which is something that can have a major impact on things like whether the account can re-use existing cached query plans. So I’ve added this element.

Instead of this:

CREATE LOGIN [loginX]
WITH	PASSWORD = 0x02<snip>9E HASHED
	, SID = 0x8E<snip>80
	, DEFAULT_DATABASE = [databaseY]
	, CHECK_POLICY = ON
	, CHECK_EXPIRATION = OFF

you now get this:

create login [loginX]
with	password = 0x02<snip>9E hashed
	, sid = 0x8E<snip>80
	, default_database = [databaseY]
	, default_language = [us_english]
	, check_policy = on
	, check_expiration = off

(While I was at it, I changed the output to lower case because that’s how I like to code.)

Here’s the new version of the procedure with the altered lines highlighted:

/*
Purpose:	To script out logins for copying to another server.
Source:		http://support.microsoft.com/kb/918992/
Modifications:	I added the default language.
*/

use master;
go

IF OBJECT_ID ('sp_hexadecimal') IS NOT NULL
  DROP PROCEDURE sp_hexadecimal
GO

CREATE PROCEDURE sp_hexadecimal
    @binvalue varbinary(256),
    @hexvalue varchar (514) OUTPUT
AS
DECLARE @charvalue varchar (514)
DECLARE @i int
DECLARE @length int
DECLARE @hexstring char(16)
SELECT @charvalue = '0x'
SELECT @i = 1
SELECT @length = DATALENGTH (@binvalue)
SELECT @hexstring = '0123456789ABCDEF'
WHILE (@i &amp;amp;lt;= @length)
BEGIN
  DECLARE @tempint int
  DECLARE @firstint int
  DECLARE @secondint int
  SELECT @tempint = CONVERT(int, SUBSTRING(@binvalue,@i,1))
  SELECT @firstint = FLOOR(@tempint/16)
  SELECT @secondint = @tempint - (@firstint*16)
  SELECT @charvalue = @charvalue +
    SUBSTRING(@hexstring, @firstint+1, 1) +
    SUBSTRING(@hexstring, @secondint+1, 1)
  SELECT @i = @i + 1
END

SELECT @hexvalue = @charvalue
GO

IF OBJECT_ID ('sp_help_revlogin') IS NOT NULL
  DROP PROCEDURE sp_help_revlogin
GO
CREATE PROCEDURE sp_help_revlogin @login_name sysname = NULL AS
DECLARE @name sysname
DECLARE @type varchar (1)
DECLARE @hasaccess int
DECLARE @denylogin int
DECLARE @is_disabled int
DECLARE @PWD_varbinary  varbinary (256)
DECLARE @PWD_string  varchar (514)
DECLARE @SID_varbinary varbinary (85)
DECLARE @SID_string varchar (514)
DECLARE @tmpstr  varchar (1024)
DECLARE @is_policy_checked varchar (3)
DECLARE @is_expiration_checked varchar (3)

DECLARE @defaultdb sysname
DECLARE @defaultlanguage sysname

IF (@login_name IS NULL)
  DECLARE login_curs CURSOR FOR

      SELECT p.sid, p.name, p.type, p.is_disabled, p.default_database_name, p.default_language_name, l.hasaccess, l.denylogin FROM
sys.server_principals p LEFT JOIN sys.syslogins l
      ON ( l.name = p.name ) WHERE p.type IN ( 'S', 'G', 'U' ) AND p.name &amp;amp;lt;&amp;amp;gt; 'sa'
ELSE
  DECLARE login_curs CURSOR FOR

      SELECT p.sid, p.name, p.type, p.is_disabled, p.default_database_name, p.default_language_name, l.hasaccess, l.denylogin FROM
sys.server_principals p LEFT JOIN sys.syslogins l
      ON ( l.name = p.name ) WHERE p.type IN ( 'S', 'G', 'U' ) AND p.name = @login_name
OPEN login_curs

FETCH NEXT FROM login_curs INTO @SID_varbinary, @name, @type, @is_disabled, @defaultdb, @defaultlanguage, @hasaccess, @denylogin
IF (@@fetch_status = -1)
BEGIN
  PRINT 'No login(s) found.'
  CLOSE login_curs
  DEALLOCATE login_curs
  RETURN -1
END
SET @tmpstr = '/* sp_help_revlogin script '
PRINT @tmpstr
SET @tmpstr = '** Generated ' + CONVERT (varchar, GETDATE()) + ' on ' + @@SERVERNAME + ' */'
PRINT @tmpstr
PRINT ''
WHILE (@@fetch_status &amp;amp;lt;&amp;amp;gt; -1)
BEGIN
  IF (@@fetch_status &amp;amp;lt;&amp;amp;gt; -2)
  BEGIN
    PRINT ''
    SET @tmpstr = '-- Login: ' + @name
    PRINT @tmpstr
    IF (@type IN ( 'G', 'U'))
    BEGIN -- NT authenticated account/group

      SET @tmpstr = 'create login ' + QUOTENAME( @name ) + ' from windows with default_database = [' + @defaultdb + '], default_language = [' + @defaultlanguage + ']'
    END
    ELSE BEGIN -- SQL Server authentication
        -- obtain password and sid
            SET @PWD_varbinary = CAST( LOGINPROPERTY( @name, 'PasswordHash' ) AS varbinary (256) )
        EXEC sp_hexadecimal @PWD_varbinary, @PWD_string OUT
        EXEC sp_hexadecimal @SID_varbinary,@SID_string OUT

        -- obtain password policy state
        SELECT @is_policy_checked = CASE is_policy_checked WHEN 1 THEN 'ON' WHEN 0 THEN 'OFF' ELSE NULL END FROM sys.sql_logins WHERE name = @name
        SELECT @is_expiration_checked = CASE is_expiration_checked WHEN 1 THEN 'ON' WHEN 0 THEN 'OFF' ELSE NULL END FROM sys.sql_logins WHERE name = @name

            SET @tmpstr = 'create login ' + QUOTENAME( @name ) + ' with password = ' + @PWD_string + ' hashed, sid = ' + @SID_string + ', default_database = [' + @defaultdb + '], default_language = [' + @defaultlanguage + ']'

        IF ( @is_policy_checked IS NOT NULL )
        BEGIN
          SET @tmpstr = @tmpstr + ', check_policy = ' + @is_policy_checked
        END
        IF ( @is_expiration_checked IS NOT NULL )
        BEGIN
          SET @tmpstr = @tmpstr + ', check_expiration = ' + @is_expiration_checked
        END
    END
    IF (@denylogin = 1)
    BEGIN -- login is denied access
      SET @tmpstr = @tmpstr + '; deny connect sql to ' + QUOTENAME( @name )
    END
    ELSE IF (@hasaccess = 0)
    BEGIN -- login exists but does not have access
      SET @tmpstr = @tmpstr + '; revoke connect sql to ' + QUOTENAME( @name )
    END
    IF (@is_disabled = 1)
    BEGIN -- login is disabled
      SET @tmpstr = @tmpstr + '; alter login ' + QUOTENAME( @name ) + ' disable'
    END
    PRINT @tmpstr
  END

  FETCH NEXT FROM login_curs INTO @SID_varbinary, @name, @type, @is_disabled, @defaultdb, @defaultlanguage, @hasaccess, @denylogin
   END
CLOSE login_curs
DEALLOCATE login_curs
RETURN 0
GO

You can download the code here.