Request Usage on SharePoint 2013 (Custom Reports)

I already explained in my previous article difference between user activity(user actions) and audit log. SharePoint logs all user actions such as site visits, list/libraries visits, page visits and so on. This information stored in SharePoint logging database. SharePoint has some out of box reports to view these data. Sometimes we need custom reports based on the request usage logged by SharePoint. These reports can be designed directly against SharePoint Logging Database either by creating new tables/views or OOTB views.

Let’s understand how we can achieve custom reports using SharePoint Logging Database.

Location:

You can do monitoring efficiently by using the logging database in SharePoint 2013. The logging database collects information from every server in the farm. The default name of the database is SharePoint_Logging. The logging database provides the option to view and customize various monitoring information in one place.

How:

The logging database uses a separate partition table for the daily data for each category. E.g. SharePoint writes the 1st day of data in dbo.TimerJobUsage_Partition0 table and for 32nd day in dbo.TimerJobUsageUsage_Partition32 table. This means, each partition table stores one particular day’s logs.

The mapping between one partition table and the exact date depends on the logging database retention period and the starting date to write logs into the logging database. You can get the mapping by observing time information inside each table. For example, if the retention period is 5 days and today’s logs are written to partition table 1, tomorrow’s logs will be written into partition table 2 and so on. After 5 days, all logs were gets deleted and new logs are written to the partition tables from first.

Access:

SharePoint Logging database has default views which can be used to access these data which is scattered between these partition tables. One of the default view is RequestUsage. This view provide the combined user activity data from all partition tables.

Actual Implementation: 

     SharePoint Logging database has retention period of 3 days by default. This can be changed using power-shell command. changing these retention period may add more load on database since SharePoint Logs millions of rows per days. I am sure changing this to more than 30 days may add lot of rows and increase the size of logging database too.

       I would recommend the intermediate table which can be used to store this data for more than specified limit days. Follow below steps. (Logging database is the only SharePoint 2013 database for which you can customize reports by directly changing the database).

1. Create new table in SharePoint Logging Database. 
2. Create new custom view which can filter these data by previous day using the above view “RequestUsage”. Be sure to filter the records since “RequestUsage” will give you millions of rows since SharePoint logs each and every minute activity of user. E.g. if you visit one page in SharePoint then it logs each and every entity used on that page as separate rows.
3. Create the SSIS package or executable which will Import the custom view output into the new custom table.
4. Schedule the SSIS Package or executable daily.
5. Create reports based on the new custom table.

User Activity Report Vs Audit Log in SharePoint 2013

It’s very important that your intended users should be benefiting value out of SharePoint for not only storing but their usage too. Management decisions are based on metrics-driven and hence SharePoint does various analysis appropriate for SharePoint Administrator to measure, monitor and produce activity and audit reports.

User activity is defined as some action taken by User on web, site, or documents (SharePoint Items). SharePoint comes with lot of out-of-box Usage and Audit reports which worth considering. Lets first understand difference between Usage and Audit.

Usage Reports:

When term Usage comes it refers to the Usage of particular web application, site, or Document. Analytics Processing Component in SharePoint Server 2013 analyzes both the content itself, and also the way that users interact with it. The results from the analysis are added to the items in the search index so that search relevance improves automatically over time. Also, the results are used in reports that help search administrators see which manual steps they can take to improve the search system.

Report	Displays
Usage Summary	Average Requests per day, Top Pages, Top Users, Top Referring Hosts, Top Referring Pages, Top Destination Sites, Top Queries, Search Results Top Destination Pages
Requests	Requests per day, Average Requests by month
Users	Unique Users, User Activity (past 30 days)
Referrers	Top Referring Hosts, Top Referring Pages
Destination	Top Destination Sites
Home Page	Home Page Requests per Day, Top Referring Pages, Top Destination Pages
Search Queries (site collection only)	Top Queries (Previous 30 days, Previous 12 Months), Queries per Scope
Search Results (site collection only)	Top Destination Pages, Queries with Low Click Through, Queries with Zero Results, Queries with Zero Best Bets

This data is not stored forever in the SharePoint Database. Usage analytics is a set of analyses that receive information about user actions, or usage events, such as clicks or viewed items, on the SharePoint site. The usage events are stored temporarily on the web front end and are pushed to the Search Service Application every 15 minutes. Usage events are kept on disk for up to 14 days before they are deleted. This number of days you can control in Central Administration or using Powershell commands. Usage analytics combines this information with information about crawled content from the Search analyses, and processes the information. Information about recommendations and usage events is added to the search index. Statistics on the different usage events is added to the search index and sent to the Analytics reporting database.

Based on these statics the different reports are generated by SharePoint Out-of-box.

To access these reports, Visit your Web Application > Site Settings > Popularity and Search Reports. Of course you can customize these reports using Powershell.

Audit Log Reports:

SharePoint Audit logs allows you to capture whole range of audits on your SharePoint Farm based on your needs. SharePoint allows you to control which web application, Site collection, or even libraries to be monitored.

It might seem like overkill, but being able to track how individual end users make use of your SharePoint environment is very helpful. Audit logs is to set them to delete logs within a reasonable amount of time. This can be managed from Central Administration and it is default to 30 days. Fortunately, you can set SharePoint to delete these outdated logs when you need to. Audit logs can quickly duplicate, and expand so fast they fill up your SQL Server so beware to enable them on your Site. The audit logs stored in the Content Database only hence you need to be extra cautious when enabling the Audit Log.

Note: Each time you set up a new site, you’ll also need to configure the audit log for it – this won’t be applied automatically. SharePoint audit logs also let you set up audit flags that allow you to track specific changes.

·       Check in/check out

·       View

·       Delete/undo

·       Update

·       Schema and profile changes

·       Security change

·       Copy/move

·       Search

·       Workflow child delete

·       Other flags for a whole range of other events

To access these reports, Visit your Web Application > Site Settings > Audit Log Reports. 

To set Audit Log settings, Visit your Web Application > Site Settings > Site Collection Audit Log Settings.

In my opinion, when you just need to monitor the usage of the Site Usage reports are the best. Whereas if you want to do the details analysis of how your site being used by users in terms of uploading, viewing, deleting documents I would recommend to activate Audit log for your site.

I came across with one requirement where my client want to know all users who has visited their intranet SharePoint web application, site collections and web sites. They want to track this for at-least 2-3 years before they discard this analytics. The catch is SharePoint stores statics but do not store the actual user wise data for more than 30 days. My next article will explain on how to achieve this.