Showing posts with label SQL Server. Show all posts
Showing posts with label SQL Server. Show all posts

Tuesday, August 3, 2021

Tableau LOD Calculations Explained with Ranking / Window functions

I started my learning journey with Tableau few months ago. 

So far , it is pretty good. However , I found LOD ( level of detail) expressions are interestingly complex 

at the first encounter.  Reading thru few examples and watching few videos to get a better understanding of these concepts. 

Being I come from database background and SQL knowledge , I am hoping , I can provide my understanding of LOD thru ranking/ analytical 

functions. 

Please see my blog posts on ranking / analytical functions  here 

https://mfzahirdba.blogspot.com/2009/10/analytics-20.html

https://mfzahirdba.blogspot.com/2009/11/analytics-20-part-ii.html

https://mfzahirdba.blogspot.com/2018/10/windowing-clause-why-it-matters.html

https://mfzahirdba.blogspot.com/2013/08/oracles-analytical-functions-vs-sql.html

https://mfzahirdba.blogspot.com/2012/09/difference-between-percentilecont-and.html


There are three LOD keywords ( FIXED , INCLUDE , EXCLUDE) in Tableau. 

FIXED LOD calculations does not depend on the visuals ( aka viz) , where as visulation impacts 

how INCLUDE and EXCLUDE are calculated. 

In my example , I have created a table in SQL server and populated few records. 

To follow along the example , feel free to use the script below. 


Create table CourseEnrollment 

( StudentName varchar(50) , 

  Department varchar(50) , 

  Subject varchar(50) , 

  EnrollmentDate Date 

 ) ;

 GO 


Insert into CourseEnrollment values ( 'Zahir Mohideen' ,  'Maths' , 'Pre Calculus' , '1988-03-01') ;

Insert into CourseEnrollment values ( 'Zahir Mohideen' ,  'Maths' , 'Multivariate Calculus' , '2010-05-16') ;

Insert into CourseEnrollment values ( 'Zahir Mohideen' ,  'Statistics' , 'Statistics 101' , '2020-07-25') ;

Insert into CourseEnrollment values ( 'Abdul Hameed' ,  'Maths' , 'Pre Calculus' , '1991-03-01') ;

Insert into CourseEnrollment values ( 'Abdul Hameed' ,  'Physics' , 'Phy101' , '2020-07-25') ;


I have connected Tableau to my underlying table . 



Here , I want to know the very first enrollment of the student no matter what department / course he enrolled into . Also , this value does not change , when the viz is changed. 

In the "sheet" tab , I create a calculated field for the FIXED LOD  , by clicking on "Create Calculated Field " as below. 



Similarly , create LOD calculations for INCLUDE and EXCLUDE. 



Let us bring in these LOD calculations into the sheet . 


The value for First Enrollment Date - Fixed is the minimum enrollment date   per student , no matter what the dimensions are available in the viz.  FIXED by Student Name .

The value for First Enrollment Date - INCLUDE is the minimum enrollment date  per student and all other available dimension in the viz . In our case , Department . So , in essence , first enrollment date per student / department. 


The value for First Enrollment Date - EXCLUDE is the minimum enrollment date  per student and all other available dimension in the viz , except student name as specified in the LOD  . In our case , Department . So , in essence , first enrollment date per department. 

These results can be derived from SQL using analytical / window functions as 


When we remove the department  / Student , the FIXED LOD remains the same ; there is a change in Include and Exclude. 







Hope , this simple example provides the LOD in the eyes of SQL developer. 

I offer support for database maintenance , performance tuning , database development. Please contact for any database related activities. 






Monday, May 4, 2020

Accessing Python Libraries in SQL Server



I have been exploring to setting up a contextualized search on one of my datasets. For this experiment,  I was planning on using Machine Learning Services  ( MLS) in SQL Server 2019.
And , the library I was word2vec . This library is not part of the Standard libraries that came with SQL Server MLS .

Took sometime , to get this installed in my SQL Server environment. I was hoping , this blog post will be helpful , if anyone is trying to install python libraries in SQL Server .

First , we need to identify the system path , where Python is installed.


C:\>sqlcmd -W -S SERNJ\SQL2019DB
1>
2> EXEC sp_execute_external_script
3>   @language =N'Python',
4>   @script=N'import sys; print("\n".join(sys.path))'
5> go
STDOUT message(s) from external script:

C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019DEV\PYTHON_SERVICES\python37.zip
C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019DEV\PYTHON_SERVICES\DLLs
C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019DEV\PYTHON_SERVICES\lib
C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019DEV\PYTHON_SERVICES
C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019DEV\PYTHON_SERVICES\lib\site-packages
C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019DEV\PYTHON_SERVICES\lib\site-packages\win32
C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019DEV\PYTHON_SERVICES\lib\site-packages\win32\lib
C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019DEV\PYTHON_SERVICES\lib\site-packages\Pythonwin
C:\ProgramData\SQL2019DEV\Temp-PY\Appcontainer1\3E21F1D4-0032-473A-A2EB-D6763111F7F9\rxLibs
C:\ProgramData\SQL2019DEV\Temp-PY\Appcontainer1\7AD5E2EF-A002-4B2E-8ED1-4BFC71DBDB57


Then navigate to the python services scripts  ( in my case , C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019DEV\PYTHON_SERVICES\Scripts ) directory and install gensim using pip.


C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019DEV\PYTHON_SERVICES\Scripts>pip install gensim
pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Collecting gensim
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/gensim/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/gensim/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/gensim/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/gensim/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/gensim/
  Could not fetch URL https://pypi.org/simple/gensim/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/gensim/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
  Could not find a version that satisfies the requirement gensim (from versions: )
No matching distribution found for gensim
pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping

For the above command to work , we need to add these directories to the path. 

C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019DEV\PYTHON_SERVICES\Lib\site-packages\conda
C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019DEV\PYTHON_SERVICES\Library\bin
C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019DEV\PYTHON_SERVICES\Scripts

Now , try the pip again to install gensim package.

C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019DEV\PYTHON_SERVICES\Library>pip install gensim

Collecting gensim
    100% |████████████████████████████████| 24.2MB 1.3MB/s
Collecting Cython==0.29.14 (from gensim)
    100% |████████████████████████████████| 1.7MB 6.5MB/s
Collecting smart-open>=1.8.1 (from gensim)
    100% |████████████████████████████████| 112kB 3.3MB/s
Requirement already satisfied: scipy>=0.18.1 in c:\program files\microsoft sql server\mssql15.sql2019dev\python_services\lib\site-packages (from gensim) (1.1.0)
Requirement already satisfied: six>=1.5.0 in c:\program files\microsoft sql server\mssql15.sql2019dev\python_services\lib\site-packages (from gensim) (1.12.0)
Requirement already satisfied: numpy>=1.11.3 in c:\program files\microsoft sql server\mssql15.sql2019dev\python_services\lib\site-packages (from gensim) (1.15.4)
Requirement already satisfied: requests in c:\program files\microsoft sql server\mssql15.sql2019dev\python_services\lib\site-packages (from smart-open>=1.8.1->gensim) (2.21.0)
Collecting boto (from smart-open>=1.8.1->gensim)
    100% |████████████████████████████████| 1.4MB 6.5MB/s
Collecting boto3 (from smart-open>=1.8.1->gensim)
    100% |████████████████████████████████| 133kB 6.5MB/s
Requirement already satisfied: idna<2 .9="">=2.5 in c:\program files\microsoft sql server\mssql15.sql2019dev\python_services\lib\site-packages (from requests->smart-open>=1.8.1->gensim) (2.8)
Requirement already satisfied: urllib3<1 .25="">=1.21.1 in c:\program files\microsoft sql server\mssql15.sql2019dev\python_services\lib\site-packages (from requests->smart-open>=1.8.1->gensim) (1.24.1)
Requirement already satisfied: certifi>=2017.4.17 in c:\program files\microsoft sql server\mssql15.sql2019dev\python_services\lib\site-packages (from requests->smart-open>=1.8.1->gensim) (2019.3.9)
Requirement already satisfied: chardet<3 .1.0="">=3.0.2 in c:\program files\microsoft sql server\mssql15.sql2019dev\python_services\lib\site-packages (from requests->smart-open>=1.8.1->gensim) (3.0.4)
Collecting s3transfer<0 .4.0="">=0.3.0 (from boto3->smart-open>=1.8.1->gensim)
    100% |████████████████████████████████| 71kB 4.6MB/s
Collecting botocore<1 .17.0="">=1.16.1 (from boto3->smart-open>=1.8.1->gensim)
    100% |████████████████████████████████| 6.2MB 6.7MB/s
Collecting jmespath<1 .0.0="">=0.7.1 (from boto3->smart-open>=1.8.1->gensim)
Requirement already satisfied: docutils<0 .16="">=0.10 in c:\program files\microsoft sql server\mssql15.sql2019dev\python_services\lib\site-packages (from botocore<1 .17.0="">=1.16.1->boto3->smart-open>=1.8.1->gensim) (0.14)
Requirement already satisfied: python-dateutil<3 .0.0="">=2.1 in c:\program files\microsoft sql server\mssql15.sql2019dev\python_services\lib\site-packages (from botocore<1 .17.0="">=1.16.1->boto3->smart-open>=1.8.1->gensim) (2.8.0)
Building wheels for collected packages: smart-open
  Running setup.py bdist_wheel for smart-open ... done
  Stored in directory: C:\Users\zahir\AppData\Local\pip\Cache\wheels\27\65\38\8d7f5fe8d7afb4e4566587b2d1933cec185fba19257836c943
Successfully built smart-open
Installing collected packages: Cython, boto, jmespath, botocore, s3transfer, boto3, smart-open, gensim
  Found existing installation: Cython 0.29.2
    Uninstalling Cython-0.29.2:
      Successfully uninstalled Cython-0.29.2
Successfully installed Cython-0.29.14 boto-2.49.0 boto3-1.13.1 botocore-1.16.1 gensim-3.8.3 jmespath-0.9.5 s3transfer-0.3.3 smart-open-2.0.0


Now , we should be able to use gensim in SQL Server . 
More to come on this. 


Stay tuned. 

Thursday, May 12, 2016

Fixing "DBProviderFactories" section issue in SSDT / SSIS / SSMS







Recently , I installed SQL Server Data tools ( SSDT) after installing SQL Server 2014 . 

As mentioned in one of my post (http://mfzahirdba.blogspot.com/2016/02/sql-server-2014-surprises.html )  , SSDT is not bundled in the installer . You will need to download and install it.

After the install , I created a SSIS project , and created a data flow task. 
When , I am try to create a new connection manager , I was greeted (!) with this error. 






Here is how , I resolved the issue .

a)  Identify the correct machine.config in the .NET framework 
     In my case , it was  
   C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config\machine.config

b)  Remove one of the tags (DbProviderFactories in section  in "system.data" 
section

  
        

After the edit ,I was able to create the connection managers with no issue.


It looks like , when you install the ODBC / .NET drivers , it adds the additional tag into the .NET framework configuration file. 

Hope this is helpful .




Friday, February 26, 2016

SQL Server 2014 surprises - ( missing SSDT)




After  installing   SQL Server 2014 ( fresh install) , I found out the SQL Server Data tools ( as called in SQL Server 2012)  /  SQL Server Business Intelligence Studio ( Pre SQL Server 2012 versions) is missing .

After few "google searches" , found a way to install to SSDT for SQL 2014.  Downloaded the installer from  Microsoft Downloads site ( https://www.microsoft.com/en-us/download/confirmation.aspx?id=42313 ) .

In my first attempt , check failed on me complaining on the rule "Same architecture installation ".  In this attempt , choose "Add features to an existing instance of SQL Server 2014".







I was able to get it to working by choosing "Perform a new installation of SQL Server 2014"



Why can't be this be bundled in the SQL Server install  as usual? I hope Microsoft changes its mind in bundling these features in SQL Server 2016 and beyond. 




Monday, August 24, 2015

Impact of LEFT or RIGHT in SQL Server Partition Function's Definition

SQL Server ( since 2005 ) has introduced partitioning to effectively manage very large databases ( VLDBs) . 
Partitioning helps the DBA in 

a) maintaining the statistics at the partition level . 
b) easily load/unload  the data ( via sliding the partition ) 
c) optimal utilizing the storage for the datasets 
   (i.e., Current Year's Data can be housed in the faster storage ; archived data can be stored in cheaper / slower storage ) . 


Here are the steps to implement partitioning 

a) Define the partition function ( PF). 
   This step defines how the data should be partitioned . 
   The option of "RIGHT" or "LEFT" dictates how the upper boundary is treated by SQL Server. 
   
b) Define the partition schema ( PS) that uses the partition function created above
   This steps associates the physical storage ( aka filegroup) that corresponds to the partitions defined in the PF

c) Create a table that utilizes the PS  created above. 
  
In this blog post , I will show the differences between LEFT and RIGHT option definied in the PF. 
Rest of the partitioning 


Let us dive into a simple example . 

C:\>sqlcmd -W
1> use my2014db
2> go
Changed database context to 'my2014db'.

Step a) 
We will create a PF with LEFT range for three values ( 10 , 20 , 30) . 


1> Create Partition Function pf_seqno_l ( int) as range left for values ( 10 , 20 , 30 ) ;
2> go

This will create 
i) parition for any values less than or equal to 10 
ii) parition for any values greater than 10 and less than or equal to 20 
iii) parition for any values greater than 20 and less than or equal to 30 
iv) parition for any values greater than 30 

Step b) 
We will create a PS that will associate filegroup to the partitions . 
Here in this example , I have assigned ALL the paritions to go into PRIMARY filegroup; typically in production environments , it will be associated with multiple filegroups. 

1> Create Partition Scheme ps_seqno_l as partition pf_seqno_l ALL to ([PRIMARY]) ;
2> go
Partition scheme 'ps_seqno_l' has been created successfully. 'PRIMARY' is marked as the next used filegroup in partition scheme 'ps_seqno_l'.

Step c) 

We will create a table that will use the PS. 

1> Create table t_seqno_l
2> (   cid int identity ,
3>     seqno int )
4>      on ps_seqno_l(seqno) ;
5> go

Let us check , if the object does all the partitions . As we can see, it has 4 partitions . 

1> SELECT partition_number , rows
2> FROM sys.partitions
3> WHERE OBJECT_ID = OBJECT_ID('t_seqno_l');
4> go

partition_number rows
---------------- ----
1 0
2 0
3 0
4 0

Let us insert a record and look at the partitions to confirm the number of records in the partitions . As the first record's value is 2 ( that <= 10 ) , that record is placed in the first partition. 


1> insert into t_seqno_l(seqno) values ( 2) ;
2> go

(1 rows affected)


1> SELECT partition_number , rows
2> FROM sys.partitions
3> WHERE OBJECT_ID = OBJECT_ID('t_seqno_l');
4> go

partition_number rows
---------------- ----
1 1
2 0
3 0
4 0

(4 rows affected)

Second record is also placed in the first partition as the values is <= 10. 

1> insert into t_seqno_l(seqno) values ( 10) ;
2> go

(1 rows affected)
1> SELECT partition_number , rows
2> FROM sys.partitions
3> WHERE OBJECT_ID = OBJECT_ID('t_seqno_l');
4> go
partition_number rows
---------------- ----
1 2
2 0
3 0
4 0

(4 rows affected)


Let us insert two more records. 

1> insert into t_seqno_l(seqno) values ( 20) ;
2> insert into t_seqno_l(seqno) values ( 50) ;
3> go

(1 rows affected)

(1 rows affected)

1> SELECT partition_number , rows
2> FROM sys.partitions
3> WHERE OBJECT_ID = OBJECT_ID('t_seqno_l');
4> go
partition_number rows
---------------- ----
1 2
2 1
3 0
4 1

1> select * from t_seqno_l;
2> go
cid seqno
--- -----
3 2
4 10
5 20
6 50

(4 rows affected)

As you can see from the above , 
- Partition 1 has two records ( seqno with the value of 2 and 10) 
- Partition 2 has one record ( seqno with the value of 20) 
- Partition 3 has no record 
- Partition 4 has one record ( seqno with the value of 50 ; which is >= 30 ) 


Now , let us try PF with right option . 
Let us repeat the steps for creating PF , PS and the table. 

This will create 
i) partition for any values less than  10 
ii) partition for any values greater than or equal to  10 and less than  20 
iii) partition for any values greater than or equal to  20 and less than  30 
iv) partition for any values greater than or equal to  30 


1> Create Partition Function pf_seqno_r ( int) as range right for values ( 10 , 20 , 30 ) ;
2> go

1> Create Partition Scheme ps_seqno_r as partition pf_seqno_r ALL to ([PRIMARY]) ;
2> go
Partition scheme 'ps_seqno_r' has been created successfully. 'PRIMARY' is marked as the next used filegroup in partition scheme 'ps_seqno_r'.

1> Create table t_seqno_r
2> (   cid int identity ,
3>     seqno int )
4>      on ps_seqno_r(seqno) ;
5> go

1> SELECT partition_number , rows
2> FROM sys.partitions
3> WHERE OBJECT_ID = OBJECT_ID('t_seqno_r');
4> go
partition_number rows
---------------- ----
1 0
2 0
3 0
4 0

(4 rows affected)

Let us insert a record. Let us insert a record and look at the partitions to confirm the number of records in the partitions . As the first record's value is 2 ( that <= 10 ) , that record is placed in the first partition. 

1> insert into t_seqno_r(seqno) values ( 2) ;
2> go

(1 rows affected)
1> SELECT partition_number , rows
2> FROM sys.partitions
3> WHERE OBJECT_ID = OBJECT_ID('t_seqno_r');
4> go
partition_number rows
---------------- ----
1 1
2 0
3 0
4 0

(4 rows affected)

Let us second record . As you can see now , this record is placed in the second partition. Here the partition layout differ.  

1> insert into t_seqno_r(seqno) values ( 10) ;
2> go

(1 rows affected)
1> SELECT partition_number , rows
2> FROM sys.partitions
3> WHERE OBJECT_ID = OBJECT_ID('t_seqno_r');
4> go
partition_number rows
---------------- ----
1 1
2 1
3 0
4 0

(4 rows affected)

1> insert into t_seqno_r(seqno) values ( 20) ;
2> insert into t_seqno_r(seqno) values ( 50) ;
3> go

(1 rows affected)

(1 rows affected)
1> SELECT partition_number , rows
2> FROM sys.partitions
3> WHERE OBJECT_ID = OBJECT_ID('t_seqno_r');
4> go
partition_number rows
---------------- ----
1 1
2 1
3 1
4 1


In short , LEFT or RIGHT dictates how the upper boundary  is placed with in the partitioned table. 

In the next few posts , we will discuss other features of partitioning. 

Thanks for reading.