Python : Reading CSV Files from Azure File Share Using Python

Introduction:

In today's data-driven world, handling and analyzing data is a crucial task for businesses and developers alike. Azure File Share provides a scalable and reliable storage solution in the cloud. Python, being a popular programming language for data processing and analysis, can be leveraged to read CSV files from Azure File Share effortlessly. In this article, we will explore step-by-step instructions to read CSV files stored in Azure File Share using Python. We had requirement in our science related project where we need to process some part of our data in Python web app. As all scientist were comfortable with python we need to process the file in python.

Prerequisites:

1. An Azure account with an active subscription.
2. An Azure File Share with CSV files uploaded.
3. Python installed on your local machine or Docker.

Step 1: Install Azure Storage File Share SDK

To interact with Azure File Share in Python, we need to install the Azure Storage File Share SDK. Open a terminal or command prompt and run the following command:

pip install azure-storage-file-share

Step 2: Import Required Libraries

Create a Python script and start by importing the necessary libraries:

from azure.storage.fileshare import ShareClient, ShareFileClient
import pandas as pd
import numpy as np

Step 3: Set Up Azure File Share Connection

Retrieve the connection string for your Azure Storage account from the Azure Portal. Replace 'your_connection_string' with your actual connection string:

connection_string = 'your_connection_string'
file_share_name = 'your_file_share_name'
file_path = '00-TestFiles/mytestfile.csv'

Step 4: Read CSV File from Azure File Share

Next, we will define a function to read the content of CSV file from the Azure File Share:
The below example read the file content into variable file_content. You need to write code to deal with it.

def read_csv_from_azure_file_share(file_path):
    share_client = ShareClient.from_connection_string(connection_string, file_share_name)
    with share_client.get_directory_client('your_directory_name') as directory_client:
        with directory_client.get_file_client(file_path) as file_client:
            file_content = file_client.download_file().readall().decode('utf-8')
    return file_content

Step 5: Process CSV Data

Now that we have the CSV file content in a string, we can use the Pandas library to process and analyze the data:

def process_csv_data(csv_content):
    df = pd.read_csv(pd.compat.StringIO(csv_content))
    # Your data processing and analysis code here
    return df

Sometimes you need particular column to be imported in numpy array. You can use below code to do so.

def read_csv_from_azure_file_share(file_path):
    share_client = ShareFileClient.from_connection_string(connection_string, file_share_name, file_path)
    DEST_FILE = "downloadedfiles/downloadedfile.csv"
    with open(DEST_FILE , "wb") as file_handle:
        stream = file_client.download_file()
        file_handle.write(stream.readall())
        array = np.loadtxt(DEST_FILE, delimiter=",", usecols=(0), skiprows=0)    
    return array

Step 6: Putting It All Together

Call the functions to read and process the CSV data:

if __name__ == '__main__':
    file_path = 'your_file_name.csv'
    csv_content = read_csv_from_azure_file_share(file_path)
    data_frame = process_csv_data(csv_content)
    print(data_frame.head())  # Display the first few rows of the DataFrame

Conclusion:

In this article, we learned how to read CSV files stored in Azure File Share using Python. By leveraging the Azure Storage File Share SDK and the Pandas library, developers can easily access and process data from Azure File Share, making it a powerful combination for data analysis tasks. Whether you are working with small or large datasets, Python and Azure File Share provide a scalable and efficient solution for handling your data in the cloud.

Azure Series : File Share - Connecting and Reading CSV Files from Azure File Share in ASP.NET Core

 In today’s data-driven world, handling CSV files is a common task in many applications. Azure File Share provides a scalable and efficient way to store and manage these files in the cloud. In this article, we will explore how to connect to an Azure File Share and read CSV files from it using ASP.NET  Core and C#.

Prerequisites

Before we begin, ensure that you have the following prerequisites in place:

1. An Azure account with an active subscription.
2. A storage account and an Azure File Share created.

Setting Up Azure File Share

1. Log in to the Azure portal (https://portal.azure.com ) and create a storage account if you haven’t already.
2. Inside your storage account, create a new file share.
3. Note down the storage account name, the file share name, and the storage account key. You will need these details to connect to the Azure File Share from your ASP.NET  Core application.

Creating an ASP.NET  Core Application

Now, let’s create an ASP.NET  Core application to read CSV files from Azure File Share.

1. Open Visual Studio or your preferred code editor.
2. Create a new ASP.NET  Core project.
3. Install the required NuGet packages:
4. Microsoft.Azure.Storage.File for Azure Storage File Share access.
5. CsvHelper for CSV file parsing.
6. Configure your Azure File Share connection in the appsettings.json file:

{
  "AzureStorageSettings": {
    "StorageAccountName": "your-storage-account-name",
    "StorageAccountKey": "your-storage-account-key",
    "FileShareName": "your-file-share-name"
  }
}

6. Create a service to interact with Azure File Share. This service should use the Azure Storage SDK to connect to the file share.

public class AzureFileShareService
{
    private readonly IConfiguration _configuration;
    public AzureFileShareService(IConfiguration configuration)
    {
        _configuration = configuration;
    }
    public CloudFileDirectory GetFileShareDirectory()
    {
        string storageAccountName = _configuration["AzureStorageSettings:StorageAccountName"];
        string storageAccountKey = _configuration["AzureStorageSettings:StorageAccountKey"];
        string fileShareName = _configuration["AzureStorageSettings:FileShareName"];
        var storageAccount = CloudStorageAccount.Parse($"DefaultEndpointsProtocol=https;AccountName={storageAccountName};AccountKey={storageAccountKey};EndpointSuffix=core.windows.net");
        var fileClient = storageAccount.CreateCloudFileClient();
        var share = fileClient.GetShareReference(fileShareName);
        return share.GetRootDirectoryReference();
    }
    // Add methods to read CSV files here
}

Reading CSV Files from Azure File Share

Now that we have our Azure File Share service set up, let’s create a method to read CSV files:

public async Task<IEnumerable<MyDataModel>> ReadCsvFileAsync(string fileName)
{
    var directory = GetFileShareDirectory();
    var fileReference = directory.GetFileReference(fileName);
    if (await fileReference.ExistsAsync())
    {
        // Open the file stream
        var stream = await fileReference.OpenReadAsync();
        // Use CsvHelper to read the CSV data
        using (var reader = new StreamReader(stream))
        using (var csv = new CsvReader(reader, new CsvConfiguration(CultureInfo.InvariantCulture)))
        {
            var records = csv.GetRecords<MyDataModel>().ToList();
            return records;
        }
    }
    else
    {
        throw new FileNotFoundException("File not found in Azure File Share.");
    }
}

In this example, MyDataModel should be replaced with your specific CSV data model.

Azure Series : RabbitMQ - Setup and Use - Building a Message Queue in RabbitMQ and Publish Message

Introduction

In today's interconnected and distributed systems, message queues play a crucial role in ensuring reliable communication between various components. RabbitMQ, a popular open-source message broker, provides a robust and scalable solution for implementing message queues. In this article, we will explore how to create a message queue in RabbitMQ using ASP.NET  Core and C#.

Prerequisites

Before we dive into the implementation, make sure you have the following tools installed:

1. .NET Core SDK (version compatible with ASP.NET  Core)
2. RabbitMQ server (either locally or on a cloud-based service)

Step 1: Set up the RabbitMQ Server

First, ensure you have RabbitMQ installed and running. You can download and install it from the official website or use a cloud-based service like RabbitMQ Cloud or CloudAMQP. You can also install it locally on your machine or use Docker to run RabbitMQ. 

Step 2: Create an ASP.NET  Core Project

Open your terminal or command prompt and create a new ASP.NET  Core project using the following command:

bash
dotnet new webapi -n MessageQueueExample
cd MessageQueueExample

Step 3: Install RabbitMQ Client Library

Next, install the RabbitMQ.Client NuGet package, which allows our ASP.NET  Core application to communicate with the RabbitMQ server:

dotnet add package RabbitMQ.Client

Step 4: Implement the Message Queue

Now, let's create a MessageQueueService class to handle interactions with the RabbitMQ server. Inside the Services folder, create a new file named MessageQueueService.cs.

using RabbitMQ.Client;
using System;
using System.Text;

namespace MessageQueueExample.Services
{
    public class MessageQueueService : IDisposable
    {
        private readonly IConnection _connection;
        private readonly IModel _channel;
        private const string QueueName = "my_queue";

        public MessageQueueService()
        {
            var factory = new ConnectionFactory() { HostName = "localhost" }; // Replace "localhost" with your RabbitMQ server address if needed
            _connection = factory.CreateConnection();
            _channel = _connection.CreateModel();
            _channel.QueueDeclare(queue: QueueName, durable: false, exclusive: false, autoDelete: false, arguments: null);
        }

        public void SendMessage(string message)
        {
            var body = Encoding.UTF8.GetBytes(message);
            _channel.BasicPublish(exchange: "", routingKey: QueueName, basicProperties: null, body: body);
        }

        public void Dispose()
        {
            _channel.Close();
            _connection.Close();
        }
    }
}

Step 5: Configure Dependency Injection

In the Startup.cs file, configure the MessageQueueService to be injected into the application:

using MessageQueueExample.Services;
// ...

public void ConfigureServices(IServiceCollection services)
{
    // ...
    services.AddSingleton<MessageQueueService>();
    // ...
}

Step 6: Sending Messages

Now that our message queue service is ready, let's use it to send messages. In your desired controller, inject the MessageQueueService, and use it to send a message:

using Microsoft.AspNetCore.Mvc;
using MessageQueueExample.Services;
// ...

[ApiController]
[Route("api/[controller]")]
public class MessageController : ControllerBase
{
    private readonly MessageQueueService _messageQueueService;

    public MessageController(MessageQueueService messageQueueService)
    {
        _messageQueueService = messageQueueService;
    }

    [HttpPost]
    public IActionResult SendMessage([FromBody] string message)
    {
        _messageQueueService.SendMessage(message);
        return Ok("Message sent successfully!");
    }
}

Conclusion

Congratulations! You have successfully implemented a message queue in RabbitMQ using ASP.NET  Core and C#. This setup will allow your applications to communicate efficiently and reliably, enabling a more scalable and resilient architecture.

Remember to handle error scenarios and add appropriate exception handling in a real-world scenario. Additionally, consider implementing message processing on the consumer side for a complete message queue system. Happy coding!

Azure Series : RabbitMQ - Setup and Use - Install RabbitMQ

Install RabbitMQ

If you plan on to host RabbitMQ locally, you can either install RabbitMQ in your local workspace or run a RabbitMQ Docker container in your local workspace.

To install RabbitMQ in your local workspace:

1. Make sure you have already installed Chocolatey Package Manager
2. RunAsAdministrator - Windows Powershell
3. From the powershell prompt run the following command to install RabbitMQ:
   choco install rabbitmq
4. Follow the installation prompts
5. After the installation completes if you have no critical errors, RabbitMQ should now be installed in your local workspace.
6. To verify RabbitMQ is now running in your local workspace, open a browser window and go to the following URL:
   http://localhost:15672/ 
7. You should see a login screen for the RabbitMQ web interface. The default RabbitMQ username and password is as follows:

To install RabbitMQ in your docker:

1. Make sure you have installed Docker. 
2. RunAsAdministrator - Windows Powershell
3. First, let’s pull the RabbitMQ docker image. We’ll use the 3-management version, so we get the Management plugin pre-installed.
   
   docker pull rabbitmq:3.12-management

4. Now let’s stand it up. We’ll map port 15672 for the management web app and port 5672 for the message broker.
   
   docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.12-management

5. Assuming that ran successfully, you’ve got an instance of RabbitMQ running! Bounce over to http://localhost:15672 to check out the management web app.

You should see a login screen for the RabbitMQ web interface. The default RabbitMQ user name and password is as follows:

username: guest
password: guest

Here you can see an overview of your RabbitMQ instance and the message broker’s basic components: Connections, Channels, Exchanges, and Queues.

Azure Series : Installation - Chocolatey Package Manager

Install Chocolatey Package Manager

What is Chocolatey? 

Chocolatey Official Installation Instructions 

Summary of Installation Steps

1. RunAsAdministrator - Windows Powershell
2. From the Powershell prompt install chocolatey using the following command:
   Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
3. If no errors occurred during installation, you can verify Chocolatey is installed using the following command:
   choco -?