azure blob
We introduced Azure Databricks, how it is created and some of its important components in my previous article here. We will look at how we can work with Azure Blob Storage in Azure Databricks in this article.
我们推出了Azure的Databricks，它是如何创建和我以前的文章一些重要的组成部分在这里 。 在本文中，我们将研究如何在Azure Databricks中使用Azure Blob存储。
Azure Blob Storage is a storage service in Azure that enables users to store large amounts of unstructured data like videos, audios, images, text, backup data, etc. This cloud service is a cost-effective and scalable approach when compared to on-premises storage options. There are four types of storage in Azure, including Blob (Binary large object) storage; you can learn about them here: Different Azure Storage types (File, Blob, Queue and Table).
Azure Blob存储是Azure中的一项存储服务，使用户能够存储大量非结构化数据，例如视频，音频，图像，文本，备份数据等。与本地部署相比，此云服务是一种经济高效且可扩展的方法存储选项。 Azure中有四种存储类型，包括Blob（二进制大对象）存储； 您可以在此处了解它们： 不同的Azure存储类型（文件，Blob，队列和表） 。
Azure Databricks is an implementation of Apache Spark on Microsoft Azure. It is a powerful chamber that handles big data workloads effortlessly and helps in both data wrangling and exploration. It lets you run large-scale Spark jobs from any Python, R, SQL, and Scala applications. Spark is written in Scala (a high-level language) and there are definitely some performance benefits if commands are run in Scala in Azure Databricks.
Azure Databricks是Microsoft Azure上Apache Spark的实现。 它是一个强大的工具箱，可轻松处理大数据工作负载，并有助于数据整理和探索。 它使您可以从任何Python，R，SQL和Scala应用程序运行大规模Spark作业。 Spark用Scala（一种高级语言）编写，如果在Azure Databricks中的Scala中运行命令，则肯定会带来一些性能上的好处。
先决条件 (Pre-requisites)
To follow along with this article, you need to have the following requisites:
要继续阅读本文，您需要满足以下条件：
- here
在这里为自己创建一个
- this site, to know how to create a Databricks service on Azure
此站点，以了解如何在Azure上创建Databricks服务
- here if you are new to the Azure Storage service. Afterward, we will require a .csv file on this Blob Storage that we will access from Azure Databricks
此处 。 之后，我们将在此Blob存储上需要一个.csv文件，我们将从Azure Databricks访问该文件
Once the storage account is created using the Azure portal, we will quickly upload a block blob (.csv) in it. Note: Azure Blob Storage supports three types of blobs: block, page and append. We can only mount block blob to DBFS (Databricks File System), so for this reason, we will work on a block blob.
使用Azure门户创建存储帐户后，我们将在其中快速上传一个块Blob（.csv）。 注意：Azure Blob存储支持三种类型的Blob：块，页面和附加。 我们只能将块Blob挂载到DBFS（Databricks文件系统），因此，出于这个原因，我们将在块Blob上工作。
Navigate to your Storage Account in the Azure portal, and click on Containers
在Azure门户中导航到您的存储帐户，然后单击“ 容器”
Click on + Container to create a new container
单击+容器创建一个新容器
Type a name for the container, I am selecting the default access level as Private and finally hit the Create button
输入容器的名称，我选择默认访问级别为“私有”，最后单击“ 创建”按钮
The container ‘azsqlshackcontainer’ is successfully created in the storage account
在存储帐户中成功创建了容器“ azsqlshackcontainer”
Click the Upload button to upload a blob
点击上传按钮上传一个Blob
This will open the Upload blade, browse your file and click on the Upload button. You can use any .csv file, in case you want to refer to the one we are using in this demonstration, you can download it from here
这将打开“上载”刀片，浏览您的文件，然后单击“上载”按钮。 您可以使用任何.csv文件，如果您要引用本演示中使用的文件，可以从此处下载。
The block blob “1000 Sales Records.csv” is successfully uploaded in the container
块blob“ 1000 Sales Records.csv”已成功上载到容器中
Our final Azure Storage account structure looks like this: Azure Storage Account (azsqlshackstorage) >> Container (azsqlshackcontainer) >> Block Blob (.csv file).
我们最终的Azure存储帐户结构如下所示：Azure存储帐户（azsqlshackstorage）>>容器（azsqlshackcontainer）>>块Blob（.csv文件）。
Let’s pick up from where we left off in our last article. For the azdatabricks cluster, created in this article, create a notebook named accessingblobstorage and select Scala language, as shown below. We will be working on this databricks notebook to access Azure Blob Storage in this article.
让我们从上一篇文章中停下来的地方开始。 对于本文中创建的azdatabricks集群，创建一个名为accessingblobstorage的笔记本，然后选择Scala语言，如下所示。 我们将在此databricks笔记本上进行工作以访问本文中的Azure Blob存储。
如何在Azure Databricks中创建Scala笔记本？ (How to create a Scala notebook in Azure Databricks?)
On the Databricks portal, click on the Workspace in the left vertical menu tab and select Create >> Notebook. In the Create Notebook dialog, give a name for your Notebook, choose Scala as the language from the Language drop-down and all the running clusters will be displayed in the Cluster drop-down. Since we just have one, so it will automatically be populated.
在Databricks门户上，单击左侧垂直菜单选项卡中的工作区，然后选择创建>>笔记本。 在“ 创建笔记本”对话框中，为笔记本命名，从“ 语言”下拉列表中选择Scala作为语言，所有正在运行的群集将显示在“群集”下拉列表中。 由于我们只有一个，因此它将自动填充。
The Scala notebook shown below is created successfully in the databricks portal.
如下所示的Scala笔记本已在databricks门户中成功创建。
大纲摘要 (Outline summary)
Since our base set-up comprising of Azure Blob Storage (with a .csv file) and Azure Databricks Service (with a Scala notebook) is in place, let’s talk about the structure of this article. We will demonstrate the following in this article:
由于我们的基本设置由Azure Blob存储（带有.csv文件）和Azure Databricks服务（带有Scala笔记本）组成，因此，我们来讨论一下本文的结构。 我们将在本文中演示以下内容：
-
We will first mount the Blob Storage in Azure Databricks using the Apache Spark Scala API. In simple words, we will read a CSV file from Blob Storage in the Databricks
我们将首先使用Apache Spark Scala API将Blob存储安装在Azure Databricks中。 简单来说，我们将从Databricks中的Blob存储中读取CSV文件
-
We will do some quick transformation to the data and will move this processed data to a temporary SQL view in Azure Databricks. We will also see how we can use multiple languages in the same databricks notebook
我们将对数据进行一些快速转换，并将处理后的数据移至Azure Databricks中的临时SQL视图。 我们还将看到如何在同一个databricks笔记本中使用多种语言
-
Finally, we will write the transformed data back to the Azure blob storage container using the Scala API
最后，我们将使用Scala API将转换后的数据写回到Azure blob存储容器中
步骤1：挂载Azure Blob存储容器 (Step 1: Mount an Azure Blob Storage container)
To get started, you will need to know the name of your container, storage account and sas (Shared access signature). This sas is granted by Microsoft to access Azure Storage resources. To generate a sas, navigate to your Storage Account, and click Shared access signature under the Settings blade. Check the appropriate values; you can generate the sas for a particular time frame as well. Finally, click on Generate SAS and connection string button to generate and copy the SAS token.
首先，您需要知道容器的名称，存储帐户和sas（共享访问签名）。 Microsoft已授予此sas访问Azure存储资源的权限。 要生成sas，请导航到您的存储帐户，然后在“设置”边栏选项卡下单击“共享访问签名”。 检查适当的值； 您也可以针对特定时间范围生成SAS。 最后，单击“ 生成SAS和连接字符串”按钮以生成并复制SAS令牌。
Type the below code in the Scala notebook and press “Ctrl + Enter” to execute it.
在Scala笔记本中键入以下代码，然后按“ Ctrl + Enter”执行它。
val containerName = "azsqlshackcontainer"
val storageAccountName = "azsqlshackstorage"
val sas = "?sv=2019-02-02&ss=b&srt=sco&sp=rwdlac&se=2020-03-30T07:53:28Z&st=2020-03-29T23:53:28Z&spr=https&sig=5vK%2FKEgTJVLoF4SX08IvwK7Tff2x42TNwtb%2B9eWzSFI%3D"
val config = "fs.azure.sas." + containerName+ "." + storageAccountName + ".blob.core.windows.net"
Don’t forget to change the values for your containerName, storageAccountName and sas accordingly.
不要忘记相应地更改containerName，storageAccountName和sas的值。
Once executed, click on the + sign (marked above in a red circle) to continue with your coding.
Execute the following code snippet that uses Databricks utility to mount the Azure Storage.
执行后，单击+号（上面用红色圆圈标记）继续进行编码。 执行以下使用Databricks实用程序安装Azure存储的代码段。
dbutils.fs.mount(
source = "wasbs://azsqlshackcontainer@azsqlshackstorage.blob.core.windows.net/1000 Sales Records.csv",
mountPoint = "/mnt/myfile",
extraConfigs = Map(config -> sas))
A few points to note here:
这里要注意几点：
-
Databricks Utilities (dbutils) offers utilities with FileSystems. This mounting sets up the connection between Azure Databricks and Azure Blob Storage
Databricks实用程序（dbutils）为FileSystems提供实用程序。 此安装建立了Azure Databricks和Azure Blob存储之间的连接
-
myfile(<mount-name>) is a DBFS path and represents what container/folder will be mounted in DBFS as specified in “source”. You can type in your <mount-name>
myfile（<mount-name>）是一个DBFS路径，代表将在“源”中指定的DBFS中装载哪些容器/文件夹。 您可以输入<mount-name>
-
config is the <conf-key> which is this “fs.azure.sas.<container-name>.<storage-account-name>.blob.core.windows.net” in our case
config是<conf-key>，在我们的例子中是“ fs.azure.sas。<容器名称>。<存储帐户名称> .blob.core.windows.net”
步骤2：读取资料 (Step 2: Read the data)
Run the following command to read the .csv file in your blob storage container.
运行以下命令以读取Blob存储容器中的.csv文件。
-
We will use a spark.read command to read the file and store it in a dataframe, mydf
我们将使用spark.read命令读取文件并将其存储在数据框mydf中
-
With header= true option, we are telling it to use the first line of the file as a header
使用header = true选项，我们告诉它使用文件的第一行作为标题
-
The default option for inferSchema is false, so by setting it to true, Spark will infer the schema of each column automatically
inferSchema的默认选项为false，因此将其设置为true，Spark将自动推断每个列的模式
-
/mnt/myfile will allow you to read from the mount, you will access files in your container as if they were local files
/ mnt / myfile将允许您从安装读取，您将访问容器中的文件，就像它们是本地文件一样
-
And finally, use the display function to display the records of the data frame
最后，使用显示功能显示数据框的记录
-
Do change the values in the command below for your mount name and file format, which is myfile and .csv, respectively, in our case
请在下面的命令中更改安装名称和文件格式的值，在本例中分别为myfile和.csv
val mydf = spark.read
.option("header","true")
.option("inferSchema", "true")
.csv("/mnt/myfile")
display(mydf)
步骤3：转换资料 (Step 3: Transform the data)
To select a few columns in the result set, execute the below command.
要在结果集中选择几列，请执行以下命令。
val selectspecificcolsdf = mydf.select("Region", "Country", "SalesChannel", "OrderPriority","TotalProfit")
display(selectspecificcolsdf)
Let’s execute the code below to rename the column from SalesChannel to SalesPlatform.
让我们执行以下代码，将列从SalesChannel重命名为SalesPlatform。
val renamedColsmyDF = selectspecificcolsdf.withColumnRenamed("SalesChannel", "SalesPlatform")
display(renamedColsmyDF)
Databricks notebooks offer a great collaborative environment for its users. Click on the comment icon above (circled in red), add your comment, and click on Comment.
Databricks笔记本为其用户提供了一个很好的协作环境。 点击上方的评论图标（红色圆圈），添加您的评论，然后点击评论 。
Create a view or a table to query this data. You can provide any temporary view name. This is treated as a local table and will be available only to this Notebook.
创建一个视图或表来查询此数据。 您可以提供任何临时视图名称。 该表被视为本地表，仅对此笔记本可用。
renamedColsmyDF.createOrReplaceTempView(“SalesData”)
Now comes the interesting part of notebooks, you can override the default language, Scala, by specifying language magic command, %<language>. Therefore, using %sql on the top, you can switch the language from Scala to Spark SQL. The below code retrieves all results from the temporary view, SalesData.
现在出现了笔记本的有趣部分，您可以通过指定语言魔术命令％<language>来覆盖默认语言Scala。 因此，在顶部使用％sql，可以将语言从Scala切换到Spark SQL。 以下代码从临时视图SalesData中检索所有结果。
%sql
SELECT * FROM SalesData
Let’s perform a simple aggregation to calculate the total profit for each region for both Online and Offline channels.
让我们执行一个简单的汇总，以计算在线和离线渠道每个区域的总利润。
%sql
SELECT Region, SalesPlatform,SUM(TotalProfit) From SalesData
group by SalesPlatform, Region
order by region
步骤4：将处理后的数据从Azure Databricks笔记本写入Blob存储容器 (Step 4: Write processed data from an Azure Databricks notebook to the Blob Storage container)
Let’s encapsulate this SELECT statement in Spark SQL to save the aggregated results in a data frame.
让我们将此SELECT语句封装在Spark SQL中，以将汇总结果保存在数据框中。
val aggdata = spark.sql("""
SELECT Region, SalesPlatform,SUM(TotalProfit) From SalesData
group by SalesPlatform, Region
order by region
""")
Below is the code snippet to write transformed and aggregated .csv data to an Azure Blob Storage container using Scala API. If, for some reason, you have to restart/pause your cluster, then make sure to execute the command set, cmd 1, mentioned at the very top of this article.
下面是使用Scala API将经过转换和聚合的.csv数据写入Azure Blob存储容器的代码段。 如果由于某种原因必须重新启动/暂停群集，请确保执行本文顶部提到的命令集cmd 1。
The below code will save the results stored in aggdata into “SalesProfitData” folder on the Blob Storage.
下面的代码会将存储在aggdata中的结果保存到Blob存储上的“ SalesProfitData”文件夹中。
aggdata.write
.option("header", "true")
.format("com.databricks.spark.csv")
.save("/mnt/result/SalesProfitData.csv")
Go to your Azure Blob Storage and you will find the SalesProfitData folder created. There is a metadata file also created.
转到您的Azure Blob存储，您将找到创建的SalesProfitData文件夹。 还创建了一个元数据文件。
Click on this newly created folder. You will see the following files:
单击此新创建的文件夹。 您将看到以下文件：
-
_started_… -> It indicates when the process was started
_started_…->它指示进程何时开始
-
_committed_… -> It depicts how many files were created
_committed_…->描述了创建了多少个文件
-
_SUCCESS -> It tells the status of the processing
_SUCCESS->告知处理状态
Also, you will find several split .csv files. Spark engine splits data into partitions for effective and fast data processing. Click on one of the files, and you can edit it, as shown below.
此外，您还会找到几个拆分的.csv文件。 Spark引擎将数据分为多个分区，以进行有效，快速的数据处理。 单击其中一个文件，然后可以对其进行编辑，如下所示。
The below screenshot shows a sample of the same file downloaded and opened as a .csv file.
下面的屏幕快照显示了下载并打开为.csv文件的相同文件的示例。
Note: Azure Databricks with Apache Spark’s fast cluster computing framework is built to work with extremely large datasets and guarantees boosted performance, however, for a demo, we have used a .csv with just 1000 records in it. In practical scenarios, Azure Databricks processes petabytes of data in a few seconds.
注意 ：带有Apache Spark快速群集计算框架的Azure Databricks旨在与超大型数据集一起使用并保证提高的性能，但是，对于演示而言，我们使用的.csv中仅包含1000条记录。 在实际情况下，Azure Databricks会在几秒钟内处理PB级数据。
结论 (Conclusion )
In this post, we learned to access Blob Storage from Azure Databricks and also demonstrated how to write wrangled data back in an Azure Blob Storage container using Azure Databricks. Along the way, we also did some hands-on coding in the Scala notebook in the Spark cluster.
在这篇文章中，我们学习了如何从Azure Databricks访问Blob存储，还演示了如何使用Azure Databricks将经过整理的数据写回到Azure Blob Storage容器中。 在此过程中，我们还在Spark集群的Scala笔记本中进行了一些动手编码。
翻译自: https://www.sqlshack.com/accessing-azure-blob-storage-from-azure-databricks/
azure blob
