win10环境运行spark报错（本机Hadoop版本hadoop2.7.1）

报错信息：

java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V 	at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native Method) 	at org.apache.hadoop.util.NativeCrc32.calculateChunkedSumsByteArray(NativeCrc32.java:86) 	at org.apache.hadoop.util.DataChecksum.calculateChunkedSums(DataChecksum.java:430) 	at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:202) 	at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:163) 	at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:144) 	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:405) 	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) 	at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) 	at org.apache.hadoop.mapred.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:108) 	at org.apache.spark.internal.io.SparkHadoopWriter.close(SparkHadoopWriter.scala:101) 	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$5.apply$mcV$sp(PairRDDFunctions.scala:1145) 	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1393) 	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1145) 	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1125) 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) 	at org.apache.spark.scheduler.Task.run(Task.scala:108) 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 	at java.lang.Thread.run(Thread.java:748) 20/05/20 14:43:55 WARN TaskSetManager: Lost task 0.0 in stage 2.0 (TID 2, localhost, executor driver): java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V 	at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native Method) 	at org.apache.hadoop.util.NativeCrc32.calculateChunkedSumsByteArray(NativeCrc32.java:86) 	at org.apache.hadoop.util.DataChecksum.calculateChunkedSums(DataChecksum.java:430) 	at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:202) 	at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:163) 	at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:144) 	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:405) 	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) 	at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) 	at org.apache.hadoop.mapred.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:108) 	at org.apache.spark.internal.io.SparkHadoopWriter.close(SparkHadoopWriter.scala:101) 	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$5.apply$mcV$sp(PairRDDFunctions.scala:1145) 	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1393) 	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1145) 	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1125) 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) 	at org.apache.spark.scheduler.Task.run(Task.scala:108) 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 	at java.lang.Thread.run(Thread.java:748)
查阅资料发现问题为缺少hadoop.dll文件

解决方案 下载hadoopd对应版本hadoop.dll放入 C:\Windows\System32下即可

（网上另一种解决方案为放在C:\Windows\System32与HADOOP_HOME/bin下或是因为下载的spark与hadoop版本不相对应）

异常信息为：

ERROR Shell: Failed to locate the winutils binary in the hadoop binary

Java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

解决方法：

1、到官网下载需要的hadoop版本的gz的压缩包，这里以hadoop2.6.1为例子，



2、解压之后，配置环境变量：

添加HADOOP_HOME，



配置CLASSPATH和PATH：





验证配置是否成功，打开cmd，输入hadoop version：



3、下载winutils.exe

到网上搜一下，就可以下载，然后将它放到hadoop解压后的bin目录下：



4、重启IDEA（这一步很重要）

        完成上面4步，就可以解决问题。虽然放着不管，也可以正常运行SparkStreaming程序，但要追求极致。

 

 

一、spark-shell.cmd 启动出错 NoClassDefFoundError

在load-spark-env.cmd 文件中增加以下设置环境变量内容，hadoop后面先在系统环境变量里设置，或者直接使用路径

@echo off

rem ###################  SET ENV ##################

rem  set SPARK_MASTER_IP=localhost

rem  set SPARK_WORKER_CORES=1

set SPARK_WORKER_MEMORY=1g

echo HADOOP_HOME: %HADOOP_HOME%

set HADOOP_CONF_DIR=”%HADOOP_HOME%\hadoop\etc\hadoop”

echo HADOOP_CONF_DIR: %HADOOP_CONF_DIR%

for /f %%i in (‘hadoop classpath’) do set SPARK_DIST_CLASSPATH=%%i

echo SPARK_DIST_CLASSPATH: %SPARK_DIST_CLASSPATH%

mkdir -p %SPARK_HOME%\temp

SET temp=%SPARK_HOME%\temp

echo %temp%

 

 

二、 Could not initialize class scala.tools.fusesource_embedded.jansi.internal.Kernel32

把scala的jline-2.11.jar 放到spark 的jars目录下

1.下载 winutils.exe：http://download.csdn.net/download/ypsong2011/9109143或者https://github.com/srccodes/hadoop-common-2.2.0-bin
2.将此文件放置在某个目录下，比如C:\winutils\bin\中。
3.在程序的一开始声明：System.setProperty("hadoop.home.dir",
 "c:\\winutil\\")


参考：https://social.msdn.microsoft.com/forums/azure/en-US/28a57efb-082b-424b-8d9e-731b1fe135de/please-read-if-experiencing-job-failures?forum=hdinsight