精华内容
下载资源
问答
  • beeline 命令行客户端连接hive

    万次阅读 2017-10-16 08:11:19
    [hadoop@mina0 bin]$ ll 总用量 888 -rwxr-xr-x. 1 hadoop hadoop 1031 4月 30 2015 beeline drwxrwxr-x. 3 hadoop hadoop 4096 10月 15 23:41 ext ... 1 hadoop hadoop 7844 5月 8 2015 hive -rwxr-xr-x
    [hadoop@mina0 bin]$ ll
    总用量 888
    -rwxr-xr-x. 1 hadoop hadoop   1031 4月  30 2015 beeline
    drwxrwxr-x. 3 hadoop hadoop   4096 10月 15 23:41 ext
    -rwxr-xr-x. 1 hadoop hadoop   7844 5月   8 2015 hive
    -rwxr-xr-x. 1 hadoop hadoop   1900 4月  30 2015 hive-config.sh
    -rwxr-xr-x. 1 hadoop hadoop    885 4月  30 2015 hiveserver2
    -rwxr-xr-x. 1 hadoop hadoop    832 4月  30 2015 metatool
    -rw-r--r--. 1 hadoop hadoop 875336 10月 15 15:56 mysql-connector-java-5.1.28.jar
    -rwxr-xr-x. 1 hadoop hadoop    884 4月  30 2015 schematool
    [hadoop@mina0 bin]$ cd ..
    [hadoop@mina0 hive]$ cd bin
    [hadoop@mina0 bin]$ ./beeline
    Beeline version 1.2.1 by Apache Hive
    beeline> !connect jdbc:hive2://localhost:10000
    Connecting to jdbc:hive2://localhost:10000
    Enter username for jdbc:hive2://localhost:10000: hadoop
    Enter password for jdbc:hive2://localhost:10000: 
    Connected to: Apache Hive (version 1.2.1)
    Driver: Hive JDBC (version 1.2.1)
    Transaction isolation: TRANSACTION_REPEATABLE_READ
    0: jdbc:hive2://localhost:10000> show databases
    0: jdbc:hive2://localhost:10000> ;
    +----------------+--+
    | database_name  |
    +----------------+--+
    | default        |
    | shizhan03      |
    +----------------+--+
    2 rows selected (1.801 seconds)
    0: jdbc:hive2://localhost:10000> show databases;
    +----------------+--+
    | database_name  |
    +----------------+--+
    | default        |
    | shizhan03      |
    +----------------+--+
    2 rows selected (0.055 seconds)
    0: jdbc:hive2://localhost:10000> use shizhan03;
    No rows affected (0.083 seconds)
    0: jdbc:hive2://localhost:10000> show tables;
    +-----------+--+
    | tab_name  |
    +-----------+--+
    | t_sz01    |
    +-----------+--+
    1 row selected (0.062 seconds)
    0: jdbc:hive2://localhost:10000>


    展开全文
  • 环境下载安装包,解压到合适位置:Hive 本地模式的搭建一、配置相关的文件:二、安装并配置MySQL二、将Hbase复制到其他节点:三、启动Hbase:命令行操作一、创建表二、添加数据三、扫描表四、修改表五、删除六、列出...

    追风赶月莫停留,平芜尽处是春山。

    环境

    • VMware 12.0.1
    • JDK 1.8.0
    • Hadoop 2.8.5
    • Zookeeper 3.4.10
    • Intellij IDEA 2019.3.1

    下载安装包,解压到合适位置:

    1. 将压缩包 apache-hive-2.3.3-bin.tar.gz,通过上传软件上传至/opt/softwares目录;
      解压软件包并移动至/opt/modules/
      下载传送门,提取码:g1hu

    Hive 本地模式的搭建

    一、配置相关的文件:

    1. 配置环境变量:
      该文件位于/etc/profile
      在该文件后面追加如下内容:

      export HIVE_HOME=/opt/modules/apache-hive-2.3.3-bin
      export PATH=$PATH:$HIVE_HOME/bin
      

      打开终端运行命令:source /etc/profile
      运行命令hive --version后若能输出当前Hive版本信息,则说明Hive环境变量配置成功。

    2. 关联Hadoop:
      复制Hive目录/opt/modules/apache-hive-2.3.3-bin/conf下的hive-env.sh.template文件为hive-env.sh,然后修改hive-env.sh添加一下内容,来指定Hadoop的安装目录。

      export HADOOP_HOME=/opt/modules/hadoop-2.8.5	# Hadoop安装目录
      
    3. 创建数据仓库目录:
      首先要启动Hadoop集群,然后依次执行以下命令来创建两个目录,并设置同组用户的权限。

      hadoop fs -mkdir /tmp 
      hadoop fs -chmod a+w /tmp
      hadoop fs -mkdir /user/hive/warehouse
      hadoop fs -chmod a+w /user/hive/warehouse
      

      如果提示/tmp文件夹存在,那么先执行hadoop fs -rm -r /tmp命令将其删除。然后再进行创建

    4. 初始化元数据信息:
      进入Hive安装目录/opt/modules/apache-hive-2.3.3-bin
      执行命令schematoll -dbType derby -initSchema
      执行完后Hive在Hive安装目录下创建了一个叫metastore_db的Derby数据库

    5. 启动 Hive CLI:
      进入metastore_db数据库所在的目录(即Hive安装目录)中执行hive命令

    二、安装并配置MySQL

    1. 执行wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm命令下载MySQL 5.6安装包
    2. 执行rpm -ivh mysql-community-release-el7-5.noarch.rpm命令进行解压
    3. 执行yum install mysql-server命令进行MySQL的安装。
    4. 执行grep temporary password /var/log/mysqld.log查看初始密码
    5. 执行mysql -u root并利用上一步得到的密码进行登录
    6. 执行set global validate_password_policy=0;set global validate_password_length=4;命令修改MySQL密码策略。
    7. 执行set password for root@localhost = password('123456');把密码改成123456(也可以自行改成其他的)。
    8. 执行exit;退出MySQL。
    9. 执行service mysqld restart命令,重启mysql服务。
    10. 执行mysql -u root -p命令,并利用刚才设置的密码登录mysql
    11. 执行create database hive_db;命令创建一个名叫hive_db的数据库。
    12. 执行create user hive IDENTIFIED by 'hive';命令创建一个名叫hive的用户,密码是hive。
    13. 执行grant all privileges on hive_db.* to hive@'centos01' identified by 'hive';命令为其赋予全局外部访问权限。
    14. 执行flush privileges;命令刷新内存
    15. 执行exit;命令退出mysql

    三、配置Hive

    1. 上传驱动包:
      上传Java连接MySQL的驱动包mysql-connector-java-5.1.44-bin.jar到Hive安装目录下的lib文件夹(/opt/modules/apache-hive-2.3.3-bin/lib)中。
      下载传送门,提取码:5vcd
    2. 修改配置文件:
      复制Hive目录/opt/modules/apache-hive-2.3.3-bin/conf下的hive-default.xml.template文件为hive-site.xml,然后把原始内容清空,最加入以下配置。
      <?xml version="1.0" encoding="UTF-8" standalone="no"?>
      <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
      <configuration>
              <!--MySQL数据库连接信息-->
              <property><!--连接 MYSQL的驱动类-->
                      <name>javax.jdo.option.ConnectionDriverName</name>
                      <value>com.mysql.jdbc.Driver</value>
              </property>
              <property><!--MySL连接地址,此处连接远程数据库,可根据实际情况进行修改-->
                      <name>javax.jdo.option.ConnectionURL</name>
                      <value>jdbc:mysql://centos01:3306/hive_db?createDatabaseIfNotExist=true</value>
              </property>
              <property><!-- MYSQL用户名-->
                      <name>javax.jdo.option.ConnectionUserName</name>
                      <value>hive</value>
              </property>
              <property><!-- MYSQL密码-->
                      <name>javax.jdo.option.ConnectionPassword</name>
                      <value>hive</value>
              </property>
      </configuration>
      
    3. 初始化元数据:
      执行schematool -dbType mysql -initSchema命令初始化元数据。
      若输出:
      Initialization script completed
      schemaTool completed
      
      则证明初始化完成
    4. 启动命令行:
      任意目录执行hive即可

    本地模式搭建完成。

    Hive 远程模式的搭建

    将centos01节点作为Hive的服务端,centos02和centos03节点作为Hive的客户端

    1. 安装Hive客户端:
      执行scp -r /opt/modules/apache-hive-2.3.3-bin/ root@centos02:/opt/modules/
      命令,将hive复制到centos02节点上。
      执行scp -r /opt/modules/apache-hive-2.3.3-bin/ root@centos03:/opt/modules/
      命令,将hive复制到centos03节点上。
    2. 修改配置文件:
      修改centos02节点上的Hive安装目录/opt/modules/apache-hive-2.3.3-bin/conf下的配置文件hive-site.xml,清除之前的配置属性,添加以下配置:
      <?xml version="1.0" encoding="UTF-8" standalone="no"?>
      <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
      <configuration>
              <property>
                      <name>hive.metastore.warehouse.dir</name>
                      <value>/user/hive/warehouse</value>
              </property>
              <property>
                      <name>hive.metastore.local</name>
                      <value>false</value>
              </property>
              <property>
                      <name>hive.metastore.uris</name>
                      <value>thrift://centos01:9083</value>
              </property>
      </configuration>
      
    3. 启动Metastore Server:
      在centos01节点上执行hive --service metastore &
      控制台部分启动日志信息如下:
      Starting Hive Metastore Server
      执行jps命令查看进程会有一个名为RunJar的进程
    4. 访问Hive
      在centos02节点上进入安装目录(/opt/modules/apache-hive-2.3.3-bin/)中执行bin/hive命令启动远程hive命令行。
    5. 测试Hive远程访问:
      在centos01节点上进入Hive命令行模式,执行create table student (id INT, name STRING);命令创建一个表
      在centos02和centos03节点上分别执行show tables;命令,若能查到student表则证明Hive远程模式搭建成功。

    Hive命令行操作

    使用命令hive进入hive命令行操作

    数据库操作

    一、创建数据库

    语法:create database 数据库名称
    例如:使用create database hive1命令创建一个叫hive1的数据库

    二、修改数据库

    语法:ALTER DATABASE 数据库名称 SET DBPROPERTIES (property_name=property_value,...);
    语法:ALTER DATABASE 数据库名称 SET OWNER user_or_role;

    三、选择数据库

    语法:use 数据库名称;
    例如:use hive1;

    四、显示数据库

    show databases;

    表操作

    一、创建表

    语法:

    CREATE [EXTERNAL] TABLE [IF NOT EXISTS] table_name
    	  [(col_name data_type [COMMENT col_comment], ...)]
    	  [COMMENT table_comment]
    	  [PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)]
    	  [CLUSTERED BY (col_name, col_name, ...)
    	    [SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS]
    	  [ROW FORMAT row_format]
    	  [STORED AS file_format]
    	  [LOCATION hdfs_path]
    

    二、查看表结构

    语法:desc 表名;
    例如:desc student;

    三、向表中插入数据

    语法:insert into 表名(property_name,property_name,...) values(property_value,property_value,...);
    例如:insert into student(id,name) values(001,张三);

    四、查询表中的数据

    语法:select * from 表名;
    例如:select * from student;查询student表中的所有内容
    例如:select name from student;查询student表中的name
    例如:select id from student;查询student表中的id

    五、本地文件导入Hive

    先创建一个可以从文件导入数据的表 语法:create table 表名(property_name property_type, property_name property_type,...) ROW FORMAT DELIMITED FIELDS TERMINA TED BY '分隔符';
    例如:create table student(id INT, name STRING) ROW FORMAT DELIMITED FIELDS TERMINA TED BY '\t';创建了一个student表,里面有id和name,并且导入的时候以制表符为分割符。

    六、删除表

    语法:drop table 表名;
    例如:drop table student;删除student表

    Hive JDBC 操作

    一、配置Maven依赖

    1. 新建一个叫做Hive_demo的Maven项目在pom.xml中引入以下java api依赖包

      <dependencies>
          <dependency>
              <groupId>org.apache.hive</groupId>
              <artifactId>hive-jdbc</artifactId>
              <version>2.3.3</version>
          </dependency>
      </dependencies>
      
    2. 打开设置File-->settings-->Maven-->ImportingImport Maven projects automatically 前面打勾。等待下载完成。

    二、创建Java Class

    import java.sql.*;
    
    public class Select {
        public static void main(String[] args) {
            String connectionURL = "jdbc:hive1://centos01:10000";
            String drivername = "org.apache.hive.jdbc.HiveDriver";
            String username = "hive";
            String password = "hive";
            try {
                Class.forName(drivername);
            }
            catch (ClassNotFoundException e) {
                e.printStackTrace();
                System.exit(1);
            }
            try {
                Connection con = DriverManager.getConnection(connectionURL, username, password);
                if(con != null) {
                    System.out.println("Connected");
                }
                else {
                    System.out.println("Not Connected");
                }
                Statement stmt = con.createStatement();
                String sql;
                ResultSet res;
                sql = "select * from u_data limit 10";
                System.out.println("Running: " + sql);
                res = stmt.executeQuery(sql);
                while (res.next()) {
                    System.out.println(String.valueOf(res.getString(1)) + "\t" + res.getString(2));
                }
            }
            catch(SQLException se) {
                se.printStackTrace();
            }
        }
    }
    
    

    若有不正确的地方还请批评指正。

    展开全文
  • 远程访问:通过JDBC连接数据仓库 支持并发访问 启动hiveserver2服务:查看该命令可知hiveserver2,等价于hive --service hiveserver2 &amp; [xiaoqiu@s150 /soft/hive/bin]$ cat hiveserver2 #...

    hive的访问:终端访问  远程访问

    终端访问:安装hive,利用shell脚本访问 不可并发访问

    远程访问:通过JDBC连接数据仓库 支持并发访问

    • 启动hiveserver2服务:查看该命令可知hiveserver2,等价于hive --service hiveserver2 &
    [xiaoqiu@s150 /soft/hive/bin]$ cat hiveserver2
    #!/usr/bin/env bash
    
    # Licensed to the Apache Software Foundation (ASF) under one or more
    # contributor license agreements.  See the NOTICE file distributed with
    # this work for additional information regarding copyright ownership.
    # The ASF licenses this file to You under the Apache License, Version 2.0
    # (the "License"); you may not use this file except in compliance with
    # the License.  You may obtain a copy of the License at
    #
    #     http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.
    
    bin=`dirname "$0"`
    bin=`cd "$bin"; pwd`
    
    . "$bin"/hive --service hiveserver2 "$@"
    

     

    [xiaoqiu@s150 /soft/hive/bin]$ hive --service hiveserver2 &
    [1] 11638
    [xiaoqiu@s150 /soft/hive/bin]$ jps
    11652 VersionInfo
    2325 DataNode
    10710 NameNode
    2665 ResourceManager
    11387 RunJar
    2525 JournalNode
    2797 NodeManager
    11677 Jps
    

     

    [xiaoqiu@s150 /soft/hive/bin]$ netstat -anop |grep 10000
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    tcp        0      0 0.0.0.0:10000           0.0.0.0:*               LISTEN      11638/java           off (0.00/0/0)
    [xiaoqiu@s150 /soft/hive/bin]$
    

    启动hiveserver2服务,查看jps,发现出现了runjar进程  查看端口10000是否启动 

    方法一:进入到beeline命令行 通过JDBC协议 连接到hive数据仓库

    beeline> !connect jdbc:hive2://localhost:10000/incubator
    Connecting to jdbc:hive2://localhost:10000/incubator
    Enter username for jdbc:hive2://localhost:10000/incubator:
    Enter password for jdbc:hive2://localhost:10000/incubator:
    Connected to: Apache Hive (version 2.3.3)
    Driver: Hive JDBC (version 2.3.3)
    Transaction isolation: TRANSACTION_REPEATABLE_READ
    0: jdbc:hive2://localhost:10000/incubator> show tables;
    +-----------+
    | tab_name  |
    +-----------+
    | t         |
    +-----------+
    1 row selected (6.595 seconds)
    0: jdbc:hive2://localhost:10000/incubator> select * from t;
    +-------+---------+--------+
    | t.id  | t.name  | t.age  |
    +-------+---------+--------+
    +-------+---------+--------+
    No rows selected (6.435 seconds)
    0: jdbc:hive2://localhost:10000/incubator>
    

     

    方法二:通过api连接hive数据仓库(前提是服务端已经启动hiveserver2服务)

    package com.hive;
    
    import java.sql.*;
    
    public class AcessApp {
        public static void main(String[] args) throws ClassNotFoundException, SQLException {
            Class.forName("org.apache.hive.jdbc.HiveDriver");
            Connection con = DriverManager.getConnection("jdbc:hive2://192.168.109.150:10000/cr","xiaoqiu","0806");
            Statement st = con.createStatement();
            ResultSet rs = st.executeQuery("SELECT * from student");
            while (rs.next()){
                System.out.println(rs.getInt(1) + "," + rs.getString(2));
            }
            rs.close();
            st.close();
            con.close();
        }
    }
    
    3,yamy
    4,sunny

     

     

     

    展开全文
  • [size=small] 基础环境: namenode 192.168.1.187 kafka3 ...这个集群是自己下的hadoop-*.tar.gz包逐个服务安装的,因此配置文件都需要手动修改,相对cloudera manager的要复杂一些。 hadoo...
    [size=small]
    基础环境:
    namenode 192.168.1.187 kafka3
    datanode 192.168.1.188 kafka4
    datanode 192.168.1.189 kafka1

    这个集群是自己下的hadoop-*.tar.gz包逐个服务安装的,因此配置文件都需要手动修改,相对cloudera manager的要复杂一些。

    hadoop 2.6.2
    hive 2.0.1 --只安装在了187上面

    1.启动hadoop
    ./start-all.sh

    2.配置hive
    # cat hive-site.xml
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
    <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
    </property>
    <property>
    <name>hive.querylog.location</name>
    <value>/hadoop/hive/log</value>
    <description>Location of Hive run time structured log file</description>
    </property>
    <property>
    <name>mapred.job.tracker</name>
    <value>http://192.168.1.187:9001</value>
    </property>
    <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
    </property>
    <property>
    <name>hive.server2.thrift.port</name>
    <value>10000</value>
    </property>
    <property>
    <name>hive.server2.thrift.bind.host</name>
    <value>192.168.1.187</value>
    </property>
    <property>
    <property>
    <name>hive.server2.enable.doAs</name>
    <value>true</value>
    </property>
    <name>hive.hwi.listen.port </name>
    <value>9999</value>
    <description>This is the port the Hive Web Interface will listen on </description>
    </property>
    <property>
    <name>datanucleus.autoCreateSchema </name>
    <value>false</value>
    </property>
    <property>
    <name>datanucleus.fixedDatastore </name>
    <value>true</value>
    </property>
    <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://192.168.1.189:3306/hive?createDatabaseIfNotExist=true</value>
    </property>
    <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
    </property>
    <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>root</value>
    </property>
    <property>
    <name>hbase.zookeeper.quorum</name>
    <value>kafka1,kafka4,kafka3</value>
    </property>
    </configuration>

    3.启动hiveserver服务
    # ./hiveserver2
    命令行模式:
    hive --service hiveserver2

    服务模式:
    ./hiveserver2

    4.测试连接:
    不用写jdbc程序,运行 bin/beeline.sh

    # ./beeline
    ls: 无法访问/opt/apache-hive-2.0.1-bin//lib/hive-jdbc-*-standalone.jar: 没有那个文件或目录
    Beeline version 2.0.1 by Apache Hive
    beeline> !connect jdbc:hive://192.168.1.187:10000 root root
    scan complete in 1ms
    scan complete in 7577ms
    No known driver to handle "jdbc:hive://192.168.1.187:10000" ---不用hive,改用hive2
    beeline>

    找到了这个包
    注意要讲hive的lib下的所有jar包都放到eclipse里面去

    # cp /opt/apache-hive-2.0.1-bin/jdbc/hive-jdbc-2.0.1-standalone.jar /opt/apache-hive-2.0.1-bin/lib/

    beeline> !connect jdbc:hive2://192.168.1.187:10000
    Connecting to jdbc:hive2://192.168.1.187:10000
    Enter username for jdbc:hive2://192.168.1.187:10000: root
    Enter password for jdbc:hive2://192.168.1.187:10000: root

    beeline> !connect jdbc:hive2://192.168.1.187:10000
    Connecting to jdbc:hive2://192.168.1.187:10000
    Enter username for jdbc:hive2://192.168.1.187:10000: root
    Enter password for jdbc:hive2://192.168.1.187:10000:
    Enter password for jdbc:hive2://192.168.1.187:10000: Error: Failed to open new session: java.lang.RuntimeException:
    org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
    User: root is not allowed to impersonate root (state=,code=0)

    重启hadoop后,还是不行,但是报错内容换了。
    在hadoop的core-site.xml中添加内容:
    <property>
    <name>hadoop.proxyuser.hadoop.hosts</name> --刚开始这里写错了一直不知道!不是hadoop用户,我是用的root用户
    <value>*</value>
    </property>
    <property>
    <name>hadoop.proxyuser.hadoop.groups</name>
    <value>root</value>
    </property>

    正确的是:
    在hadoop的core-site.xml中添加内容:
    <property>
    <name>hadoop.proxyuser.root.hosts</name>
    <value>*</value>
    </property>
    <property>
    <name>hadoop.proxyuser.root.groups</name>
    <value>root</value>
    </property>

    beeline> !connect jdbc:hive2://192.168.1.187:10000
    Connecting to jdbc:hive2://192.168.1.187:10000
    Enter username for jdbc:hive2://192.168.1.187:10000: root
    Enter password for jdbc:hive2://192.168.1.187:10000:
    16/06/02 11:22:00 [main]: INFO jdbc.HiveConnection: Transport Used for JDBC connection: null
    Error: Could not open client transport with JDBC Uri: jdbc:hive2://192.168.1.187:10000: java.net.ConnectException: 拒绝连接 (state=08S01,code=0)


    把$HIVE_HOME/lib下的所有hive开头的jar包都拷贝过去
    # ./beeline
    Beeline version 2.0.1 by Apache Hive --报错没有了
    beeline>

    开启hive的log
    cd /opt/apache-hive-2.0.1-bin/conf
    cp hive-log4j2.properties.template hive-log4j2.properties
    vi hive-log4j2.properties
    property.hive.log.dir = /hadoop/hive/log
    property.hive.log.file = hive.log

    # more hive.log

    2016-06-03T10:20:16,883 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:OperationManager is inited.
    2016-06-03T10:20:16,884 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:SessionManager is inited.
    2016-06-03T10:20:16,884 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:CLIService is inited.
    2016-06-03T10:20:16,884 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:ThriftBinaryCLIService is inited.
    2016-06-03T10:20:16,884 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:HiveServer2 is inited.
    2016-06-03T10:20:17,022 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:OperationManager is started.
    2016-06-03T10:20:17,022 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:SessionManager is started.
    2016-06-03T10:20:17,023 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:CLIService is started.
    2016-06-03T10:20:17,023 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:ThriftBinaryCLIService is started.
    2016-06-03T10:20:17,023 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:HiveServer2 is started.
    2016-06-03T10:20:17,038 INFO [main]: server.Server (Server.java:doStart(252)) - jetty-7.6.0.v20120127
    2016-06-03T10:20:17,064 INFO [main]: webapp.WebInfConfiguration (WebInfConfiguration.java:unpack(455)) - Extract jar:file:/opt/apache-hive-2.0.1-bin/lib/hive-jdbc-2.
    0.1-standalone.jar!/hive-webapps/hiveserver2/ to /tmp/jetty-0.0.0.0-10002-hiveserver2-_-any-/webapp
    2016-06-03T10:20:17,582 INFO [Thread-10]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(100)) - Starting ThriftBinaryCLIService on port 10000 with 5...500
    worker threads

    2016-06-03T10:20:17,023 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:ThriftBinaryCLIService is started.
    2016-06-03T10:20:17,023 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:HiveServer2 is started.
    2016-06-03T10:20:17,038 INFO [main]: server.Server (Server.java:doStart(252)) - jetty-7.6.0.v20120127
    2016-06-03T10:20:17,064 INFO [main]: webapp.WebInfConfiguration (WebInfConfiguration.java:unpack(455)) - Extract jar:file:/opt/apache-hive-2.0.1-bin/lib/hive-jdbc-2.0.1-standalone.jar!/hive-webapps/hiveserver2/ to /tmp/jetty-0.0.0.0-10002-hiveserver2-_-any-/webapp
    2016-06-03T10:20:17,582 INFO [Thread-10]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(100)) - Starting ThriftBinaryCLIService on port 10000 with 5...500 worker threads
    2016-06-03T10:20:17,781 INFO [main]: handler.ContextHandler (ContextHandler.java:startContext(737)) - started o.e.j.w.WebAppContext{/,file:/tmp/jetty-0.0.0.0-10002-hiveserver2-_-any-/webapp/},jar:file:/opt/apache-hive-2.0.1-bin/lib/hive-jdbc-2.0.1-standalone.jar!/hive-webapps/hiveserver2
    2016-06-03T10:20:17,827 INFO [main]: handler.ContextHandler (ContextHandler.java:startContext(737)) - started o.e.j.s.ServletContextHandler{/static,jar:file:/opt/apache-hive-2.0.1-bin/lib/hive-jdbc-2.0.1-standalone.jar!/hive-webapps/static}
    2016-06-03T10:20:17,827 INFO [main]: handler.ContextHandler (ContextHandler.java:startContext(737)) - started o.e.j.s.ServletContextHandler{/logs,file:/hadoop/hive/log/}
    2016-06-03T10:20:17,841 INFO [main]: server.AbstractConnector (AbstractConnector.java:doStart(333)) - Started SelectChannelConnector@0.0.0.0:10002
    2016-06-03T10:20:17,841 INFO [main]: server.HiveServer2 (HiveServer2.java:start(438)) - Web UI has started on port 10002


    网页可以打开,看到hiveserver2
    http://192.168.1.187:10002/hiveserver2.jsp


    1..通过日志,可以看到hiveserver2是正常开启的,但就是一直报错: User: root is not allowed to impersonate root

    设置hadoop的core-site.xml<configuration>
    <property>
    <name>hadoop.tmp.dir</name>
    <value>/hadoop/tmp</value>
    </property>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://192.168.1.187:9000</value>
    </property>
    <property>
    <name>dfs.name.dir</name>
    <value>/hadoop/name</value>
    </property>
    <property>
    <name>hadoop.proxyuser.root.hosts</name>
    <value>192.168.1.187</value>
    </property>
    <property>
    <name>hadoop.proxyuser.root.groups</name>
    <value>root</value>
    </property>
    <property>
    <name>fs.checkpoint.period</name>
    <value>3600</value>
    <description>The number of seconds between two periodic checkpoints.</description>
    </property>
    <property>
    <name>fs.checkpoint.size</name>
    <value>67108864</value>
    </property>
    <property>
    <name>fs.checkpoint.dir</name>
    <value>/hadoop/namesecondary</value>
    </property>
    </configuration>
    搞了很久才发现,在187上对hadoop的core-site.xml做的修改,没有传到另外两个节点


    2. 设置impersonation,这样hive server会以提交用户的身份去执行语句,如果设置为false,则会以起hive server daemon的admin user来执行语句
    [html]
    <property>
    <name>hive.server2.enable.doAs</name>
    <value>true</value>
    </property>

    3. JDBC方式
    hive server 1的driver classname是org.apache.hadoop.hive.jdbc.HiveDriver,Hive Server 2的是org.apache.hive.jdbc.HiveDriver,这两个容易混淆。

    # hiveserver2 --终于成功啦!!
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.1-bin/lib/hive-jdbc-2.0.1-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/opt/hadoop-2.6.2/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
    OK


    # cd /opt/apache-hive-2.0.1-bin/bin
    # ./beeline
    Beeline version 2.0.1 by Apache Hive
    beeline> !connect jdbc:hive2://192.168.1.187:10000
    Connecting to jdbc:hive2://192.168.1.187:10000
    Enter username for jdbc:hive2://192.168.1.187:10000: root
    Enter password for jdbc:hive2://192.168.1.187:10000:
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.1-bin/lib/hive-jdbc-2.0.1-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/opt/hadoop-2.6.2/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
    Connected to: Apache Hive (version 2.0.1)
    Driver: Hive JDBC (version 2.0.1)
    16/06/03 15:44:19 [main]: WARN jdbc.HiveConnection: Request to set autoCommit to false; Hive does not support autoCommit=false.
    Transaction isolation: TRANSACTION_REPEATABLE_READ
    0: jdbc:hive2://192.168.1.187:10000>
    0: jdbc:hive2://192.168.1.187:10000> show tables;
    INFO : Compiling command(queryId=root_20160603154642_dd611020-8d3f-4abe-9bd5-7f2fda519007): show tables
    INFO : Semantic Analysis Completed
    INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from deserializer)], properties:null)
    INFO : Completed compiling command(queryId=root_20160603154642_dd611020-8d3f-4abe-9bd5-7f2fda519007); Time taken: 0.291 seconds
    INFO : Concurrency mode is disabled, not creating a lock manager
    INFO : Executing command(queryId=root_20160603154642_dd611020-8d3f-4abe-9bd5-7f2fda519007): show tables
    INFO : Starting task [Stage-0:DDL] in serial mode
    INFO : Completed executing command(queryId=root_20160603154642_dd611020-8d3f-4abe-9bd5-7f2fda519007); Time taken: 0.199 seconds
    INFO : OK
    +---------------------------+--+
    | tab_name |
    +---------------------------+--+
    | c2 |
    | hbase_runningrecord_temp |
    | rc_file |
    | rc_file1 |
    | runningrecord_old |
    | sequence_file |
    | studentinfo |
    | t2 |
    | test_table |
    | test_table1 |
    | tina |
    +---------------------------+--+
    11 rows selected (1.194 seconds)
    0: jdbc:hive2://192.168.1.187:10000>


    创建项目:hivecon
    新建包:hivecon
    新建类:testhive
    package hivecon;
    import java.sql.Connection;
    import java.sql.DriverManager;
    import java.sql.ResultSet;
    import java.sql.Statement;

    public class testhive {
    public static void main(String[] args)throws Exception {
    // TODO Auto-generated method stub
    Class.forName("org.apache.hive.jdbc.HiveDriver");
    Connection conn=DriverManager.getConnection("jdbc:hive2://192.168.1.187:10000","root","");
    System.out.println("连接:"+conn);
    Statement stmt=conn.createStatement();
    //String tablename="";
    String query_sql="select systemno from runningrecord_old limit 1";
    ResultSet rs=stmt.executeQuery(query_sql);
    System.out.println("是否有数据:"+rs.next());
    }
    }
    可以直接执行:
    ERROR StatusLogger Unrecognized format specifier [msg]
    ERROR StatusLogger Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
    ERROR StatusLogger Unrecognized format specifier [n]
    ERROR StatusLogger Unrecognized conversion specifier [n] starting at position 56 in conversion pattern. --日志的报错暂时忽略
    连接:org.apache.hive.jdbc.HiveConnection@64485a47
    是否有数据:false


    ---再添加一些操作:
    package hivecon;
    import java.sql.Connection;
    import java.sql.DriverManager;
    import java.sql.ResultSet;
    import java.sql.Statement;

    public class testhive {
    private static String sql = "";
    private static ResultSet res;

    public static void main(String[] args)throws Exception {
    // TODO Auto-generated method stub
    Class.forName("org.apache.hive.jdbc.HiveDriver");
    Connection conn=DriverManager.getConnection("jdbc:hive2://192.168.1.187:10000","root","");
    System.out.println("连接:"+conn);
    Statement stmt=conn.createStatement();
    String query_sql="select systemno from runningrecord_old limit 1";
    ResultSet rs=stmt.executeQuery(query_sql);
    System.out.println("是否有数据:"+rs.next());

    //创建的表名
    String tableName = "tinatest";

    /** 第一步:存在就先删除 **/
    sql = "drop table " + tableName;
    stmt.execute(sql);

    /** 第二步:不存在就创建 **/
    sql = "create table " + tableName + " (key int, value string) row format delimited fields terminated by ','";
    stmt.execute(sql);

    // 执行“show tables”操作
    sql = "show tables '" + tableName + "'";
    System.out.println("Running:" + sql);
    res = stmt.executeQuery(sql);
    System.out.println("执行“show tables”运行结果:");
    if (res.next()) {
    System.out.println(res.getString(1));
    }

    // 执行“describe table”操作
    sql = "describe " + tableName;
    System.out.println("Running:" + sql);
    res = stmt.executeQuery(sql);
    System.out.println("执行“describe table”运行结果:");
    while (res.next()) {
    System.out.println(res.getString(1) + "\t" + res.getString(2));
    }
    // 执行“load data into table”操作
    String filepath = "/tmp/test2.txt";
    sql = "load data local inpath '" + filepath + "' into table " + tableName;
    System.out.println("Running:" + sql);
    stmt.executeUpdate(sql);
    // 执行“select * query”操作
    sql = "select * from " + tableName;
    System.out.println("Running:" + sql);
    res = stmt.executeQuery(sql);
    System.out.println("执行“select * query”运行结果:");
    while (res.next()) {
    System.out.println(res.getInt(1) + "\t" + res.getString(2));
    }
    conn.close();
    conn = null;
    }
    }

    --执行结果:
    连接:org.apache.hive.jdbc.HiveConnection@64485a47
    是否有数据:true
    Running:show tables 'tinatest'
    执行“show tables”运行结果:
    tinatest
    Running:describe tinatest
    执行“describe table”运行结果:
    key int
    value string
    Running:load data local inpath '/tmp/test2.txt' into table tinatest
    Running:select * from tinatest
    执行“select * query”运行结果:
    1 a
    2 b
    3 tina

    去hive里面验证:
    hive> show tables;
    OK
    c2
    hbase_runningrecord_temp
    rc_file
    rc_file1
    runningrecord_old
    sequence_file
    studentinfo
    t2
    test_table
    test_table1
    tina
    tinatest
    Time taken: 0.065 seconds, Fetched: 12 row(s)
    hive> select * from tinatest;
    OK
    1 a
    2 b
    3 tina
    Time taken: 3.065 seconds, Fetched: 3 row(s)

    [/size]
    展开全文
  • Linux通过beeline连接远程Hive

    千次阅读 2020-03-01 16:09:41
    Linux需要连接远程Hive,可以使用beeline。 一、确保服务器已经安装配置了java环境; 二、下载hadoop和hive相关的包 wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-2.7.3.tar.gz wget ...
  • Windows中使用cmd命令行连接远程Linux

    千次阅读 2020-12-02 09:47:56
    目前市场流行的远程连接工具众多,很多同学们在学习的时候一上来就学习的是使用工具连接 , 忘记了Windows自带的cmd也可以使用ssh/sftp来连接操作远程Linux,话不多说 , 来看一看怎么操作吧! 1,测试电脑是否已经内置了 ...
  • 通过远程jdbc方式连接hive数据仓库

    千次阅读 2017-07-16 23:28:50
    1.启动hiveserver2服务器,监听端口10000 $>hive --service hiveserver2 & ...2.通过beeline命令行连接到hiveserver2 $>beeline //进入beeline命令行(于hive --service beeline) $beeline>!help //查看帮助 $
  • hive以hadoop集群为基础,提供hdfs的SQL支持; hive一般可以以mysql为元数据存储,默认Derby;hadoop,hive安装自行百度吧; 介绍hive的远程访问:  未配置之前使用beeline的...1,在hive服务的安装节点的hive-...
  • hive以hadoop集群为基础,提供hdfs的SQL支持; hive一般可以以mysql为元数据存储,默认Derby;hadoop,hive安装自行百度吧; 介绍hive的远程访问:  未配置之前使用beeline...1,在hive服务的安装节点的hive-sit...
  • 如果能直接在spark sql命令行窗口中访问hive就会简单明了很多。四台虚拟机:bigdata1,bigdata2,bigdata3,bigdata4在bigdata2中配置hive-site.xml&lt;?xml version="1.0" enc...
  • JDBC连接Hive

    2017-11-08 17:25:38
    Hive 3种连接模式 : CLI :命令行界面(Command Line Interface for batch scripting),即在hive的bin目录下通过 ./hive命令启动的shell命令行界面WebUI : 通过 ./hive –service hwi 来启动,端口9999...
  • HIVE仓库扩展:连接hive数据库

    千次阅读 2018-03-08 14:55:46
    一、命令行客户端工具:HiveCLI CLI是和Hive交互的最简单/最常用方式,你只需要在一个具备完整Hive环境下的Shell终端中键入hive即可启动服务。BeelineBeeline是Hive新的命令行客户端工具。Hive客户端工具后续将使用...
  • 文章同步自 [ lxw的...本文简单介绍一下Hive命令行接口(Hive Command Line)及Hive交互Shell(Hive Interactive Shell)的一些使用。 七、使用Hive命令行 7.1 Hive Command Line 输入$HIVE_HOME/bin/hive –H 或者
  • DataGrip连接Hive详细步骤

    千次阅读 2021-01-05 14:11:01
    文章目录前言一、开启Hive服务二、下载依赖jar包三、配置步骤总结 前言 我们知道Hive有三种使用方式:CLI命令行,HWI(hie web interface)、Thrift客户端连接方式。由于前2中方式界面不友好,且笨拙单一,多...
  • Hive 连接

    2019-03-28 22:12:39
    通过远程jdbc方式连接hive数据仓库 1.启动hiveserver2服务器,监听端口10000 $>hive --service hiveserver2 2.通过beeline命令行连接到hiveserver2 $>beeline //进入...
  • beeline连接Hive

    2021-02-26 10:01:42
    HiveServer2提供了一个新的命令行工具Beeline,它基于SQLLine CLI的JDBC客户端。...Beeline使用JDBC连接到HiveServer2时,如果显示User: root is not allowed to impersonate hive,是因为远程访问Had
  • 前言 本文讲解如何通过数据库客户端界面工具DBeaver连接hive...业务人员没有权限通过命令行连接hive 领导喜欢在界面工具上查看hive里的数据 2、为什么使用DBeaver 其实在网上搜一下,连接hive的工具还有很多,使用...
  • Python连接Hive

    万次阅读 2018-01-17 14:48:40
      HiveServer是一个可选的服务,能够允许远程客户端使用各种编程语言向hive提交请求并检索结果。Hiveserver是建立在Apache Thrift上的,所以有时候称呼其为Thrift Server,尽管因为HiverServer2也是建立在Thrift之...
  • Squirrel 连接Hive

    千次阅读 2017-11-09 13:53:22
    看到见小伙伴们开发调试都在命令行上,经常需要window下编写hive sql,再拷贝到linux中进行执行,效率感觉比较低。于是,想寻找一个客户端工具来提供开发调试的效率。网上一问度娘,发现了Squirrel这个客户端工具,...
  • 连接 Hive 的四种方法

    千次阅读 2020-03-02 15:29:42
    $HIVE_HOME/bin/hive(连接命令) HiveServer2 and Beeline $HIVE_HOME/bin/hiveserver2(h2的启动命令) $HIVE_HOME/bin/beeline -u jdbc:hive2://$H2_HOST:$H2_PORT(连接命令) HCatalog $HIVE_HOME/bin/h...
  • python连接hive

    千次阅读 2014-01-27 18:16:39
    1 准备连接hive的python代码 在使用Python连接hive之前需要将hive中的文件拷贝到python的sys.path中 cp -r $HIVE_PATH/lib/py /usr/local/lib/python2.7/site-packages 或者将hive中连接代码,设法加入到...
  • 1、hive 命令行模式 直接输入./hive/bin/hive的执行程序,或者输入 ...2、jdbc远程连接hiveserver2 (1)、hiveserver2增加了权限控制,需要在hadoop的配置文件中配置增加权限。 在hadoop的core-site.xml中添加如...

空空如也

空空如也

1 2 3 4 5 ... 20
收藏数 3,593
精华内容 1,437
关键字:

命令行远程连接hive服务