精华内容
下载资源
问答
  • pyhive.rar

    2020-03-10 10:56:38
    pyhive使用时所需要的的四个库,可以离线安装,针对不能联网的主机。
  • OK,本文来看看本地在PyCharm中使用pyhive连接hive数据库时踩过的坑吧,先看看一眼小编的测试代码~ 首先是封装了一个连接hive获取数据的类,因为是测试,所以写的简单了些,如下: ''' @File : sql.py @Author: ...

           OK,本文来看看本地在PyCharm中使用pyhive连接hive数据库时踩过的坑吧,先看看一眼小编的测试代码~

           首先是封装了一个连接hive获取数据的类,因为是测试,所以写的简单了些,如下:

    '''
    @File  : sql.py
    @Author: xzw
    @Date  : 2020/6/14
    @Desc  : 封装类
    '''
    
    from pyhive import hive
    import pandas as pd
    
    
    class TestHive(object):
        '''
        连接Hive获取数据
        '''
    
        def __init__(self, host, port, username, database):
            '''
            初始化方法
            :param host: 地址
            :param port: 端口号
            :param username: 用户名
            :param database: 数据库名称
            '''
            self.conn = hive.Connection(host=host, port=port, username=username, database=database)
    
        def get_data(self, sql):
            '''
            获取数据
            :param sql: SQL语句
            :return: 返回得到的数据
            '''
            data = pd.read_sql(sql, self.conn)
            return data
    

           然后是测试类:

    '''
    @File  : manager.py
    @Author: xzw
    @Date  : 2020/6/14
    @Desc  : 访问Hive获取数据
    '''
    
    from connect_hive.sql import TestHive
    from fastapi import FastAPI
    import uvicorn
    
    app = FastAPI()
    
    
    @app.get("/get_hive_data")
    def get_data():
        '''
        获取hive中的数据
        :return:
        '''
        host, port, username, database = 'cdh-master', 10000, 'hive', 'test'
        instance = TestHive(host, port, username, database)
        result = instance.get_data("select * from xzw")
        return {"result": result}
    
    
    if __name__ == '__main__':
        uvicorn.run(app="manager:app", host="127.0.0.1", port=9999, reload=True, debug=True)
    

           上面就是整个过程的简单代码,想象着很快就能拿到hive中的数据了,然后……坑来了~

           第一个坑:

    thrift.transport.TTransport.TTransportException: Could not start SASL: b'Error in sasl_client_start (-4) SASL(-4): no mechanism available: Unable to find a callback: 2'

           解决办法是在Connection中添加auth="NOSASL":

    self.conn = hive.Connection(host=host, port=port, username=username, database=database, auth="NOSASL")

           第二个坑:

    Connection Issue: thrift.transport.TTransport.TTransportException: TSocket read 0 bytes

           解决办法是在hive-site.xml中增加下面属性:

    <property>
    	<name>hive.server2.authentication</name>
    	<value>NOSASL</value>
    </property>

           当然,小编使用的是CDH搭建的集群,直接在监控界面配置即可,如下所示:

    展开全文
  • pyhive离线安装.rar

    2020-09-10 20:03:37
    离线环境下pyhive的安装依赖。 python版本:3.8 操作系统:linux 包含:对应版本信息的所有依赖项及一个readme说明文件。
  • pyhive的安装

    千次阅读 2019-09-15 13:39:41
    这里写自定义目录标题连接hive数据库linux下安装pyhive 连接hive数据库 连接hive数据库,许多文章推荐pyhive包,经过小仙女本人多次尝试与周围童鞋交流,发现window系统貌似无法使用pyhive,mmp!!! 所以window...

    连接hive数据库

    连接hive数据库,许多文章推荐pyhive包,经过小仙女本人多次尝试与周围童鞋交流,发现window系统貌似无法使用pyhive,mmp!!!

    所以window系统下,如何连接hive数据库?

    1. 果断换包,使用impyla,安装途中遇到各种坑,可以参考下面绝世好文章:
      https://blog.csdn.net/Xiblade/article/details/82318294
      有童鞋推荐使用happybase,没有用过,但感觉是一个可以让人快乐的python包
    2. 在电脑硬件还可以情况下,可以选择装一个linux虚拟机
      我选择VMware和Ubantu,方法自行百度,坑很少

    linux下安装pyhive

    安装上了linux系统,再安装Anoconda,完毕之后打开命令行,依次输入:

        pip install sasl
        pip install thrift
        pip install thrift-sasl
        pip install pyhive
    

    然后你就发现安装sasl和thrift-sasl又是各种坑,欸,连接一个hive数据库简直太难了!报错如下:

    Collecting sasl
      Downloading https://files.pythonhosted.org/packages/8e/2c/45dae93d666aea8492678499e0999269b4e55f1829b1e4de5b8204706ad9/sasl-0.2.1.tar.gz
    Requirement already satisfied: six in ./anaconda3/lib/python3.7/site-packages (from sasl) (1.12.0)
    Building wheels for collected packages: sasl
      Building wheel for sasl (setup.py) ... error
      ERROR: Complete output from command /home/liuju/anaconda3/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-nc9goial/sasl/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-8ytz4zr3 --python-tag cp37:
      ERROR: running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.7
      creating build/lib.linux-x86_64-3.7/sasl
      copying sasl/__init__.py -> build/lib.linux-x86_64-3.7/sasl
      running egg_info
      writing sasl.egg-info/PKG-INFO
      writing dependency_links to sasl.egg-info/dependency_links.txt
      writing requirements to sasl.egg-info/requires.txt
      writing top-level names to sasl.egg-info/top_level.txt
      reading manifest file 'sasl.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      writing manifest file 'sasl.egg-info/SOURCES.txt'
      copying sasl/saslwrapper.cpp -> build/lib.linux-x86_64-3.7/sasl
      copying sasl/saslwrapper.h -> build/lib.linux-x86_64-3.7/sasl
      copying sasl/saslwrapper.pyx -> build/lib.linux-x86_64-3.7/sasl
      running build_ext
      building 'sasl.saslwrapper' extension
      creating build/temp.linux-x86_64-3.7
      creating build/temp.linux-x86_64-3.7/sasl
      gcc -pthread -B /home/liuju/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Isasl -I/home/liuju/anaconda3/include/python3.7m -c sasl/saslwrapper.cpp -o build/temp.linux-x86_64-3.7/sasl/saslwrapper.o
      cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
      In file included from sasl/saslwrapper.cpp:254:
      sasl/saslwrapper.h:22:10: fatal error: sasl/sasl.h: No such file or directory
       #include <sasl/sasl.h>
                ^~~~~~~~~~~~~
      compilation terminated.
      error: command 'gcc' failed with exit status 1
      ----------------------------------------
      ERROR: Failed building wheel for sasl
      Running setup.py clean for sasl
    Failed to build sasl
    Installing collected packages: sasl
      Running setup.py install for sasl ... error
        ERROR: Complete output from command /home/liuju/anaconda3/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-nc9goial/sasl/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-o68u9xve/install-record.txt --single-version-externally-managed --compile:
        ERROR: running install
        running build
        running build_py
        creating build
        creating build/lib.linux-x86_64-3.7
        creating build/lib.linux-x86_64-3.7/sasl
        copying sasl/__init__.py -> build/lib.linux-x86_64-3.7/sasl
        running egg_info
        writing sasl.egg-info/PKG-INFO
        writing dependency_links to sasl.egg-info/dependency_links.txt
        writing requirements to sasl.egg-info/requires.txt
        writing top-level names to sasl.egg-info/top_level.txt
        reading manifest file 'sasl.egg-info/SOURCES.txt'
        reading manifest template 'MANIFEST.in'
        writing manifest file 'sasl.egg-info/SOURCES.txt'
        copying sasl/saslwrapper.cpp -> build/lib.linux-x86_64-3.7/sasl
        copying sasl/saslwrapper.h -> build/lib.linux-x86_64-3.7/sasl
        copying sasl/saslwrapper.pyx -> build/lib.linux-x86_64-3.7/sasl
        running build_ext
        building 'sasl.saslwrapper' extension
        creating build/temp.linux-x86_64-3.7
        creating build/temp.linux-x86_64-3.7/sasl
        gcc -pthread -B /home/liuju/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Isasl -I/home/liuju/anaconda3/include/python3.7m -c sasl/saslwrapper.cpp -o build/temp.linux-x86_64-3.7/sasl/saslwrapper.o
        cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
        In file included from sasl/saslwrapper.cpp:254:
        sasl/saslwrapper.h:22:10: fatal error: sasl/sasl.h: No such file or directory
         #include <sasl/sasl.h>
                  ^~~~~~~~~~~~~
        compilation terminated.
        error: command 'gcc' failed with exit status 1
        ----------------------------------------
    ERROR: Command "/home/liuju/anaconda3/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-nc9goial/sasl/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-o68u9xve/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-nc9goial/sasl/
    
    

    其实sasl包的官网上有说明:https://pypi.org/project/sasl/0.1.3/,大概意思就是sasl包依赖gcc
    在这里插入图片描述
    在安装pyhive相关包之前安装gcc,需要在命令行里输入

    sudo apt-get install python-dev libsasl2-dev gcc
    

    在这里插入图片描述
    在这里插入图片描述
    然后就ojbk了!

    展开全文
  • 安装pyhive

    2021-05-19 18:03:57
    1、安装pyhive包:sasl、thrift、thrift-sasl、PyHive Pycharm的Terminal中运行: pip install thrift -i https://pypi.tuna.tsinghua.edu.cn/simple pip install thrift-sasl -i ...

    一、环境:Windows + Pycharm+ Python37
    1、安装pyhive包:sasl、thrift、thrift-sasl、PyHive
    Pycharm的Terminal中运行:

    pip install thrift -i https://pypi.tuna.tsinghua.edu.cn/simple
    pip install thrift-sasl -i https://pypi.tuna.tsinghua.edu.cn/simple
    pip install PyHive -i https://pypi.tuna.tsinghua.edu.cn/simple
    

    其中sasl在Windows安装下安装需要编译环境,为了方便可直接从Link下载编译好的包,直接进行安装,比如python37 64bit电脑选择sasl-0.2.1-cp37-cp37m-win_amd64.whl:

    pip install windos路径/sasl-0.2.1-cp37-cp37m-win_amd64.whl
    

    2、安装完毕,执行代码会报以下错误

    thrift.transport.TTransport.TTransportException: Could not start SASL: b'Error in sasl_client_start (-4) SASL(-4): no mechanism available: Unable to find a callback: 2'
    

    解决方案:
    在Windows中使用管理员权限打开控制台,在控制执行一段命令即可:

    REG ADD "HKEY_LOCAL_MACHINE\SOFTWARE\Carnegie Mellon\Project Cyrus\SASL Library" /v SearchPath /t REG_SZ /d "项目路径\项目名\venv3.7\Lib\site-packages\sasl\sasl2"
    

    3、就可以使用pyhive来通过python连接hive了。

    from pyhive import hive
    
    conn = hive.Connection(host='xxx', port=xxx, username='xxx',password='xxx', database='xxx', auth='LDAP')
    cursor = conn.cursor()
    # cursor.execute('show tables')
    cursor.execute('select count(1) from xx.xxx where xxx="xxx"')
    for result in cursor.fetchall():
        print(result)
    

    二、环境:Linux + Python36

    1、切换成 xxx 用户 
    su xxx
    2、查看python36路径
    which python36
    3、切换成root用户
    exit 
    4、安装pyhive
    /xxx/python36 -m pip install pyhive
    

    参考链接:[解决方案] Windows pyhive Could not start SASL

    展开全文
  • CentOs安装pyhive

    千次阅读 2019-09-11 16:50:05
    yum install gcc-c++ python-devel.x86_64 cyrus-sasl-devel.x86_64 pip install sasl ...pip install PyHive 以下为代码demo,功能是打印出所有表的建表语句。代码中连接的hive使用的LDAP验证方式,所...

    yum install gcc-c++ python-devel.x86_64 cyrus-sasl-devel.x86_64

     

    pip install sasl
    pip install thrift
    pip install thrift-sasl

    pip install PyHive

     

    以下为代码demo,功能是打印出所有表的建表语句。代码中连接的hive使用的LDAP验证方式,所以设置了用户名和密码。

    import os
    import sys  
    reload(sys)  
    sys.setdefaultencoding('utf8')  
    from pyhive import hive
    
    username = your user name
    password = your pwd
    
    #database = 'test'
    #dir_name = '/home/lwx/data/'+'dw'+'.sql'
    
    cnn=hive.Connection(host='your ip add', port='10000',auth='LDAP',username = username,password = password )
    cursor=cnn.cursor()
    dbs = ['test1','test2','test3']
    
    for db in dbs:
        cursor.execute("show tables in %s"%(db))
        for table in cursor.fetchall():
            cursor.execute("show create table %s.%s"%(db,table[0]))
            for i in cursor.fetchall():
    	    print str(i[0])

    如果没有设置验证方式,如下

    import os
    import sys  
    reload(sys)  
    sys.setdefaultencoding('utf8')  
    from pyhive import hive
    
    
    cnn=hive.Connection(host='your ip add', port='10000',username = username ) #connection中也可以指定db
    cursor=cnn.cursor()
    dbs = ['test1','test2','test3']
    
    for db in dbs:
        cursor.execute("show tables in %s"%(db))
        for table in cursor.fetchall():
            cursor.execute("show create table %s.%s"%(db,table[0]))
            for i in cursor.fetchall():
    	    print str(i[0]) # print or other operations~

    add : 2019-10-22

    安装sasl遇到错误 :  error: sasl/sasl.h: No such file or directory

    解决: yum -y install cyrus-sasl cyrus-sasl-devel cyrus-sasl-lib

     

     

    展开全文
  • 安装pyHive报错

    2020-11-10 11:08:59
    安装pyhive还需要安装其他依赖包 pip install sasl pip install thrift pip install thrift-sasl pip install PyHive 安装sasl遇到的问题 Collecting sasl Using cached ...
  • pyhive创建表

    2020-12-06 00:19:59
    from pyhive import hive conn = hive.Connection(host='hadoop104',port=10000,username='baiyun' ,database='gmall') cursor = conn.cursor() cursor.execute('show tables') cursor.execute('show tables') ...
  • python 利用pyhive 连接hiveserver2,系统linux 系统,通过pd 导出excel
  • anaconda3.6.5安装pyhive

    千次阅读 2018-06-25 17:48:10
    3.来到anaconda官网,搜索需要的包,pyhive(不是anaconda前缀的也行,例如biaze/pyhive) https://anaconda.org/anaconda/   如上截图,选择 anaconda 对应的pyhive,点击进去 点击FIles,选择linux-64/...
  • python 利用pyhive 导出excel 周数据,为 自然周数据
  • PyHive2 PyHive2 是一个 Python 包,通过 HIVE 查询促进分布式计算。 PyHive2 允许在 Python 中轻松使用 HQL(Hive SQL),并允许在 Hive 中轻松使用 Python 对象和 Python 函数。 安装 PyHive2 要求ant(为了构建...
  • 新环境需要使用pyhive,使用下面命令进行安装 pip install pyhive 安装pyhive之后,使用会报错: from thrift.Thrift import TType, TMessageType, TFrozenDict, TException, TApplicationException ...
  • python连接hive的工具可以用 pyhive 和 impala,不管是哪个配置都比较麻烦。需要的依赖包比较多。 https://github.com/cloudera/impyla https://github.com/dropbox/PyHive pyhive模块没有提供数据库连接池的API。...
  • linux与 win上装pyhive

    2021-07-17 21:53:46
    pyHIVe在win和linux上的安装 在windows上安装 sasl需要下载对应whl文件进行离线安装,其他直接进行安装 pip install sasl pip install thrift pip install thrift-sasl pip install pyhive linux上装 pyhive ,与...
  • pyhive python连接hive

    2018-11-15 17:04:18
    from pyhive import hive conn = hive.Connection(host='1*.30',auth='LDAP',port=10000,username='*',password='*',database='*') cursor = conn.cursor() cursor.execute("select * from dp_ods.*...
  • win10安装pyhive

    2020-07-10 17:50:03
    Python连接Hive(基于PyHive)
  • 本文的目的是在Windows环境下安装python的第三方包pyhive。 正文 1. 安装整体过程和所需要的安装包 先说整体需要安装的包 1. 安装pyhive包 pip/pip3 install pyhive 2. Windows安装Microsoft Visual C++ 14.0 环境...
  • pip install PyHive 2.连接hive 注意端口默认为10000 from pyhive import hive conn = hive.Connection(host='192.168.100.100', port=10000, username='root', database='default') cursor = conn.cursor() c
  • 使用PyHive操作Hive

    2017-10-23 17:57:00
    使用PyHive操作Hive 废话 搜了一下,看到了StackOverFlow的回答,试了一下前两个方案,感觉第二个更简洁,这里记录一下,更详细的见参考。 安装依赖 pip install sasl pip install thrift pip install thrift-sasl ...
  • python使用pyhive连接hive

    2019-09-17 15:18:47
    首先安装pyhive pip install pyhive 连接hive通过LDAP connection = hive.connect(host='HIVE_HOST', port=10000, ...
  • Python连接Hive(基于PyHive)

    万次阅读 热门讨论 2018-06-16 16:34:30
    要想使用python连接hive,...pip install PyHive   但是我们在安装sasl的时候可能会报错,导致安装不上,这个时候就得去sasl下载地址下载我们所需要的sasl,记得要和我们python版本匹配,我这里选择下载的是sa...
  • python pyhive查询hive

    千次阅读 2019-07-12 10:24:52
    安装phive pip install sasl pip install thrift pip install thrift-sasl pip install PyHive 工具类 class HiveClient: def __init__(self, db_host, user, password, database, port=10000): """ ...
  • PyHive PyHive是和的Python 和接口的集合。 用法 数据库API from pyhive import presto # or import hive or import trino cursor = presto . connect ( 'localhost' ). cursor () cursor . execute ( 'SELECT * ...
  • 导入pyhive from pyhive import hive if __name__ == '__main__': conn =hive.Connection(host="192.168.139.128",port="10000",username="root", database="hivereview",auth="NOSASL") cursor = conn.cursor()...
  • centos7 pyhive连接hive(基于kerberos)。由于基于kerberos校验的hive服务器通过python连接的过程很麻烦,故整理文档。
  • python下载pyhive包:

    2021-06-11 16:08:03
    windows安装pyhive: ** 打开cmd,依次输入以下命令: pip install sasl pip install thrift pip install thrift-sasl pip install sasl 输入pip install sasl 时我就掉入了坑里 Running setup.py clean for ...
  • python连接hive有两种方式,pyhive和impyla,impyla连接较为方便,pyhive在linux上很方便,在windows上较麻烦,本文主要介绍pyhive的安装方法 1.从官网下载sasl https://www.lfd.uci.edu/~gohlke/pythonlibs/#sasl ...

空空如也

空空如也

1 2 3 4 5 ... 20
收藏数 924
精华内容 369
关键字:

pyhive