HBase
The DolphinDB hbase plugin can establish a connection to HBase via Thrift and load data from the HBase database.
Recommended version:
- HBase: version 1.2.0
- Thrift: version 0.14.0
Installation (with installPlugin
)
Required server version: DolphinDB 2.00.10.8 or higher
Supported OS:
- Linux x64: Requires DolphinDB Server version 2.00.10 or higher.
- Linux ARM: Requires DolphinDB Server version 200.14.2/300.2.2 or higher.
Installation Steps:
(1) Use listRemotePlugins to check plugin information in the plugin repository.
Note: For plugins not included in the provided list, you can install through precompiled binaries or compile from source. These files can be accessed from our GitHub repository by switching to the appropriate version branch.
login("admin", "123456")
listRemotePlugins()
(2) Invoke installPlugin for plugin installation
installPlugin("hbase")
(3) Use loadPlugin to load the plugin before using the plugin methods.
loadPlugin("hbase")
Start Thrift server:
Run the following command to start the Thrift server with port specified as 9090:
$HBASE_HOME/bin/hbase-daenom.sh start thrift -p 9090
You can close the Thrift server with the following command:
$HBASE_HOME/bin/hbase-daemon.sh stop thrift
Method References
connect
Syntax
connect(host, port, [isFramed])
Details
Build a connection to HBase via Thrift server and return an HBase handle.
Parameters
- host: A STRING scalar indicating the server address to connect to.
- port: An integer indicating the port number of the Thrift server.
- isFramed: A BOOLEAN scalar indicating the way of data transmission. If set to false (default), data is transmitted through
TBufferedTransport
; if set to true, data is transmitted throughTFramedTransport
.
Examples
conn = hbase::connect("192.168.1.114",9090)
Note: If the connection remains inactive for a while (1min by default), HBase will automatically close it. If you try to operate through this connection, the No more data to read
error will be reported. In such a case, you have to execute hbase::connect
to reconnect.
You can configure with hbase.thrift.server.socket.read.timeout
and hbase.thrift.connection.max-idletime
to change the timeout.
The following configuration parameters change the timeout to 1 day.
<property>
<name>hbase.thrift.server.socket.read.timeout</name>
<value>86400000</value>
<description>eg:milisecond</description>
</property>
<property>
<name>hbase.thrift.connection.max-idletime</name>
<value>86400000</value>
</property>
showTables
Syntax
showTables(hbaseConnection)
Details
Return all table names of the connected database.
Parameters
- hbaseConnection: The handle returned by
hbase::connect
.
Examples
conn = hbase::connect("192.168.1.114", 9090)
hbase::showTables(conn)
deleteTable
Syntax
deleteTable(hbaseConnection, tableName)
Details
Delete the tables in the database.
Parameters
- hbaseConnection: The handle returned by
hbase::connect
. - tableName: A STRING scalar/vector indicating the name of the table to be deleted.
Examples
conn = hbase::connect("192.168.1.114", 9090)
hbase::deleteTable(conn, "demo_table")
getRow
Syntax
getRow(hbaseConnection, tableName, rowKey, [columnName])
Details
Return the specific record with rowKey.
Parameters
- hbaseConnection: The handle returned by
hbase::connect
. - tableName: A STRING scalar indicating the name of the table to be read.
- rowKey: A STRING scalar indicating the index of the row to be read.
- columnName: A STRING scalar/vector indicating the name of the column to be read. If not specified, all columns are read by default.
Examples
conn = hbase::connect("192.168.1.114", 9090)
hbase::getRow(conn, "test", "row1")
load
Syntax
load(hbaseConnection, tableName, [schema])
Details
Import the HBase results into a DolphinDB in-memory table. The data types supported for schema are described in "Data Type Mappings".
Parameters
- hbaseConnection: The handle returned by
hbase::connect
. - tableName: A STRING scalar indicating the name of the table to be loaded.
- schema (optional): If specified, it is a table containing names of the columns to be imported and their data types. The column names specified in schema must be consistent with the HBase column names. If not specified, the table will be created based on the first row with each column of STRING type. Note that each row must have the same size.
Examples
conn = hbase::connect("192.168.1.114", 9090)
t = table(["cf:a","cf:b", "cf:c", "cf:time"] as name, ["STRING", "INT", "FLOAT", "TIMESTAMP"] as type)
t1 = hbase::load(conn, "test", t)
Data Type Mappings
The following is the data type mappings when an HBase table is imported to DolphinDB. Data stored in HBase must conform to the types specified in the table below, otherwise NULL values will be returned.
Type | HBase | DolphinDB |
---|---|---|
BOOL | true, 1, FALSE | true, true, false |
CHAR | a | a |
SHORT | 1 | 1 |
INT | 21 | 21 |
LONG | 112 | 112 |
FLOAT | 1.2 | 1.2 |
DOUBLE | 3.5 | 3.5 |
SYMBOL | s0 | "s0" |
STRING | name | "name" |
DATE | 20210102, 2021.01.02 | 2021.01.02, 2021.01.02 |
MONTH | 201206, 2012.12 | 2012.06M, 2021.12M |
TIME | 052013140, 05:20:01.999 | 05:20:13.140, 05:20:01.999 |
MINUTE | 1230, 13:30 | 12:30m, 13:30m |
SECOND | 123010, 13:30:10 | 12:30:10, 13:30:10 |
DATETIME | 20120613133010, 2012.06.13 13:30:10, 2012.06.13T13:30:10 | 2012.06.13T13:30:10, 2012.06.13T13:30:10, 2012.06.13T13:30:10 |
TIMESTAMP | 20210218051701000, 2012.06.13 13:30:10.008, 2012.06.13T13:30:10.008 | 2021.02.18T05:17:01.000, 2012.06.13T13:30:10.008, 2012.06.13T13:30:10.008 |
NANOTIME | 133010008007006, 13:30:10.008007006 | 13:30:10.008007006, 13:30:10.008007006 |
NANOTIMESTAMP | 20120613133010008007006, 2012.06.13 13:30:10.008007006, 2012.06.13T13:30:10.008007006 | 2012.06.13T13:30:10.008007006, 2012.06.13T13:30:10.008007006, 2012.06.13T13:30:10.008007006 |