Skip to content

hikariCP 注入自定义数据源

最近公司集成clickhouse的过程中,进行了一系列的性能及压力测试。其中应用使用了官方的clickhouse-jdbc驱动,数据源连接池使用的hikariCP,测试结果显示如果查询时间超长就会报read timeout错误,经过排查官方issues给出了解决办法:

ClickHouseProperties properties = new ClickhouseProperties();
properties.setSocketTimeout(timeout);
new ClickHouseDataSource(url , properties);

or

Properties properties = new Properties();
properties.put("socket_timeout", timeout);
new ClickHouseDataSource(url, properties);

但是实际情况我们使用的默认的hikariCP连接池,进行的默认配置,无法设置该参数。
样例代码如下:

配置文件大致如下:

spring.datasource.type=com.zaxxer.hikari.HikariDataSource
spring.datasource.driver-class-name: ru.yandex.clickhouse.ClickHouseDriver
spring.datasource.hikari.connectionTimeout=30000
spring.datasource.hikari.idleTimeout=600000
spring.datasource.hikari.maxLifetime=1800000

connectionTimeout 参数根本无效,经过代码排查发现,其问题在于cliclhouse-JDBC 是通过http方式进行连接,并且没有完全实现jdbc接口。设置超时代码如下:

hikariCP部分:

com.zaxxer.hikari.pool.PoolBase

/**
 * Set the network timeout, if <code>isUseNetworkTimeout</code> is <code>true</code> and the
 * driver supports it.
 *
 * @param connection the connection to set the network timeout on
 * @param timeoutMs the number of milliseconds before timeout
 * @throws SQLException throw if the connection.setNetworkTimeout() call throws
 */
private void setNetworkTimeout(final Connection connection, final long timeoutMs) throws SQLException
{
   if (isNetworkTimeoutSupported == TRUE) {
      connection.setNetworkTimeout(netTimeoutExecutor, (int) timeoutMs);
   }
}

clickHouse-JDBC部分:

ru.yandex.clickhouse.ClickHouseConnectionImpl

public void setNetworkTimeout(Executor executor, int milliseconds) throws SQLException {
   
}

经过仔细研究代码发现,clickhosue超时的设置是在http请求建立时设置的,使用的Httpclient。

ru.yandex.clickhouse.util.ClickHouseHttpClientBuilder

public CloseableHttpClient buildClient() throws Exception {
    return HttpClientBuilder.create()
            .setConnectionManager(getConnectionManager())
            .setKeepAliveStrategy(createKeepAliveStrategy())
            .setDefaultConnectionConfig(getConnectionConfig())
            .setDefaultRequestConfig(getRequestConfig())
            .setDefaultHeaders(getDefaultHeaders())
            .disableContentCompression() // gzip здесь ни к чему. Используется lz4 при compress=1
            .build();
}

设置socketTimeout部分:

private RequestConfig getRequestConfig() {
    return RequestConfig.custom()
            .setSocketTimeout(properties.getSocketTimeout())
            .setConnectTimeout(properties.getConnectionTimeout())
            .build();
}

而hikariCP找到可以直接设置数据源的方法:

com.zaxxer.hikari.HikariConfig

/**
 * Set a {@link DataSource} for the pool to explicitly wrap.  This setter is not
 * available through property file based initialization.
 *
 * @param dataSource a specific {@link DataSource} to be wrapped by the pool
 */
public void setDataSource(DataSource dataSource)
{
   checkIfSealed();
   this.dataSource = dataSource;
}

最终解决办法,修改定义数据源中的语句:

HikariConfig config = new HikariConfig(properties);
ClickHouseProperties ckProperties = new ClickhouseProperties(); 
ckProperties.setSocketTimeout(timeout); 
ClickHouseDataSource chDs=new ClickHouseDataSource(url , ckProperties);
config.setDataSource(chDs);
return new HikariDataSource(config);

发表评论

电子邮件地址不会被公开。 必填项已用*标注