Skip to main content

Hadoop Hive Session issue in weblogic data source

Hi Everyone,

Recently i come with a requirement about Hadoop Hive integration in Oracle SOA, For same i wrote a blog which you can find on below link.

https://oraclesoaosbbpmapi.blogspot.com/2019/03/hadoop-hive-data-base-fetching-records.html

After successfully configuring hive in weblogic, We faced a new issue which is about connection.

Before going deep about issue and solution let's see how HIVE DB manage connection on server side.


  • when ever we make a data base connection for HIVE DB, it create an internal session and generates a session id which is valid for a specific period of time which is configured on server level.
  • After that period of time Hive create a new session id and process the query.
How weblogic data source manage connection:
  • When we create a data source in weblogic, We define a connection pool size by which we can use existing connections which are not in use by any other resource by which weblogic minimize the connection cost.
Issue due to complete different behavior of connection management in both env:
  • We run a query around 11AM IST to fetch the data.
  • On hive DB we configured a session time out 30 minutes.
  • When we run a query again after 11:35AM, It will generate hive session error in place of re-creating a new connection in place of creating new session with hive.
Why no new connection is created:
  • When we configured weblogic data source and define connection pool, at that time we configured two properties which are
  • Initial capacity: We usually make it as 1.
  • Minimum capacity: We usually make it as 1.
  • Due to property Minimum Capacity, Weblogic always keep at-least one minimum connection in it's connection pool and when a resource request for a connection it releases same from it's pool with existing session ID which will invalidated after session time.
When you will run an instance, You will see below error message in SOA instance.

Invalid SessionHandle: SessionHandle [af5ff8d7-2f51-4d75-8d4c-52f15e049121]


How to fix this issue: 

There are total three solution that i found which will work without any failure.

  1. Data source connection pool configuration
  2. Hadoop Session time out configuration, It is not safe and guaranteed to work in all case.
  3. Data source based HIVE session management.

  • Hadoop Session time out configuration, It is not safe and guaranteed to work in all case:
    • Increasing the timeout for the Apache Hadoop environment for resolving the behavior:      In the Cloudera Manager (CM), navigate to: Hive > Configuration > Search: Session


  • Data source based HIVE session management: Add below properties in JDBC Connection string
    • jdbc:hive2://IP:10000/default?hive.server2.idle.session.timeout=3600ms
    • While configuring about connection string you may get below error if client side session handling is not enabled.
    • Test failed: Failed to open new session: org.apache.hive.service.cli.HiveSQLException: java.lang.IllegalArgumentException: Cannot modify hive.server2.idle.session.timeout at runtime. It is not in list of params that are allowed to be modified at runtime
    • For same you to make some change in hiveserver2 hive-site.xml file.
    • <property> <name>hive.security.authorization.sqlstd.confwhitelist</name> <value>hive.server2.idle.session.timeout</value> </property>Data source connection pool configuration

  • Weblogic data source connection pool:
    • Click on Hive data source
    • Go to connection pool tab
    • Search for initial capacity=0
    • Search for minimum capacity=0
    • Activate the change.

By any one the way, you can fix the issue.


Have question post me here!!!!!!!!!!
Labels:

Comments

Post a Comment

Popular posts from this blog

Oracle SOA 12C rest adapter with Custom HTTP headers

Most existing web applications are connected through web services, which are commonly known as SOAP services. More and more users are relying on mobile devices for communication, and they’re looking for lighter ways to access enterprise information on the go. REST services are the answer for the mobile device platform, because they get rapid responses and fast access to data. Oracle SOA Suite 12 c  provides a complete set of service infrastructure components for designing, deploying, and managing composite applications. Oracle SOA Suite 12 c  enables services to be created, managed, and orchestrated into composite applications and business processes. Some time we have need to send HTTP headers in REST service, In OSB we use header component and add what ever is needed but in oracle SOA 12C it's little bit different. Let see how we can do it. Create one SOA Application. Create one SOA Sample project inside SOA Application. Go to composite and drag drop REST adapt
22 comments
Read more

Dynamic Routing in OSB 12C

Dynamic Routing in OSB cab be used when the BusinessService endpoint required to be determine at runtime in message flow. Consider a scenario where OSB has to route the incoming requests to 2 different services based on the CustomerType element value sent in the payload. So create a XQuery resource with the following contents. Observe that we are using the absolute path of business service in configuration as required by dynamic routing. Following is the XML schema that we use:>>  Customer.xsd <?xml version="1.0" encoding="windows-1252" ?> <xsd:schema targetNamespace="http://xmlns.oracle.com/schema/Customer" xmlns:xsd="http://www.w3.org/2001/XMLSchema"             xmlns:ns1="http://xmlns.oracle.com/schema/Customer">   <xsd:complexType name="Customer">     <xsd:sequence>       <xsd:element name="CustomerId" type="xsd:string"/>       <xsd:elemen
14 comments
Read more

Swagger API document from Any WADL & Schema in Oracle SOA

Hi everyone, Hope everyone is doing well these days, Recently i started a project work over how to generate swagger API document for your any REST API, In case if you don't know what is swagger please go and check " https://swagger.io/tools/swagger-editor/ ". It's a great and easy to use tool which will help to create user friendly, human readable form API documentation with extension for generating API client in different languages with capability of testing your API from same. What is swagger editor, Design, describe, and document your API on the first open source editor fully dedicated to OpenAPI-based APIs. The Swagger Editor is great for quickly getting started with the OpenAPI (formerly known as the Swagger Specification) specification, with support for Swagger 2.0 and OpenAPI 3.0.  What benefits you will get by using swagger, Runs Anywhere, The Editor works in any development environment, be it locally or in the web. Smart Feedback, Validate you
8 comments
Read more